Secret APIs now submit the native SQL statements with the credential
bound as an Arrow string parameter:
```sql
CREATE SECRET "api-key" TYPE = GENERIC_STRING SECRET_STRING = $1;
ALTER SECRET "api-key" SET SECRET_STRING = $1;
DESCRIBE SECRET "api-key";
```
`create_secret` and `alter_secret` execute SQL through the existing
query-backed job adapter and wait for that query to complete. The job's
`id` is the query ID and its type is `query`; credentials stay out of
SQL text. `describe_secret` reads the native `name`,
`created_at_millis`, and `updated_at_millis` columns. This replaces the
unreleased `AS $1` syntax and JSON-envelope response without a
compatibility fallback.
Python methods and namespace handling remain the same:
```python
db.create_secret("api-key", "credential", namespace_path=["team"])
db.alter_secret("api-key", "rotated", namespace_path=["team"])
info = db.describe_secret("api-key", namespace_path=["team"])
```
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
Fixes#4749
Python's `__version__` forwards Cargo's SemVer spelling, so prereleases
report values such as `0.41.0-beta.17` while the installed package
metadata reports `0.41.0b17`.
Normalize the native version with the existing
`packaging.version.Version` dependency before exposing it in Python.
Keep native version sourcing so imports still work with `lancedb-compat`
and do not require distribution metadata. Regression tests cover
metadata equality and stable, alpha, beta, release-candidate, and
development versions with metadata lookups disabled.
Validation:
- Rebuilt the native extension through the uv-managed development
environment.
- Confirmed the metadata regression fails on the original code with the
reported string mismatch.
- Import tests, package metadata tests, the remote client user-agent
test, and CI Python dependency tests: 14 passed, 2 skipped (Windows-only
wheel check and uninstalled `lancedb-compat`).
- macOS ARM64/Python 3.13 dependency resolution and `uv lock --check`
pass with the platform-specific test dependency pins from current
`main`.
- Python-tree Ruff lint and formatting checks, focused Pyright check,
Rust formatting, and PR diff whitespace checks pass.
- Repository-wide Ruff lint reports existing findings in unchanged
`docs/web-tests` files.
<!-- lance-gatekeeper-fix:v1 agent=3e23555743cc116ea53ca8a7e3c4ebb1
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4750
Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior.
Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL.
`LanceModel` instances with validation aliases write field-name keys to Arrow, while `to_pydantic` previously reconstructed models by alias. This normalizes field-name rows to the applicable single-component validation alias for both sync and async reads, while preserving existing alias keys, alias precedence, name-only validation, and collision behavior.
Adds sync and async regression coverage for string aliases, `AliasChoices`, single-component `AliasPath`, `populate_by_name`, and explicit `validate_by_name` precedence.
Allow OpenAIEmbeddings.ndims() to use an explicitly configured dim for model
names outside the three built-in OpenAI names, so schemas can be built for
compatible models.
Preserve existing model defaults and request behavior. Add regression coverage
for the resulting Arrow schema dimensions and for all known OpenAI model
dimensions.
Allows Python namespace implementations to read request dictionary entries as attributes while preserving mapping and `model_dump()` behavior. Missing dunder attributes continue to raise `AttributeError`.
Adds a real namespace connection regression covering list/open operations and omitted optional fields.
Fixes#4701
The invalid-table-name diagnostic now includes the missing space before
"periods are allowed".
The regression test covers the corrected message returned for `my/table`.
Add a regression test for the public optimize-all path on a table with an IVF_HNSW_SQ index. The test creates 512 rows, runs the full optimize action, and verifies that the single retained index still covers all 512 rows with zero unindexed rows.
Test-only change; no production behavior changes.
References #3211.
The core `col()` camelCase integration coverage requested by #3211 is already on `main` from #4493. This PR adds the adjacent raw-SQL table-surface regressions that were previously uncovered:
- unquoted `where("camelCase = 1")`
- `update` and `delete` with unquoted camelCase predicates
- `mergeInsert` with a camelCase join key, including matched-update and not-matched-insert input
- `addColumns` `valueSql` referencing a camelCase column
- `createIndex` on a camelCase column
These are characterization tests for the existing local-backend behavior. They do not add `col()` coverage or require a live remote service.
## Summary
- add a regression test for object-form `createTable` on two independent local Node.js connections
- verify both connections use the same table name while preserving each database's rows
## Scope
This test covers the public local `connect(tmpDir)` path. It does not exercise LanceDB Cloud or remote create requests, and it does not establish a remote root cause or a complete fix for #952.
References #952
Fix the dead README ecosystem links by pointing readers to the current official LanceDB documentation for LangChain and LlamaIndex.
Original contribution by @sarankumar1325.
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.
Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.
Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.
Validation:
- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.
Fixes#4776
<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
A second flattened Function declaration fails when the remote describe
schema omits an existing binding's internal assignment field. The client
incorrectly requires that server-owned field and compares it as a
physical user output.
Exclude assignment fields from binding/schema validation and the
output-schema comparison. Filter declared assignment fields from remote
schemas, including seeded caches and snapshot handles, while retaining
schema metadata.
The regression tests declare two all-nullable flattened bindings
consecutively with no assignment declaration, a hidden assignment field,
or a returned assignment field. They also cover cached and snapshot
schemas and server-owned assignment types and metadata. All four
regression cases failed before the fix.
Validation:
- `cargo fmt --all`
- `cargo test --quiet --profile ci --features remote -p lancedb --lib --
table::computed_columns::tests remote::table::tests` — 379 passed, 30
ignored.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`
Fixes#4778
<!-- lance-gatekeeper-fix:v1 agent=c55eff66e22d736a6b954a9eeec88898
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.
```python
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"private_hub", "hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing SQL transport]
SQL --> Metadata[Integration and token Secret reference]
```
Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.
```python
info = db.create_storage_integration(
"gcs", "gs://customer-data/raw/", storage_provider="GCP",
gcp_credentials_json_secret=SecretReference(
"gcp_json", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```
```sql
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
SQL --> Config[Configuration and one Secret reference]
```
Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
Register Azure Blob integrations through Flight SQL using an explicit
storage account and three Secret references for tenant ID, client ID and
client secret. Extends the database-owned integration APIs with
provider-specific inputs while keeping the same create, describe, list
and drop operations.
```python
info = db.create_storage_integration(
"customer_blobs", "az://customer-data/raw/",
storage_provider="AZURE", azure_storage_account="customeraccount",
azure_tenant_id_secret="tenant", azure_client_id_secret="client",
azure_client_secret_secret="credential",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("customer_blobs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("customer_blobs")
```
Sync and async connections accept the same options. Use
`SecretReference` for cross-database or namespace qualification. Values
remain in Secrets; returned metadata includes only references and setup
guidance.
```sql
CREATE STORAGE INTEGRATION customer_blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = public.tenant
STORAGE_AZURE_CLIENT_ID_SECRET = public.client
STORAGE_AZURE_CLIENT_SECRET_SECRET = public.credential;
DESCRIBE STORAGE INTEGRATION customer_blobs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION customer_blobs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Flight SQL]
SQL --> Metadata[Configuration and Secret references]
```
Azure supports service principal client credentials only. Adds SQL
rendering, provider validation and Python Flight SQL regressions across
Utf8, LargeUtf8 and Utf8View results. Local builds and tests are
excluded as requested; validation runs in CI.
Add `on_transform_error="fill"` to `StreamingDataset`, replacing failed
transform rows at their original positions with a deep-copied fill value or
the result of `repair(exc, row_batch)`.
Fill mode preserves split length, elastic step layout, worker accounting, and
checkpoint progress. Repair failures remain fatal, and `rows_filled` is shared
across workers.
Validated with focused fill and adjacent dataloader tests (10 and 6 passed)
and the complete applicable CI run for
6b51b33e56: four workflows and fifteen checks
all succeeded.
Closes#3916
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
Resolve retry settings from the six documented `LANCE_CLIENT_RETRY_*`
environment variables through the shared retry configuration. Explicit
configuration takes precedence, invalid values return `InvalidInput`, and
status lists tolerate whitespace and trailing commas while a blank list
disables status-based retries.
Saturate calculated backoff at one hour instead of panicking when the
exponential exceeds the representable `Duration` range, including retry
counts configured from the environment.
Includes coverage for defaults, all six environment variables, explicit
configuration precedence, status parsing, invalid environment and direct
configuration values, real process-environment wiring, and saturating
backoff.
Closes#2350
Originally authored by Joshua Snider (@jsnider3).
Replaces #4322, keeping only its property graph defined over tables.
A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs.
- **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it.
- **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph.
- **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust.
```python
import lancedb
from lancedb.graph import EdgeTable, Endpoint, NodeTable
db = lancedb.connect("db://demo", api_key="...")
db.create_property_graph(
"social",
nodes=[NodeTable("person", key="person_id", label="Person")],
edges=[
EdgeTable(
"knows",
label="KNOWS",
source=Endpoint("src_id", references=("person", "person_id")),
destination=Endpoint("dst_id", references=("person", "person_id")),
)
],
)
db.execute_query("""
SELECT * FROM GRAPH_TABLE (social
MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30
COLUMNS (p.name AS person, f.name AS friend))
""").read_all()
```
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Fixes#4403
Implement remote drop_all_tables() support by listing tables and dropping them individually, including tables within namespaces. Handle paginated table listings and propagate errors appropriately. Add tests covering root and namespace tables, pagination, empty databases, and listing failures to ensure reliable remote database cleanup.
Rename the SQL-facing table functions:
- `vector_dedup` becomes `dedupe`.
- `vector_duplicate_pairs` becomes `duplicate_pairs`.
This is an intentional breaking rename without compatibility aliases. It updates the native table-function diagnostics and SQL coverage; the coordinated Sophon PR will update distributed binding and deployment integration.
BREAKING CHANGE: `vector_dedup` is now `dedupe`, and `vector_duplicate_pairs` is now `duplicate_pairs`; no compatibility aliases are provided.
The affected jobs were cancelled before runner assignment with GitHub's scheduled macOS 14 brownout annotation. macOS 14 runner images retire on November 2, 2026: https://github.com/actions/runner-images/issues/13518.
This moves the Python and CLI workflows to `macos-15` and keeps two distinct Rust macOS legs with `macos-15` and `macos-26`. All selected labels are public ARM64 runners, preserving the architecture of the previous plain `macos-14` jobs.
Validation: `git diff --check` and Ruby YAML parsing passed for all three changed workflows. A targeted `actionlint` check also passed; existing repository-wide custom-runner, reusable-workflow input, and shellcheck findings are outside this diff.
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Adds `image` to `@udf`, setting `runtime.environment.image` so a
Function can be built on a digest-pinned registry image instead of the
default base, for dependencies pip, conda and apt cannot express. The
client refuses references that are not pinned by digest; the server
validates the rest. The field is omitted when unset, so existing
Function identities are unchanged.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Removes the Codex Fix CI workflow. It ran an AI agent that read CI logs,
which outside contributors can write to, while the agent had the
repository's write token and no sandbox. Hardening it would take ongoing
work, and we don't use it enough to justify that, so we're removing it.
## Merge Risk
**Door:** two-way. The workflow can be restored from history.
**Blast Radius:** ci. Only affects manual dispatches of this workflow.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
Document vectorized `pyarrow.Array` UDFs for Blob v2 inputs and outputs.
Clarify annotation-selected array execution and the adapter-v1 client/executor
boundary. Add public regression coverage for registering, hydrating, executing,
publishing, and reading back Blob arrays, including large, empty, and null values.
Use `ThreadingHTTPServer` for the shared remote-test mock so concurrent `table.add` requests are not handled serially by the test server.
Validation:
- Reproduced the connection-reset/response-decoding failure against base `13749a3f138fde2647c8bdb8a1c05545026aee57`: 27 targeted runs passed, then the 28th matched the original CI failure.
- The patched target passed 100/100 runs; `test_remote_db.py` passed 261 tests with 2 skipped.
- Ruff 0.9.9 format and lint checks passed.
- Current-head GitHub CI: 19/19 check runs and 5/5 workflow runs succeeded.
## Summary
- Add a downstream IVF-HNSW-SQ merge regression using descending row IDs.
- Detect any reintroduction of merge-time row sorting without allocating the original multi-gigabyte fixture.
## Root cause
Lance 4 sorted each partition by row ID with `RecordBatch::take`. Arrow represented those take indices as `UInt32`, and `take_fixed_size_list` overflowed when `num_rows * vector_dimension` exceeded `u32::MAX`. The Lance version pinned by current `main` already includes lance-format/lance#6332, which removes that unnecessary sort. This test locks the corrected behavior at the LanceDB dependency boundary: the old implementation reorders the sentinel row IDs, while the fixed implementation preserves them.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb --lib table::create_index::tests::test_ivf_hnsw_sq_merge_preserves_shuffled_row_order -- --exact`
- Full current-head PR CI: 40 successful checks and one expected Windows ARM check skipped because that job runs only on `main`.
Fixes#3126
## Summary
- lock in the upstream fix that batches legacy row-ID fragment reservations into one coordinated commit
- exercise 32 indexed compaction tasks with 64 workers, exceeding the former 20-retry contention threshold
- verify data integrity, compaction metrics, and the expected two-commit transaction footprint
## Root cause
Parallel compaction used to reserve fragment IDs independently in every rewrite task. On high-core machines, those internal commits contended with one another and could exhaust the commit retry budget. The pinned Lance implementation now gathers the completed tasks and reserves all required fragment IDs once before the final rewrite commit; this regression test protects that behavior at the LanceDB integration boundary.
## Validation
- cargo fmt --all -- --check
- cargo test --quiet --features remote -p lancedb --lib table::optimize::tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples (passes with four pre-existing redundant_pub_crate warnings in remote-table code)
Fixes#1971
<!-- lance-gatekeeper-fix:v1 agent=18fd17ea143769e1da7c5aa5222ecdda generation=1 -->
## Summary
- Compare equal-size 1,000-value deletes confined to one BTree page and spread across many pages.
- Use an isolated current-thread allocation counter instead of a wall-clock threshold.
- Warm index pages first so the assertion detects repeated predicate materialization rather than I/O.
## Root cause
Lance previously rebuilt a large `IN` predicate for every BTree page, making indexed
lookups scale as `O(pages x values)`. The pinned Lance revision compiles the predicate
once and reuses it across pages. The regression guard observes the allocation scaling
caused by that invariant directly.
Fixes#2085
`rust/lancedb/src/io/object_store/io_tracking.rs` was a near-verbatim copy of lance-io's `utils::tracking_store`, and had drifted behind it. It was `#[cfg(test)]`-only with two usages, both in tests, so this PR points those at `lance_io::utils::tracking_store::IOTracker` and deletes the copy.
lance-io's `test-util` feature is enabled as a dev-dependency, which brings the `assert_io_eq!` / `assert_io_lt!` macros. Those print the recorded requests (method, path, byte range) when an assertion fails, instead of just the counter that didn't match, so the IO-counting tests in `table/dataset.rs` now say *which* IO happened when they fail.
`test_stats_does_not_read_data_files` no longer plumbs an `object_store_wrapper` at all: every lance-io `ObjectStore` now carries its own tracker, reachable as `store.io_tracker()`. This also makes the test stronger — the optimized local reader bypasses the `object_store` layer, so reads that would have gone uncounted through a wrapper are recorded by the built-in tracker.
Fixes#4076
## Not included
Two lance-side items from the issue are left for follow-up there, since neither can be fixed from this repo:
- `IOTracker::wrap_paginated` returns `Some(original)` on the premise that a pushed-down listing counts itself, which only holds for the local lister — a paginated listing against S3 is invisible to the tracker. The deleted copy had the identical bug, so this is not a regression.
- The deleted copy had poison-tolerant locking (`unwrap_or_else(|e| e.into_inner())`, from #3196) where lance still `.unwrap()`s. That tolerance, and the two unit tests covering it, go away with the file; it is worth carrying upstream instead of re-forking here.
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.
### Python experience
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```
For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.
### SQL experience
```sql
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```
CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.
Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
## Problem
The Python test extra pins `pylance==15.0.0b7` for every platform. On macOS, pip cannot resolve that pin: the configured `lance-format` index publishes b7 wheels for Linux and Windows but not macOS, while public PyPI returns HTTP 404 for the version and the `lancedb` index contains no 15.x release. The macOS Python dependency installation consequently cannot resolve the pin.
## Change
- Keep the exact `pylance==15.0.0b7` test pin on non-macOS platforms.
- Use `pylance==15.0.0b4` on macOS. This is the newest release in this range with a macOS wheel (`cp310-abi3-macosx_11_0_arm64`).
- Refresh `python/uv.lock` for the platform-specific resolutions.
The production Rust/Lance inputs are unchanged: `Cargo.lock` is byte-identical to `main` and remains on Lance `15.0.0-beta.7`. No tests, assertions, coverage, or CI checks are removed or weakened.
## Validation
- Confirmed the wheel identity from the configured index and matched its published SHA-256 to `uv.lock`; wheel metadata reports `Name: pylance`, `Version: 15.0.0b4`.
- Verified the b4 APIs used by the Python tests (`blob_field`, `blob_array`, `BlobType`, `Blob`, `dataset`, `write_dataset`, `FFILanceTableProvider`, and `DirectoryNamespace`). The Python test suite does not reference the b7-only `mark_sstables_as_compacted`, `mem_wal_dir_id`, `mem_wal_index_details`, `clustering_id`, `LanceFragment`, `IndexSegment`, `UpdateConfig`, or `max_source_fragments` changes.
- Verified scalar read/write interoperability in both directions between b4 and the beta.7 LanceDB native binding, plus beta.7 reads of a b4-written blob dataset with byte and null fidelity.
- `uv lock --check` with both configured Fury indexes and `unsafe-best-match`.
- CI dependency tests.
- Ruff 0.9.9 format and lint checks.
- Targeted Python tests: 911 passed, 22 skipped.
- Full Python suite (`-m "not slow and not s3_test"`): 2858 passed, 157 skipped, 70 deselected.
## Summary
Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.
**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.
**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.
**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.
**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Updates Rust workspace and Java Lance dependencies to
[v15.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.7),
and pins Python tests to the published `pylance==15.0.0b7` release with
refreshed Cargo and uv lockfiles. No compatibility fixes were required.
Validation: workspace Clippy with all features and warnings denied,
Cargo formatting, uv lock consistency, and all 18 CI-script tests pass;
repository-wide Ruff diagnostics are pre-existing.
---------
Co-authored-by: Lu Qiu <lu@lancedb.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add a LanceDB command line with the following functionality:
- `lancedb auth`: log in with OAuth, check cached OAuth session status,
or log out.
- `lancedb profile`: Create, edit, list, show, select, and delete
connection profiles.
- When no subcommand is given, we will attempt to create a SQL shell.
Update ci/check_cargo_lock.py and its regression tests to allow
pruning dependency edges while their target packages remain in
Cargo.lock. Removing the CLI from the Python source distribution
can disable optional features such as Axum tracing, even though
other SDK dependencies still require the tracing package.
## Summary
Replaces the generated availability table under **Compare features** on
the landing page with a hand-written list of features, grouped by area,
with separate Embedded and Enterprise columns.
The generated table listed every page that declares `availability`,
including overview and basic CRUD pages. Rows such as "Overview"
appeared three times without context, and pages like Insert, Update,
Database and the integrations showed up as if they were features.
| Area | Features |
| --- | --- |
| Search | Vector, full-text and hybrid search, filtering, reranking,
query plans, distributed search |
| Indexing | Vector, full-text and scalar indexes |
| SQL and curation | SQL, aggregation, sampling, joins, ranking and
deduplication, near-duplicate detection, chunking, changes since a
version, views |
| Feature engineering | Computed columns, materialized views, embedding
registry |
| Training | PyTorch data loader, permutations, distributed training |
| Data management | Branches, compaction and cleanup, background jobs,
Real Time engine |
| Security | Authentication, authorization |
Each ✅/❌ matches the linked page's `availability` on `main`; SQL, which
declares none, is Enterprise-only. Functions are left out until it's
settled whether they are published.
The same list, limited to 1.10 features, is in the Enterprise 1.10
editions in lancedb/docs#398.
Checked by assembling the docs site with this change: `mint validate`
passes and `mint broken-links --check-anchors --check-redirects` finds
no broken links. Vale reports no new warnings.
After this merges, the committed `lancedb` skill in lancedb/docs needs a
rebuild (`make assemble && make skill`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes#4733
PyArrow expects Blob V2 storage structs when constructing a record batch
and cannot infer the extension array from raw bytes. LanceDB already
converts binary values during ingestion, but batch writers had no
supported array constructor.
Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes
and nulls, using the same extension type resolver as `blob()`. The
helper preserves empty blobs and null blobs and works with both
pylance's type and LanceDB's fallback. Document the descriptor fields,
exclusive inline/external representations, batch construction, and
existing binary conversion in synchronous and asynchronous
`create_table` / `add` APIs.
Validation:
- Reproduced the reported `ArrowTypeError` on the original code; both
new regression cases fail against the original schema module.
- 256 focused blob tests and schema doctests passed; one optional PyAV
test skipped.
- Both new regression cases also passed with PyArrow 25, with and
without pylance imports.
- Python package Ruff checks and formatting passed.
- MkDocs build passed, with an existing unrelated `SecretReference`
cross-reference warning. Repository-wide Ruff reports 48 existing errors
in `docs/web-tests`.
<!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
Changes links to home page sections from `/#<anchor>` to
`/index#<anchor>`.
The docs site's Assemble CI fails `mint broken-links --check-anchors` on
every `/#architecture` link since #4732 moved the Architecture page onto
the home page. The anchor exists; Mint 4.2.888's checker can't resolve
any anchor on the root page `/`, but resolves `/index#…`.
`docs.lancedb.com/index` serves the home page, so readers land in the
same place.
Verified by assembling the site with this change and the matching
Enterprise docs change: `mint validate` passes and `mint broken-links
--check-anchors --check-redirects` finds no broken links. The
`docs.json` redirects keep `/#architecture`, which the checker accepts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Addresses some comments on the staging branch. Rebased onto #4732, which
rebuilt `index.mdx` and the quickstart and reverted the landing copy to
the pre-v0.4 headings; the final copy is reapplied here.
| Comment | Page | Change |
| --- | --- | --- |
| Taylor: drop the comma | `index.mdx` | "one, unified platform" → "one
unified platform" |
| Taylor: "Preparing t…" + "Data doesn't slow down" | `index.mdx` |
"Training data and experimentation slow down…" → "Preparing training
data and running experiments slow down…" |
| Fei: subtitles should align with the new home page | `index.mdx` |
Follows the final homepage copy (v0.4): "Train on the right data faster"
with the hero subtitle, "Accelerate the data flywheel for model
research" with its intro, and the five v0.4 sections as steps, each
linking to the page that backs it (search, curation, branches, versions,
computed columns and materialized views, `StreamingDataset`). Wording
the docs can't back yet is left out: "zero-copy", "lineage", "cached
across clouds". Anchors renamed to match; nothing links to the old ones
|
| Fei: quickstart seems OSS-only | `quickstart.mdx` | Tip at the top:
for Enterprise only the connection changes, linking the Enterprise
connection in step 2 |
Fei's "list all Enterprise features as a catalog" comment is now covered
by #4732's feature comparison table (`/#compare`), so the separate table
this PR added to `basics/offerings.mdx` was dropped along with that
page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Updates the overview diagrams on the landing page. Rebased onto #4732,
which moved the architecture and offerings content into `index.mdx` and
drew it as Mermaid diagrams; this PR replaces those with the guide's
diagrams.
- **Architecture** (`#architecture`): the suite diagram
(`lancedb-suite.svg`) replaces the first Mermaid diagram. Its Enterprise
boxes read Search, Data exploration & curation, Feature engineering,
Training, matching the guide's product hierarchy.
- **LanceDB Embedded** (`#1-lancedb-oss`): adds the guide's Embedded
diagram. Its storage box reads "Local Disk or Object Storage", since
Embedded supports both.
- **LanceDB Enterprise** (`#2-lancedb-enterprise`): the guide's
Distributed diagram replaces the second Mermaid diagram. This also drops
that diagram's "column + MV refresh" label; automatic refresh is still
gated.
- **Functions** are still unpublished, so the "Functions ·" label is
removed from the Distributed diagram's jobs fleet. At release, restore
the designer's original image from the Notion guide.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
Fix the blob-column default-projection claim and add the previously
undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
chained Functions over blob outputs, GPU-produced blobs, Secrets,
multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
precise, verified catch-up semantics, and add the missing `ALTER
MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
(52 files); activate the existing availability-comparison mechanism on
`basics/offerings.mdx` for a full feature-by-feature comparison table.
Jack Ye
BHAWESHBHASKAR
lancedb-gatefixer[bot]
Hien Hoang
alexcastgonz
Yui Numazawa
SHAIK SHIVAJI
Prince Jain
sarankumar
importcpp
Bruno Ramirez
Joshua Snider
ryan-haynesandRyan Haynes
somil jain
Xuanwo
Colin Patrick McCabe
Wyatt Alt
Will JonesandClaude Opus 5.5
Brendan ClementandClaude Opus 5.5
