## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Fixes#4403
Implement remote drop_all_tables() support by listing tables and dropping them individually, including tables within namespaces. Handle paginated table listings and propagate errors appropriately. Add tests covering root and namespace tables, pagination, empty databases, and listing failures to ensure reliable remote database cleanup.
Rename the SQL-facing table functions:
- `vector_dedup` becomes `dedupe`.
- `vector_duplicate_pairs` becomes `duplicate_pairs`.
This is an intentional breaking rename without compatibility aliases. It updates the native table-function diagnostics and SQL coverage; the coordinated Sophon PR will update distributed binding and deployment integration.
BREAKING CHANGE: `vector_dedup` is now `dedupe`, and `vector_duplicate_pairs` is now `duplicate_pairs`; no compatibility aliases are provided.
The affected jobs were cancelled before runner assignment with GitHub's scheduled macOS 14 brownout annotation. macOS 14 runner images retire on November 2, 2026: https://github.com/actions/runner-images/issues/13518.
This moves the Python and CLI workflows to `macos-15` and keeps two distinct Rust macOS legs with `macos-15` and `macos-26`. All selected labels are public ARM64 runners, preserving the architecture of the previous plain `macos-14` jobs.
Validation: `git diff --check` and Ruby YAML parsing passed for all three changed workflows. A targeted `actionlint` check also passed; existing repository-wide custom-runner, reusable-workflow input, and shellcheck findings are outside this diff.
Package the standalone lancedb CLI with every stable and beta GitHub
release. Publish archives for Linux x86_64, Linux ARM64, and Apple
Silicon macOS, along with SHA-256 checksums, licenses, and third-party
notices.
Synchronize the CLI version with the SDKs and validate that release tags
match both manifests. Add PR and manual packaging dry runs, runtime
compatibility checks, and smoke tests of extracted executables. GitHub
release creation now waits for all three archives to pass validation;
SDK registry publishing remains independent.
Update installation instructions and the release checklist. Linux
binaries target glibc 2.28, retaining existing x86_64 CPU requirements;
macOS binaries target macOS 14 or newer.
Adds `image` to `@udf`, setting `runtime.environment.image` so a
Function can be built on a digest-pinned registry image instead of the
default base, for dependencies pip, conda and apt cannot express. The
client refuses references that are not pinned by digest; the server
validates the rest. The field is omitted when unset, so existing
Function identities are unchanged.
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
Stacked on #4674.
Remote Secret creation, rotation, description, listing, and drop now use the SQL backend. Writes bind the credential to `CREATE SECRET name AS $1` or `ALTER SECRET name AS $1`, keeping values out of SQL text. Description returns metadata; listing uses namespace-scoped continuation tokens.
The existing synchronous APIs submit a SQL query and wait internally through the same query-job adapter as function operations. The single query identity is retained internally; no registry job or extra query ID is introduced. Pollable shared results contain status or metadata and never credential values.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Removes the Codex Fix CI workflow. It ran an AI agent that read CI logs,
which outside contributors can write to, while the agent had the
repository's write token and no sandbox. Hardening it would take ongoing
work, and we don't use it enough to justify that, so we're removing it.
## Merge Risk
**Door:** two-way. The workflow can be restored from history.
**Blast Radius:** ci. Only affects manual dispatches of this workflow.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Remote function registration, exact-version lookup, listing, and drop now execute SQL through the Flight client. Registration waits for the durable image build and catalog publication before yielding its typed `FunctionVersion`. SQL also covers source-inclusive description and pinned metadata updates.
Registration and drop handles use `job_type="query"`; the single `job.id` is the SQL query UUID and can be inspected with `describe_query()` on the submitting connection. Waiting drains the SQL result once and caches completion for repeated waits. The drop boolean reports catalog removal; physical cleanup continues separately. Query handles have no registry history and cannot be reopened with `open_job()`.
Document vectorized `pyarrow.Array` UDFs for Blob v2 inputs and outputs.
Clarify annotation-selected array execution and the adapter-v1 client/executor
boundary. Add public regression coverage for registering, hydrating, executing,
publishing, and reading back Blob arrays, including large, empty, and null values.
Use `ThreadingHTTPServer` for the shared remote-test mock so concurrent `table.add` requests are not handled serially by the test server.
Validation:
- Reproduced the connection-reset/response-decoding failure against base `13749a3f138fde2647c8bdb8a1c05545026aee57`: 27 targeted runs passed, then the 28th matched the original CI failure.
- The patched target passed 100/100 runs; `test_remote_db.py` passed 261 tests with 2 skipped.
- Ruff 0.9.9 format and lint checks passed.
- Current-head GitHub CI: 19/19 check runs and 5/5 workflow runs succeeded.
## Summary
- Add a downstream IVF-HNSW-SQ merge regression using descending row IDs.
- Detect any reintroduction of merge-time row sorting without allocating the original multi-gigabyte fixture.
## Root cause
Lance 4 sorted each partition by row ID with `RecordBatch::take`. Arrow represented those take indices as `UInt32`, and `take_fixed_size_list` overflowed when `num_rows * vector_dimension` exceeded `u32::MAX`. The Lance version pinned by current `main` already includes lance-format/lance#6332, which removes that unnecessary sort. This test locks the corrected behavior at the LanceDB dependency boundary: the old implementation reorders the sentinel row IDs, while the fixed implementation preserves them.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb --lib table::create_index::tests::test_ivf_hnsw_sq_merge_preserves_shuffled_row_order -- --exact`
- Full current-head PR CI: 40 successful checks and one expected Windows ARM check skipped because that job runs only on `main`.
Fixes#3126
## Summary
- lock in the upstream fix that batches legacy row-ID fragment reservations into one coordinated commit
- exercise 32 indexed compaction tasks with 64 workers, exceeding the former 20-retry contention threshold
- verify data integrity, compaction metrics, and the expected two-commit transaction footprint
## Root cause
Parallel compaction used to reserve fragment IDs independently in every rewrite task. On high-core machines, those internal commits contended with one another and could exhaust the commit retry budget. The pinned Lance implementation now gathers the completed tasks and reserves all required fragment IDs once before the final rewrite commit; this regression test protects that behavior at the LanceDB integration boundary.
## Validation
- cargo fmt --all -- --check
- cargo test --quiet --features remote -p lancedb --lib table::optimize::tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples (passes with four pre-existing redundant_pub_crate warnings in remote-table code)
Fixes#1971
<!-- lance-gatekeeper-fix:v1 agent=18fd17ea143769e1da7c5aa5222ecdda generation=1 -->
## Summary
- Compare equal-size 1,000-value deletes confined to one BTree page and spread across many pages.
- Use an isolated current-thread allocation counter instead of a wall-clock threshold.
- Warm index pages first so the assertion detects repeated predicate materialization rather than I/O.
## Root cause
Lance previously rebuilt a large `IN` predicate for every BTree page, making indexed
lookups scale as `O(pages x values)`. The pinned Lance revision compiles the predicate
once and reuses it across pages. The regression guard observes the allocation scaling
caused by that invariant directly.
Fixes#2085
`rust/lancedb/src/io/object_store/io_tracking.rs` was a near-verbatim copy of lance-io's `utils::tracking_store`, and had drifted behind it. It was `#[cfg(test)]`-only with two usages, both in tests, so this PR points those at `lance_io::utils::tracking_store::IOTracker` and deletes the copy.
lance-io's `test-util` feature is enabled as a dev-dependency, which brings the `assert_io_eq!` / `assert_io_lt!` macros. Those print the recorded requests (method, path, byte range) when an assertion fails, instead of just the counter that didn't match, so the IO-counting tests in `table/dataset.rs` now say *which* IO happened when they fail.
`test_stats_does_not_read_data_files` no longer plumbs an `object_store_wrapper` at all: every lance-io `ObjectStore` now carries its own tracker, reachable as `store.io_tracker()`. This also makes the test stronger — the optimized local reader bypasses the `object_store` layer, so reads that would have gone uncounted through a wrapper are recorded by the built-in tracker.
Fixes#4076
## Not included
Two lance-side items from the issue are left for follow-up there, since neither can be fixed from this repo:
- `IOTracker::wrap_paginated` returns `Some(original)` on the premise that a pushed-down listing counts itself, which only holds for the local lister — a paginated listing against S3 is invisible to the tracker. The deleted copy had the identical bug, so this is not a regression.
- The deleted copy had poison-tolerant locking (`unwrap_or_else(|e| e.into_inner())`, from #3196) where lance still `.unwrap()`s. That tolerance, and the two unit tests covering it, go away with the file; it is worth carrying upstream instead of re-forking here.
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.
### Python experience
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```
For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.
### SQL experience
```sql
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```
CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.
Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
## Problem
The Python test extra pins `pylance==15.0.0b7` for every platform. On macOS, pip cannot resolve that pin: the configured `lance-format` index publishes b7 wheels for Linux and Windows but not macOS, while public PyPI returns HTTP 404 for the version and the `lancedb` index contains no 15.x release. The macOS Python dependency installation consequently cannot resolve the pin.
## Change
- Keep the exact `pylance==15.0.0b7` test pin on non-macOS platforms.
- Use `pylance==15.0.0b4` on macOS. This is the newest release in this range with a macOS wheel (`cp310-abi3-macosx_11_0_arm64`).
- Refresh `python/uv.lock` for the platform-specific resolutions.
The production Rust/Lance inputs are unchanged: `Cargo.lock` is byte-identical to `main` and remains on Lance `15.0.0-beta.7`. No tests, assertions, coverage, or CI checks are removed or weakened.
## Validation
- Confirmed the wheel identity from the configured index and matched its published SHA-256 to `uv.lock`; wheel metadata reports `Name: pylance`, `Version: 15.0.0b4`.
- Verified the b4 APIs used by the Python tests (`blob_field`, `blob_array`, `BlobType`, `Blob`, `dataset`, `write_dataset`, `FFILanceTableProvider`, and `DirectoryNamespace`). The Python test suite does not reference the b7-only `mark_sstables_as_compacted`, `mem_wal_dir_id`, `mem_wal_index_details`, `clustering_id`, `LanceFragment`, `IndexSegment`, `UpdateConfig`, or `max_source_fragments` changes.
- Verified scalar read/write interoperability in both directions between b4 and the beta.7 LanceDB native binding, plus beta.7 reads of a b4-written blob dataset with byte and null fidelity.
- `uv lock --check` with both configured Fury indexes and `unsafe-best-match`.
- CI dependency tests.
- Ruff 0.9.9 format and lint checks.
- Targeted Python tests: 911 passed, 22 skipped.
- Full Python suite (`-m "not slow and not s3_test"`): 2858 passed, 157 skipped, 70 deselected.
## Summary
Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.
**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.
**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.
**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.
**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Updates Rust workspace and Java Lance dependencies to
[v15.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.7),
and pins Python tests to the published `pylance==15.0.0b7` release with
refreshed Cargo and uv lockfiles. No compatibility fixes were required.
Validation: workspace Clippy with all features and warnings denied,
Cargo formatting, uv lock consistency, and all 18 CI-script tests pass;
repository-wide Ruff diagnostics are pre-existing.
---------
Co-authored-by: Lu Qiu <lu@lancedb.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add a LanceDB command line with the following functionality:
- `lancedb auth`: log in with OAuth, check cached OAuth session status,
or log out.
- `lancedb profile`: Create, edit, list, show, select, and delete
connection profiles.
- When no subcommand is given, we will attempt to create a SQL shell.
Update ci/check_cargo_lock.py and its regression tests to allow
pruning dependency edges while their target packages remain in
Cargo.lock. Removing the CLI from the Python source distribution
can disable optional features such as Axum tracing, even though
other SDK dependencies still require the tracing package.
## Summary
Replaces the generated availability table under **Compare features** on
the landing page with a hand-written list of features, grouped by area,
with separate Embedded and Enterprise columns.
The generated table listed every page that declares `availability`,
including overview and basic CRUD pages. Rows such as "Overview"
appeared three times without context, and pages like Insert, Update,
Database and the integrations showed up as if they were features.
| Area | Features |
| --- | --- |
| Search | Vector, full-text and hybrid search, filtering, reranking,
query plans, distributed search |
| Indexing | Vector, full-text and scalar indexes |
| SQL and curation | SQL, aggregation, sampling, joins, ranking and
deduplication, near-duplicate detection, chunking, changes since a
version, views |
| Feature engineering | Computed columns, materialized views, embedding
registry |
| Training | PyTorch data loader, permutations, distributed training |
| Data management | Branches, compaction and cleanup, background jobs,
Real Time engine |
| Security | Authentication, authorization |
Each ✅/❌ matches the linked page's `availability` on `main`; SQL, which
declares none, is Enterprise-only. Functions are left out until it's
settled whether they are published.
The same list, limited to 1.10 features, is in the Enterprise 1.10
editions in lancedb/docs#398.
Checked by assembling the docs site with this change: `mint validate`
passes and `mint broken-links --check-anchors --check-redirects` finds
no broken links. Vale reports no new warnings.
After this merges, the committed `lancedb` skill in lancedb/docs needs a
rebuild (`make assemble && make skill`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes#4733
PyArrow expects Blob V2 storage structs when constructing a record batch
and cannot infer the extension array from raw bytes. LanceDB already
converts binary values during ingestion, but batch writers had no
supported array constructor.
Add `lancedb.schema.blob_array(payloads)` for iterables of inline bytes
and nulls, using the same extension type resolver as `blob()`. The
helper preserves empty blobs and null blobs and works with both
pylance's type and LanceDB's fallback. Document the descriptor fields,
exclusive inline/external representations, batch construction, and
existing binary conversion in synchronous and asynchronous
`create_table` / `add` APIs.
Validation:
- Reproduced the reported `ArrowTypeError` on the original code; both
new regression cases fail against the original schema module.
- 256 focused blob tests and schema doctests passed; one optional PyAV
test skipped.
- Both new regression cases also passed with PyArrow 25, with and
without pylance imports.
- Python package Ruff checks and formatting passed.
- MkDocs build passed, with an existing unrelated `SecretReference`
cross-reference warning. Repository-wide Ruff reports 48 existing errors
in `docs/web-tests`.
<!-- lance-gatekeeper-fix:v1 agent=20d1a296d76ff5eb914c0848428a747a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
Changes links to home page sections from `/#<anchor>` to
`/index#<anchor>`.
The docs site's Assemble CI fails `mint broken-links --check-anchors` on
every `/#architecture` link since #4732 moved the Architecture page onto
the home page. The anchor exists; Mint 4.2.888's checker can't resolve
any anchor on the root page `/`, but resolves `/index#…`.
`docs.lancedb.com/index` serves the home page, so readers land in the
same place.
Verified by assembling the site with this change and the matching
Enterprise docs change: `mint validate` passes and `mint broken-links
--check-anchors --check-redirects` finds no broken links. The
`docs.json` redirects keep `/#architecture`, which the checker accepts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Addresses some comments on the staging branch. Rebased onto #4732, which
rebuilt `index.mdx` and the quickstart and reverted the landing copy to
the pre-v0.4 headings; the final copy is reapplied here.
| Comment | Page | Change |
| --- | --- | --- |
| Taylor: drop the comma | `index.mdx` | "one, unified platform" → "one
unified platform" |
| Taylor: "Preparing t…" + "Data doesn't slow down" | `index.mdx` |
"Training data and experimentation slow down…" → "Preparing training
data and running experiments slow down…" |
| Fei: subtitles should align with the new home page | `index.mdx` |
Follows the final homepage copy (v0.4): "Train on the right data faster"
with the hero subtitle, "Accelerate the data flywheel for model
research" with its intro, and the five v0.4 sections as steps, each
linking to the page that backs it (search, curation, branches, versions,
computed columns and materialized views, `StreamingDataset`). Wording
the docs can't back yet is left out: "zero-copy", "lineage", "cached
across clouds". Anchors renamed to match; nothing links to the old ones
|
| Fei: quickstart seems OSS-only | `quickstart.mdx` | Tip at the top:
for Enterprise only the connection changes, linking the Enterprise
connection in step 2 |
Fei's "list all Enterprise features as a catalog" comment is now covered
by #4732's feature comparison table (`/#compare`), so the separate table
this PR added to `basics/offerings.mdx` was dropped along with that
page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Updates the overview diagrams on the landing page. Rebased onto #4732,
which moved the architecture and offerings content into `index.mdx` and
drew it as Mermaid diagrams; this PR replaces those with the guide's
diagrams.
- **Architecture** (`#architecture`): the suite diagram
(`lancedb-suite.svg`) replaces the first Mermaid diagram. Its Enterprise
boxes read Search, Data exploration & curation, Feature engineering,
Training, matching the guide's product hierarchy.
- **LanceDB Embedded** (`#1-lancedb-oss`): adds the guide's Embedded
diagram. Its storage box reads "Local Disk or Object Storage", since
Embedded supports both.
- **LanceDB Enterprise** (`#2-lancedb-enterprise`): the guide's
Distributed diagram replaces the second Mermaid diagram. This also drops
that diagram's "column + MV refresh" label; automatic refresh is still
gated.
- **Functions** are still unpublished, so the "Functions ·" label is
removed from the Distributed diagram's jobs fleet. At release, restore
the designer's original image from the Notion guide.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
Fix the blob-column default-projection claim and add the previously
undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
chained Functions over blob outputs, GPU-produced blobs, Secrets,
multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
precise, verified catch-up semantics, and add the missing `ALTER
MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
(52 files); activate the existing availability-comparison mechanism on
`basics/offerings.mdx` for a full feature-by-feature comparison table.
Split the Java 17 workflow into a matrix for Java 11 and Java 17.
Run Checkstyle once in a dedicated Java 17 job and skip the lifecycle-bound Checkstyle execution in the matrix builds to avoid redundant work.
Closes#1331.
Adds a Python `nvidia` embedding provider for Nemotron 3.5 Embed 1B and 8B. It works with NVIDIA's hosted API and self-hosted NIM `/v1` endpoints, and uses the existing `openai` client. The OpenAI provider can't send NVIDIA's passage/query input types or pick Nemotron's dimensions.
- Documents are embedded as `passage`, searches as `query`.
- Default dimensions are 2048 (1B) and 4096 (8B), and `dim` is checked against each model's supported range. Self-hosted model IDs need an explicit `dim`.
- Response count, order, dimensions and values are checked. Null and empty text get a null vector without a request.
- Each request is retried on its own with LanceDB's retry policy. Invalid input or responses fail immediately.
- `NVIDIA_API_KEY` is only sent to the hosted API. Custom endpoints need an explicit `api_key` (a `$var:` reference), and endpoints restored from table metadata must be allowed with `allow_endpoint`.
- Request dimensions are sent through `extra_body`, preserving the NVIDIA wire contract while supporting the declared minimum `openai==1.6.1`.
Python and text only. Docs are in `integrations/embedding/nvidia`.
Originally authored by Taylor (@tsmithv11).
### Testing
- `test_nvidia_embeddings.py`: 45 passed, 2 deselected with both the locked `openai==2.44.0` and minimum `openai==1.6.1`. These use the real OpenAI client with mock HTTP transport and real local tables.
- Embedding, registry, field-metadata and trust suites: 163 passed, 4 skipped, 10 deselected.
- Historical live experiments: hosted 8B passed a five-request smoke test; hosted 1B returned HTTP 500. No live request was made in this stage, so current hosted 1B availability was not verified. Enterprise coverage uses the repository mock HTTP server, not a live deployment.
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns.
The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry.
Closes#4652.
Freeze the SDK product baseline on a release branch before validating a
Lance RC, so later main commits do not change the candidate. Require the
RC pin PR's CI and applicable reviews to complete with no blocking
findings before Lance stable publication. Draft-skipped reviews are
missing evidence, and green CI does not clear a request-changes
recommendation.
After the upstream release, update only the validated dependency
references, rerun checks and refresh review, then publish from the
verified release branch. Document Cargo override boundaries, exact
commit/artifact evidence, and the workflow's actual stable input
(`type=stable`).
Validation: Markdown fences, public-scope and diff checks passed; the
documented workflow inputs were checked against the repository workflow.
Support `time` and `timedelta` values in `table.update` by generating explicit Arrow casts.
- Preserve Arrow duration/time units and pandas nanosecond precision.
- Reject timezone-aware `time` values before data can be changed.
- Cover time64 us/ns, duration s/us/ns, negative timedeltas, filters, and nanosecond round trips.
Adds `apt` to `@udf` and `system_packages` to the Rust `PythonEnvironmentSpec`. The server adds the packages to the Function's image before installing its Python environment, for shared libraries that pip and conda cannot provide, such as FFmpeg's for torchcodec. The field is omitted when empty, so existing Function identities are unchanged.
Closes#1605.
Adds `EmbeddingFunctionRegistry.list_functions()` to expose registered lookup names in registration order without accessing `_functions`. The method returns an independent snapshot, and every returned name can be passed to `get()`.
Tests cover empty registries, aliases and class names, registration order, snapshot isolation, and reset.
The Node SDK's vector index options only take `numPartitions`. The Rust index builders (`target_partition_size` in `rust/lancedb/src/index/vector.rs`) and the Python SDK already support a target partition size, so this adds `targetPartitionSize` to `IvfPqOptions`, `IvfRqOptions`, `IvfFlatOptions`, `HnswPqOptions` and `HnswSqOptions` and passes it to their builders. When both are set, `numPartitions` is used, as it is for Rust and Python callers.
The reason to want it: an index built from a partition count records no target partition size, so `table.optimize()` splits and joins it against the index type's default and can merge small partitions down to one or two. Building with a target records it in the index details, and optimize keeps to it. https://github.com/lance-format/lance/pull/10154 fixes this on the Lance side for indexes built from a partition count; this lets Node users set the size directly.
The new tests build IvfPq, IvfFlat, IvfRq, IvfHnswPq and IvfHnswSq indexes with `targetPartitionSize` and check the size recorded in the index details, and check that `numPartitions` wins when both are set. Each option's doc comment states that index type's default target. Docs regenerated with `pnpm run docs`.
Map dynamic projections by output name to their source columns in the remote
torch test mock so permutation take queries preserve the aliased ordering
column. Run the regression in the embeddings-installed Python test job and
require torch to import so a missing dependency cannot skip coverage.
The Python test jobs spend much of their pytest time waiting on the in-process mock HTTP servers.
1. Each server ran `serve_forever()` with the default 0.5 s poll interval, so every `server.shutdown()` blocked for up to 0.5 s. Around 300 tests use such a server (test_remote_db, test_authz/test_catalog, test_listing, the permutation helpers), about 150 s per test job on every platform. A shared `start_server_thread()` helper in `tests/utils.py` serves with a 0.01 s poll interval, so shutdown returns in about 10 ms.
2. The servers bind IPv4 only, but clients used `http://localhost:<port>`. The client tries ::1 first and falls back after the happy-eyeballs delay, about 0.3 s per connection on Windows. The mock handlers close the connection after each response, so one listing test took 9.5 s on Windows CI versus 0.6 s on Linux. Clients now use 127.0.0.1, as test_catalog, test_listing and test_namespace already do.
Measured on Windows with the 0.41.0b6 wheel and `-m "not slow and not s3_test"`: full suite 546.7 s to 212.3 s; test_remote_db.py, test_listing.py and test_catalog.py together 284 s to 14 s on current main. Collected test ids are identical (1898) and pass/fail/skip results match. No assertion, test or marker changed.
## Summary
- cover vector-index prewarming through the Python API using the logical
index name returned by list_indices
- prevent regression to interpreting a vector index name as a schema
column
## Root cause
The Lance version used by the reported release opened a vector index by
treating the prewarm argument as a column before resolving the index
metadata. Current LanceDB dependencies contain the upstream
metadata-based resolution, but LanceDB only exercised prewarming for FTS
indexes, leaving the vector behavior unguarded.
## Validation
- uv run --extra tests pytest
python/tests/test_index.py::test_create_vector_index -q
- python/.venv/bin/ruff format --check python/python/tests/test_index.py
- python/.venv/bin/ruff check .
Fixes#2897
<!-- lance-gatekeeper-fix:v1 agent=c2f8273221061d8e2b6f7393066e68c6
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Closes#3693.
`columns` is documented on `StreamingDataset` now, but `connection_factory` still wasn't. This adds it to the parameter list, in signature order.
The text follows `__getstate__` / `__setstate__`: with a factory, pickling the dataset replaces the table's reopen state with its name, and each DataLoader worker calls the factory to reopen the table. The factory is pickled along with the rest of the dataset, so it has to be picklable, and credentials stay out of the pickle only when the factory loads them in the worker instead of capturing them (the same guidance as `Permutation.with_connection_factory`). Without a factory, the table's own reopen state is pickled.
This picks up #3697, which documented the same parameter and was closed by its author without a merge.
Binding a nullable column needed an explicit input schema, and that
forced spelling out the output schema too. Either schema can now be
given alone; the other side is inferred from annotations.
The udf docstring now says a bare annotation is non-nullable and
`Optional[T]` binds a nullable column, and that a `bytes` input counts
against the per-row input limit, with Blob V2 and `BlobReader` for large
values.
## Summary
- explain that Rust callers should use the Arrow crates re-exported by `lancedb::arrow`
- update the crate quick-start doctests to build Arrow values from those re-exports
- update the checked-in `simple` example so it cannot drift to an incompatible Arrow crate version
Fixes#1207.
## Validation
- `cargo fmt --all -- --check`
- `cargo metadata --no-deps --format-version 1 --quiet`
- static checks that the documented crates are publicly re-exported and both examples use them
- `git diff --check`
- attempted `cargo check --quiet -p lancedb --example simple`; dependency compilation stopped in the unmodified `lance-encoding` build script because `protoc` is not installed in this environment
Function authors annotate inputs with lancedb.BlobReader, but the
packaged source binds that annotation to the worker's
lance_blob.BlobReader, so grepping the artifact for the client class
finds nothing. Say so on the class.
`@udf` documented batch annotations without explaining that `pa.Array`
and `pa.RecordBatch` do not specify concrete Arrow element types or
field schemas. Document the requirement for both explicit schemas
(including class `__call__` methods), add an array example, and make
inference errors point callers to `input_schema` and `output_schema`.
Fixes#4721
Validation:
- Five new regression cases pass; all four diagnostic cases fail with
the original implementation.
- Function packaging, first-class Function slice 1/2 tests, and UDF
doctests: 168 passed, 3 failed. This includes the upstream BlobReader
annotation coverage. The three existing packaging tests fail while
importing temporary modules, before UDF registration; all three failures
also reproduce with the unchanged source and tests at `057a39a1`.
- Rebuilt the native extension in the uv-managed tests/dev environment.
- Python lint and formatting checks pass (`ruff check python`, `ruff
format --check python`); `cargo fmt --all` also passes without changes.
- Repository-wide `ruff check .` reports 48 existing errors under
`docs/web-tests`.
<!-- lance-gatekeeper-fix:v1 agent=2c6567ee3f128d3a0c3b9c68e36eca38
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote Python connections inherited the unsupported-operation stubs for
`namespace_exists` and `table_exists`, so both raised
`NotImplementedError` even when namespace and table operations worked.
Add the missing remote overrides, async methods, and PyO3 bindings using
the existing namespace describe and table open paths. Only an explicit
not-found result returns `False`; permission, transport, and server
errors propagate. This preserves authentication, dynamic headers,
retries, and nested namespace handling, and rechecks cached tables
before reporting that they exist.
The async namespace helper also recognizes native `NamespaceNotFound`
errors so missing local namespaces return `False`. The table and
namespace guides document Python remote support and distinguish it from
the unsupported `exists` routes used by direct REST namespace clients.
Validation:
- All 40 focused regression cases pass, covering successful and missing
lookups, local and remote namespaces, dynamic headers, permission/server
errors, invalid table identifiers, and cached tables that disappear.
- The original inherited stubs reproduce the reported
`NotImplementedError`; the local namespace regression reproduced the
native missing-namespace `RuntimeError` before its fix.
- Full remote connection suite: 255 passed. Local namespace tests: 14
passed.
- `cargo clippy --quiet -p lancedb-python --features remote --lib`,
`cargo fmt --all`, Python Ruff lint, and Python formatting checks pass.
- Root `ruff check .` reports 48 existing errors in documentation
examples under `docs/web-tests`; the Python subtree passes and those
unrelated files are unchanged.
Fixes#4722
<!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Function descriptions include `source_digest`, but both the Rust
response model and Python `FunctionVersion` discarded it. Clients
therefore could not compare a catalog entry with
`udf_def.registration_request.artifact.digest`.
Preserve the digest through Rust serialization and expose it on Python
`FunctionVersion`, with a documented Rust accessor. Missing or null
digests decode as `None`, keeping older responses compatible. Regression
tests cover serialization, immutability, registration jobs, get, and
synchronous/asynchronous list responses while keeping the image digest
independent.
Validation:
- Confirmed that the original Python model silently discarded a
server-provided source digest.
- Rebuilt the Python extension in the uv-managed environment; both
focused Function test files passed (106 tests).
- Focused Rust Function model tests passed (10 tests).
- Full workspace `cargo check --quiet --features remote --tests
--examples` and `cargo clippy --quiet --features remote --tests
--examples` passed.
- Rust formatting and Ruff checks/formatting for changed Python files
passed.
- Repository-wide Ruff reports 138 existing violations in unchanged
docs/CI files.
Fixes#4723
<!-- lance-gatekeeper-fix:v1 agent=c145f25136f7b908217e04bde8a2548b
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Reading a Blob v2 table in a fresh process could return its storage
struct if neither `blob()` nor pylance had registered the extension.
Both local and remote schema reads were affected, so `field.type` could
depend on import order.
Restore metadata-only blob fields on every table schema read through the
existing BlobType resolver and Arrow's schema deserializer. Preserve
nested fields, nullability, custom metadata, and both supported storage
layouts. The fallback deserializer now honors the stored shape too.
Fresh-process regressions cover local and remote tables, sync/async
reads, and operation with and without pylance. The four native cases
failed before removing the native-table guard. Existing merge-insert
tests now require the restored extension type while continuing to verify
updated and inserted blob contents.
Validation:
- All 12 fresh-process local/remote schema regression cases pass.
- `uv run --extra tests pytest python/tests/test_table.py
python/tests/test_blob.py python/tests/test_namespace.py
python/tests/test_util.py -q -m 'not slow and not s3_test'`: 835 passed,
11 skipped.
- Python package Ruff lint and formatting checks pass.
- Repository-wide `ruff check .` reports 48 pre-existing diagnostics in
unchanged `docs/web-tests` files.
Fixes#4719
<!-- lance-gatekeeper-fix:v1 agent=67c656bb5889e698efce5f2fd9d58ea7
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Function names containing periods or hyphens passed `@udf` validation
and failed only when registered with the server. Restrict the name
pattern to ASCII letters, digits, and underscores, preserving the
existing first-character rule so invalid names fail when the decorator
runs.
Document the accepted format and add regression coverage for dotted and
hyphenated names, including trailing punctuation, alongside valid name
overrides.
Fixes#4720
Validation:
- The four new invalid-name cases failed before the validator change
because no `ValueError` was raised.
- `cd python && uv run --extra tests pytest
python/tests/test_first_class_function_slice2.py -q`: 86 passed.
- Ruff 0.9.9: `ruff check python` and `ruff format --check python`
passed.
- Repository-root `ruff check .` reports 48 pre-existing errors in
`docs/web-tests/`.
<!-- lance-gatekeeper-fix:v1 agent=5e2441d3d264b685a6cee3c3c21ab048
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Packaged UDFs omit globals used only in postponed annotations, so a
`BlobReader` input has a correct wire signature but
`typing.get_type_hints()` on the loaded function raises `NameError`.
Use the resolved input signature to bind reader annotations to an
imported `lance_blob.BlobReader` alias. Preserve signature nullability
and avoid collisions with existing globals. Regression tests cover
functions and callable classes, eager and postponed annotations,
aliases, forward references, qualified names, and body globals.
Validation:
- Confirmed the new regression fails with `NameError: name 'BlobReader'
is not defined` against the original packager at
`057a39a1c70e15c78461bb7e3db8cef4c067cd3e`.
- The packaging, first-class Function slice 1 and 2, and BlobReader
suites pass all 159 tests with Python file-import directory caches
refreshed before lookups. The standard run passes 156 tests; three
temporary-module import failures reproduce unchanged on the original
packager because directory timestamps do not advance between file writes
in this environment. All 37 new regression cases pass in the standard
run.
- Python Ruff lint and formatting checks pass. Repository-wide Ruff lint
reports 48 existing errors in documentation examples.
Fixes#4718
<!-- lance-gatekeeper-fix:v1 agent=a32bb6efa79217b3c1c5c9d7a7ff9928
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
ryan-haynesandRyan Haynes
somil jain
Jack Ye
Xuanwo
Colin Patrick McCabe
Wyatt Alt
Will JonesandClaude Opus 5.5
lancedb-gatefixer[bot]
Brendan ClementandClaude Opus 5.5
alexcastgonz
Divyanshu Singh
Taylor
BHAWESHBHASKAR
Yang Cen
Jay Pokale
Ishita Singh
geoffgscott
Lei Xu
max
Bilal Atique
medimedi