mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
dc6a08a25f4c8065af7c11bc534ea53bbfc5c9d1
304
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yang Cen
|
2dedde9b6c |
feat(python)!: expose typed query errors and request metadata (#4832)
Remote SQL failures currently become generic runtime errors, and HTTP table lookup can discard server error metadata. Preserve transport status, current request ID, SQL job ID and worker request ID across Rust and Python. Select typed column, table, reference and schema exceptions only when an explicit server code matches the transport status. Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed SQL jobs retain `JobFailedError` with a typed SQL cause when available. Missing metadata stays absent; message text never determines the exception class. Local missing-column errors retain their type and qualified-field suggestions, and `JobFailure.source` documents structured remote Flight SQL failures. This branch integrates main at `a9ad4997`, including its released Lance 15.0.0-beta.9 dependency update and remote optimize support. The only textual conflict was two independent additions to `test_remote_db.py`; both test groups and all original assertions are retained. The feature has no dependency or lockfile delta against current main. SDK runtime validation on `03ee633a` (runtime source unchanged in `f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python 3.14 environment: - Rebuilt the native extension from the integrated source. All 22 targeted Python tests passed: native local errors, HTTP/Flight metadata, legacy sync/async table-not-found behavior, and the three main remote-optimize regressions. - All 52 Rust SQL, metadata/source and nested-field tests passed, including the new main optimize cases. - Root and Python-binding Rust Clippy with `-D warnings`, Rust formatting, Python package Ruff and conflict-file formatting passed. - Repository-wide Ruff reports 59 existing documentation findings, and its format check identifies six documentation files. Every affected file is byte-identical to current main; these checks are not reported as passing. - Direct integration review and AST comparison confirm both conflicting test groups were preserved. Earlier exception-constructor/pickling evidence remains applicable to the unchanged Python error modules. The unlocked build exhausted disk again on a runner with the same label but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing runner and dependency caches, serialize Cargo compilation, and remove only completed test/bench executables reported by Cargo. This job never runs those binaries; normal binaries, libraries and build-script outputs remain available. Fresh resolution, the ci profile, all features, and test/bench compilation and linking are retained. Three probes using Cargo and target-path validation pass with `-D warnings`: all fixture targets link while ordinary outputs remain usable; an undefined symbol still produces Cargo exit 101 and its linker diagnostic; files outside the target directory are never removed. The [current-head build-no-lock job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065) passed on `f3f2d0b8`: all compilation/linking completed, 20 completed test executables were removed, and `target` retained about 7.3 GiB. This successful run had a 145 GiB root with about 78 GiB free afterward; it was not an identical rerun of the failed 72 GiB image. The driver tests also passed in this job. All previous-head jobs ended naturally before the push; new-head CI and review gates still apply. Typed remote input exceptions require the server metadata contract. Older servers retain generic transport errors and existing missing-table compatibility. The optimize tests use mock servers and do not establish downstream server acceptance. BREAKING CHANGE: Rust callers with exhaustive matches on `lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants. Local `Error::ColumnNotFound` failures now raise `lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass. Python callers that previously caught only `RuntimeError` for this local failure must catch `ColumnNotFoundError` or `ValueError`. |
||
 Jack Ye
|
82cc4e82a1 |
fix: tag the final release commit after syncing lockfiles (#4825)
Release tags currently point to the version bump before the lockfile amend, so tagged CLI builds fail with `--locked` even though the branch has the synced lockfile. Keep the newly created, unpublished tag on the final amended release commit before pushing. Existing published tags and packages remain unchanged. [Remote preview dry run](https://github.com/lancedb/lancedb/actions/runs/41764901500) passed and moved the local tag after the lockfile amend; no tag was pushed. |
||
 Yang CenandXuanwo
|
10625f6dda |
chore: update lance dependency to v15.0.0-beta.9 (#4814)
Update Rust Lance dependencies to [v15.0.0-beta.9](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.9), including the scoped stable-row-ID conversion fix. Python tests use published beta.9 wheels on Linux/Windows while macOS retains beta.4. Java stays on beta.8, matching the original base, so this update does not depend on beta.9 Maven publication or macOS Java packaging. The dependency updater preserves workspace inheritance, the WAL test wrapper uses the new StoredBatch module, and Linux/remote Rust CI uses its installed mold linker. The Python lockfile also records the already-declared OCI extra. CLI CI now caches Cargo downloads under a separate key instead of restoring compiled targets: the previous runner exhausted its disk during cache restoration, before compilation. Its test matrix and commands are unchanged. This branch includes #4820's row-freshness changes: successful rows become current while skipped rows retain their values and remain eligible for retry. Repair identities, replay handling and Restore fencing protect publication completion. BREAKING CHANGE: `StalenessPlan::row_dirty` is a new public field. Rust callers using exhaustive struct literals must initialize it. Validation: - All reported non-Java checks passed on `f5fa66fcbcf31a8eb188935f728735560c77c873`, including Rust lint/Linux/remote, Python, NodeJS and CLI checks. Current head `c660bc0ac9fd5920ad4b64be9c19f732ef4c56a5` preserves those product sources, manifests, lockfiles and compiler/test commands; only the Java pin restoration and CLI cache configuration differ. Current-head CI is tracked separately. - The complete Java directory is identical to the original PR base. The scope correction reuses [Xuanwo's prepared and validated commit](https://github.com/lancedb/lancedb/pull/4814#issuecomment-6626088466); no new Java build was run. - CLI workflow YAML parses successfully, its inputs match the pinned rust-cache action schema, and the existing jobs, matrix and commands are unchanged. Hosted execution must confirm the cache-restoration repair. - Existing validation includes the updater regressions and 32 CI-script tests, dependency-policy checks, count/WAL coverage, and #4820's 42 freshness tests, strict core Clippy and Rust doctest. Root Ruff previously reported unchanged documentation-example issues; Python-package Ruff passed. This PR adopts the existing released Lance beta.9. A subsequent official SDK release and normal Sophon dependency adoption remain separate delivery steps. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Jack YeandClaude Sonnet 5
|
de95ed3f6d |
docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
  importcppandimportcpp
|
8172298044 |
test(python): expand minimal dependency coverage
Expand the Linux Python test run without pandas and pylance from two files to the suite while retaining the regular dependency run for optional-package modules. Move optional imports into tests that need them and use Arrow assertions where pandas is not required, preserving the existing behavioral assertions. Validation: - Physical no-pandas/no-pylance broad run: 2884 passed, 316 skipped, 8 existing marker cases deselected. - Regular affected suite: 1687 passed, 39 skipped. - Current-head hosted integration: all 15 checks and 4 workflows passed. Co-authored-by: importcpp <57519936+importcpp@users.noreply.github.com> |
||
|
Xuanwo
|
2d70bec53a |
feat: add UTM-tagged docs links to SDK warnings and enterprise-only errors (#4797)
Add a central documentation link registry (`lancedb::docs::DocLink`) in the Rust core that builds `docs.lancedb.com` URLs with UTM parameters (`utm_source=lancedb_sdk`, `utm_medium=warning|error`, `utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any `#anchor` fragment and without user-identifying data. - Initialize `utm_content` per language binding (`python`, `nodejs`, `cli`, defaulting to `rust`) and expose the link builder to Python so `DBConnection` and `Job` stubs reuse the core definitions. - Append published documentation links to Enterprise-only `NotSupported` / `NotImplementedError` errors across database, job, catalog, and table operations. - Emit rate-limited maintenance warnings (once per table per process when `warn` logging is enabled) when opening a native table with at least 100 small fragments (`< 100k` physical rows each) or at least 1,000,000 unindexed rows on an optimizable user index, reading only manifest metadata without opening data or `_indices/` files, and document the thresholds in the compaction and reindexing guides. - Verify published/in-repo `docs/web/` targets and `#anchor` fragments in unit tests and include extracted SDK links in the scheduled `docs-link-check` workflow. Linear: OSS-2772 |
||
|
Xuanwo
|
5ca8d836e7 |
fix(deps)!: upgrade candle to remove vulnerable zip
## Problem `RUSTSEC-2026-0429` affects `zip 7.4.0`, which reaches the optional `sentence-transformers` feature through `candle-core 0.9.2`. The `deny` check on `main` fails on this advisory. ## Change Update `candle-core`, `candle-nn`, and `candle-transformers` to `0.11.1`. The focused lockfile resolution replaces `zip 7.4.0` with `zip 9.2.2`, and no other zip version remains in the all-features graph. This also pulls candle's required newer transitive `tokenizers` and `safetensors` releases; the directly declared `tokenizers 0.19.1` and the embedding implementation remain unchanged. The optional `sentence-transformers` Rust API exposes this dependency version: `SentenceTransformersEmbeddingsBuilder::device` accepts `D: Into<candle_core::Device>`, and the `Device` type is now from candle-core 0.11. The default CPU path constructs that device internally and is unchanged. `deny.toml` is unchanged. Its obsolete zip ignores now produce `advisory-not-detected` warnings, which do not fail the check. ## Validation - The `sentence-transformers` feature builds with tests and examples. - The existing CPU example loads the default model and retrieves `The human body has 206 bones.` for the matching query. - A local contract check verified 384-dimensional, finite, non-zero Float32 output and higher similarity for the matching fact than for an unrelated fact. It was not committed because it would add an 88 MB model download to all-features CI. - `cargo deny` `0.20.2`, the version used by the Rust workflow, passes `advisories`, `bans`, `licenses`, and `sources`. - The locally installed `cargo-deny 0.18.9` is not green: it treats informational unsound advisories differently from the workflow's `0.20.2`. The result above uses the exact CI version. - Targeted clippy, formatting, lockfile review, and the reverse `zip` dependency tree pass. - Hosted CI completed successfully for exact head `bffbcda6490fa1ed51838b3801e01ae2f82faed6`. CI mitigation: Reclaim completed lancedb CI-profile artifacts before the Blob v2 example after disk sampling showed a minimum of 24,764,416 available bytes. This may rebuild lancedb for the example; the exact OS cause of the SIGBUS remains unproven. BREAKING CHANGE: The optional `sentence-transformers` Rust API now takes `candle_core::Device` from candle-core 0.11. Downstream crates that explicitly construct and pass a candle-core 0.9 `Device` to `SentenceTransformersEmbeddingsBuilder::device` must use a `Device` from candle-core 0.11.x aligned with lancedb. This normally means updating a direct candle-core dependency; if the value comes through a re-export, update the dependency that supplies it. Callers that use the default CPU path without passing a `Device` do not need to migrate. |
||
  TaylorandBrendan Clement
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
|
Xuanwo
|
6063229de9 |
fix(ci): replace retiring macos-14 runners
The affected jobs were cancelled before runner assignment with GitHub's scheduled macOS 14 brownout annotation. macOS 14 runner images retire on November 2, 2026: https://github.com/actions/runner-images/issues/13518. This moves the Python and CLI workflows to `macos-15` and keeps two distinct Rust macOS legs with `macos-15` and `macos-26`. All selected labels are public ARM64 runners, preserving the architecture of the previous plain `macos-14` jobs. Validation: `git diff --check` and Ruby YAML parsing passed for all three changed workflows. A targeted `actionlint` check also passed; existing repository-wide custom-runner, reusable-workflow input, and shellcheck findings are outside this diff. |
||
 Colin Patrick McCabe
|
2044179a8b |
feat: add a CLI release binary (#4751)
Package the standalone lancedb CLI with every stable and beta GitHub release. Publish archives for Linux x86_64, Linux ARM64, and Apple Silicon macOS, along with SHA-256 checksums, licenses, and third-party notices. Synchronize the CLI version with the SDKs and validate that release tags match both manifests. Add PR and manual packaging dry runs, runtime compatibility checks, and smoke tests of extracted executables. GitHub release creation now waits for all three archives to pass validation; SDK registry publishing remains independent. Update installation instructions and the release checklist. Linux binaries target glibc 2.28, retaining existing x86_64 CPU requirements; macOS binaries target macOS 14 or newer. |
||
 Will JonesandClaude Opus 5.5
|
87057b6616 |
ci: remove Codex Fix CI workflow (#4630)
## Summary Removes the Codex Fix CI workflow. It ran an AI agent that read CI logs, which outside contributors can write to, while the agent had the repository's write token and no sandbox. Hardening it would take ongoing work, and we don't use it enough to justify that, so we're removing it. ## Merge Risk **Door:** two-way. The workflow can be restored from history. **Blast Radius:** ci. Only affects manual dispatches of this workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 
|
d5013459eb |
chore: update lance dependency to v15.0.0-beta.6 (#4742)
Updates Rust and Java Lance dependencies to [v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6), pins Python tests to published pylance 15.0.0b6, and refreshes both lockfiles. Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans and MemWAL directory handling, updates the ABI metadata test, and removes the obsolete Lance Arrow patch now fixed upstream. Validation: workspace clippy with all features/tests and warnings denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script tests, and targeted Python regressions (54 passed, 28 skipped); repository-wide Ruff retains existing documentation-example failures, and the upstream macOS wheel build failed while Linux and Windows wheels are published. --------- Co-authored-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colin Patrick McCabe
|
301ccad3aa |
feat: add a LanceDB command line (#4600)
Add a LanceDB command line with the following functionality: - `lancedb auth`: log in with OAuth, check cached OAuth session status, or log out. - `lancedb profile`: Create, edit, list, show, select, and delete connection profiles. - When no subcommand is given, we will attempt to create a SQL shell. Update ci/check_cargo_lock.py and its regression tests to allow pruning dependency edges while their target packages remain in Cargo.lock. Removing the CLI from the Python source distribution can disable optional features such as Axum tracing, even though other SDK dependencies still require the tracing package. |
||
 Divyanshu Singh
|
005a9a353a |
ci(java): run Java 11 and 17 builds in parallel with matrix strategy
Split the Java 17 workflow into a matrix for Java 11 and Java 17. Run Checkstyle once in a dedicated Java 17 job and skip the lifecycle-bound Checkstyle execution in the matrix builds to avoid redundant work. Closes #1331. |
||
 Lei Xu
|
03bd909dab |
test(python): resolve aliased projections in the torch remote mock server
Map dynamic projections by output name to their source columns in the remote torch test mock so permutation take queries preserve the aliased ordering column. Run the regression in the embeddings-installed Python test job and require torch to import so a missing dependency cannot skip coverage. |
||
|
Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan Clement
|
4359688f32 |
ci: let the Vale action install its pinned CLI (#4708)
Let the pinned Vale action install Vale 3.24.0 instead of maintaining a separate download and extraction step |
||
|
Brendan Clement
|
149cf1052a |
fix(ci): bundle Vale rules to avoid private release downloads (#4707)
The prose check fails because it downloads its Vale rules from the now-private docs repository. Keep the five unchanged rules from `vale-lancedb-v0.1.0` in this repository and disable `vale sync` in CI. This also works for contributors without access to the docs repository. The update instructions record the release and archive checksum. |
||
|
Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
 Jack YeandLance Release
|
197403e891 |
fix: honor downstream Lance dependency overrides (#4671)
Consumers that provide a local Lance checkout cannot resolve the dependency graph because the SDK directly selects a vendored `lance-arrow`, producing two path packages with the same name and version. The Rust core now declares its upstream Git dependency separately so the consuming workspace can override that source consistently. Python continues inheriting the vendored workspace dependency, keeping the sliced-struct validity fix in Maturin source distributions. Standalone SDK builds still select that same fix through the existing root patch. Adds a downstream consumer CI regression that resolves a local Lance override and checks that exactly one canonical `lance-arrow` package is selected. The Lance version updater also updates the core declaration in stable, preview and local modes. Public APIs and query behavior are unchanged. The workspace declaration lint excludes path dependencies because the Python binding intentionally uses the bundled source. Duplicate and unused external declarations remain errors; advisory, license, banned-package and source checks continue covering the full dependency graph. --------- Co-authored-by: Lance Release <lance-dev@lancedb.com> |
||
 KarthikSandXuanwo
|
702389654e |
fix(python): align pylance extra and preview install indexes (#4556)
Fixes #4248. The preview index still contains pylance 1.0.0b6. uv's default first-index resolution can select it, or backtrack to an old LanceDB release when installing the extra. Meanwhile, the extra still permits pylance 5 even though current main bundles Lance 14. Raise the optional dependency to `pylance>=14.0.0b12`, the version already pinned in the test extra, and keep the lockfile metadata consistent. Document both Fury indexes and uv's `unsafe-best-match` strategy in the Python README and the source-owned quickstart (`docs/web/quickstart.mdx`). Explain the index trust requirement and recommend installing the extra to enforce the version floor. Add CI checks that reject older Lance majors and ensure the pylance and tests extras remain compatible. Run these checks when either dependency manifest changes, so a future Lance major bump cannot silently leave the Python minimum behind. ### Validation - Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects `pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and the unexpected `blob_handling` scanner argument using a real blob table. - Resolved the modified `python/pyproject.toml` with the pylance extra against both indexes: selects `pylance==14.0.0b18`, matching the current Rust pin. - Ran both documented pip and uv installations in isolated environments: `lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both package dependency checks pass. - New regression fails on the original dependency bound (27 rejected-version cases incorrectly accepted), then passes with the fix. Full CI-script suite: 3 tests pass. - `uv lock --check` with both indexes passes; focused Ruff checks pass; `git diff --check` passes. - Repository-wide Ruff reports 49 existing errors in untouched `docs/web-tests` files. Unrelated formatting changes were discarded. Runtime smoke checks used released native wheels; source dependency resolution was tested directly from the modified pyproject. No Rust or Python runtime code changes. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
5a79c7d22d |
ci: run aarch64 Windows tests on main only (#4637)
Run the aarch64 Windows build/test cycle only on pushes to main so PRs wait for a single Windows platform (x86_64). Main keeps the same native aarch64 coverage as before. GitHub evaluates job-level `if` before the matrix strategy is applied, so the aarch64 target is split into its own `windows-arm` job instead of being filtered inside the existing matrix. The job keeps the historical check name. |
||
|
Xuanwo
|
444f7f3bde |
ci: refresh apt lists before installing protobuf in docs build (#4636)
The docs deploy workflow installs protobuf-compiler without refreshing apt lists first. Ubuntu noble-updates replaced `protobuf 3.21.12-8.2ubuntu0.3` with `.0.4`, so stale package lists in older runner images make `apt install` fail with 404 and exit code 100. Run `apt update` before installing, matching the other workflows. |
||
|
Xuanwo
|
a5079b0fc6 | chore: update lance dependency to v15.0.0-beta.1 (#4628) | ||
 Tobias
|
1ad97b148b |
build(python): publish an sdist alongside the wheels (#4522)
Closes #4246. lancedb publishes no source distribution, so a platform without a prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist to the Python release. ## The sdist job The release workflow builds an sdist with the maturin version of the wheel jobs. On a tag, the publish job uploads it with the wheels: to PyPI, and to Fury for beta tags. The job also runs on the pull request dry run, which now starts on a change to `python/pyproject.toml`. ## Checks - The job builds a wheel from the sdist with `--locked` and smoke-tests it in a fresh venv. - `ci/check_sdist.py` fails if a tracked file of the Python package is missing from the tarball. - `twine check --strict` checks that the metadata renders. `pydistcheck` enforces PyPI's 100 MiB size limit. - The unit tests of both check scripts run in the CI scripts workflow. ## Cargo.lock maturin ships the repository lockfile, which pins crates that only the dropped `nodejs` member uses, so `cargo --locked` fails (PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the repository workspace, so a lockfile that has drifted from `Cargo.toml` fails there. It then removes the nodejs-only pins and repacks the sdist. `ci/check_cargo_lock.py` fails if the prune changes any package it keeps, or any dependency of one. The published file is the repacked one. ## Behavior change After the first release with an sdist, pip on a platform with no current wheel builds the newest version from source, which needs a Rust toolchain and `protoc`. Before, pip installed the last version with a wheel for that platform. |
||
|
Brendan Clement
|
52eed381bb |
ci: add an advisory Vale check for documentation prose (#4581)
Checks `docs/web` against the shared LanceDB style rules, so the same mechanical mistakes reach review. **Change.** - Add `.vale.ini`, pinning the shared LanceDB Vale package by release (`vale-lancedb-v0.1.0`), and ignore the synced `.vale/` directory. - Add the Docs prose style workflow. On pull requests that change `docs/web` pages or `.vale.ini`, it annotates only added or changed lines and never fails on a style finding; an install, sync or configuration error still fails it. - Link the style guide from `CONTRIBUTING.md` and show how to check a page locally. Validation: Tested in [temporary PR #4595](https://github.com/lancedb/lancedb/pull/4595). Vale flagged all five deliberate style issues on the changed prose lines, left inline code and unchanged lines alone, and kept the check green. [Passing workflow run](https://github.com/lancedb/lancedb/actions/runs/39299187786/job/118219554825). |
||
|
Will JonesandClaude Opus 5.5
|
eabec507a8 |
ci: pin protoc v36.2 with checksums across wheel builds (#4535)
## Summary The Python wheel builds used different protoc versions: v24.4 on Linux, v21.12 on Windows, and whatever Homebrew shipped on macOS. None of them verified the download. A new composite action, `.github/actions/setup-protoc`, now holds one pin, v36.2, with a SHA-256 for each platform's zip. - macOS and Windows: the action downloads protoc, checks its hash, and installs it. - Linux: protoc has to be installed inside the manylinux container, so the build reads the version and hash from the action's outputs and checks the hash there. protobuf does not publish checksum files for protoc. The pinned hashes are the digests GitHub recorded for the v36.2 release assets. Older releases such as v24.4 have none, which is why this moves to v36.2. protoc only runs at build time, through `prost-build` in the Lance crates. The Linux protoc binary is statically linked, so it runs in the manylinux 2_17 container. The action is also added to the path triggers of `pypi-publish.yml` and `python.yml`, so a change to the pin runs the wheel builds. Follow-up to https://github.com/lancedb/lancedb/pull/4522#issuecomment-6160848344. The sdist job in that PR can use the same action once this merges. ## Evidence Running the action's install script locally on linux-aarch_64 installs `libprotoc 36.2`, and a wrong hash fails with `protoc checksum mismatch`. `cargo check -p lancedb` with `PROTOC` set to protoc 36.2 builds the Lance protos. The PyPI Publish dry run on this PR covers the Linux, macOS and Windows builds. ## Merge Risk **Door:** Two-way. CI only. **Blast Radius:** Python wheel builds. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan Clement
|
07e626ec48 |
fix: check docs web test license headers (#4474)
Remove scaffold that accidentally caused the post-merge Rust license-header check to fail. Include `docs/web-tests/**` in the PR workflow paths so future docs test changes run the license checks before merge. Validation: `cargo fmt --all`; `git diff --check`; license-header-checker v1.5.0 using all four workflow commands on a clean checkout; Cargo metadata confirms all six Rust example targets remain unchanged. |
||
 LanceDB RobotandWyatt Alt
|
17e9a657e6 |
chore: update lance dependency to v14.0.0-beta.1 (#4445)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core to [v14.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.1). No compatibility fixes were required. Validation passed: `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. --------- Co-authored-by: Wyatt Alt <wyatt.alt@gmail.com> |
||
|
Will JonesandClaude Opus 5.5
|
11b679a2db |
ci: add critical fixes section to release notes (#4361)
Adds a "🚨 Critical Fixes" section to the release notes, directly below Highlights, for PRs labeled `critical-fix`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 
|
2734e10767 |
fix(node): update arrow to 21 (#4362)
Thanks @vincentkoc for starting this! I was just looking into this myself. I started with your PR and added: - make arrow 21 the default - widen the peer range to <22 so if they release 21.3 that'll be included - have to do skipLibCheck in tsconfig.json, because of flatbuffers 25 which needs TypeScript 5.7 - fix override in Symbol.hasInstance - support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView, IntervalMonthDayNano) - arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20 (adds 2 sec) - put the list of arrow types in one place - update NODEJS_THIRD_PARTY_LICENSES.md - regenerated typescript docs (sorry about reorderings, one time cost) - reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome. git diff -w shows 110 lines edited) - fixed Sliced Utf8View / BinaryView serialization error, by just converting them to Utf8 and Binary (as our rust code would anyway) --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
0327618a62 |
feat(sql): add native vector dedup and partitioned mv refresh (#4303)
## Summary
Materialize a source-preserving dedup result from an indexed, pinned
snapshot with a short declaration. The default greedy policy keeps the
smallest unassigned snapshot row ID and removes only its directly
matching neighbors: For A < B < C, an A–B–C chain without an A–C edge
keeps A and C.
```python
from lancedb import vector_dedup
view = db.create_materialized_view(
"images_clean",
vector_dedup("images", column="phash", distance_threshold=4),
)
cleaned = view.table
```
The Python source expression captures the source version once at
creation and
waits for the materialized result by default. Use `dataset_version=...`
for an
explicit snapshot, `with_no_data=True` for declaration only, or
`create_materialized_view_async(...)` for a job handle. AsyncConnection
supports
the same expression. It uses the existing MV REST endpoints and registry
jobs,
without opening a SQL query connection. Source names are exact
root-namespace
table names. Additional select/where/limit options are rejected for this
source.
The SQL declaration remains available:
```sql
CREATE MATERIALIZED VIEW images_clean AS
SELECT * FROM vector_dedup('images', 7, 'phash', 4);
REFRESH MATERIALIZED VIEW images_clean;
```
This is the host-facing SQL entry point. The MV builder now accepts a
typed native vector source; the Rust SDK also supports the same
definition through `MaterializedViewDefinition::from_sql` and
`prepare_definition`; `view.refresh().execute()` executes it locally.
Hosts distribute its units using their existing job infrastructure. The
raw `vector_duplicate_pairs` source remains available separately when
callers want the pair relation.
## Implementation and contract
- Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native
partition primitive computes pairs from index representations without
source-vector aggregation or a Python pair loop.
- Extend partitioned MV refresh with two dependent waves. Physical
segment/partition tasks stream pairs through a bounded external sort,
select direct representatives and write per-source-fragment bitmap
artifacts. Source-fragment tasks start after every selection receipt
completes and stream retained original rows, including blob payloads,
into uncommitted output fragments.
- Reuse normal blob write coercion and row-mask scans, including stable
row-ID translation after compaction. Null vectors, isolated rows and
source deletions are preserved correctly. Conflicting representative
decisions across overlapping index scopes fail before publication.
- Expose `dependency_units()` and `write_refresh_partition_with_inputs`
alongside the existing plan/commit API. Receipts bind
snapshot/configuration/policy/destination generation. Only a complete
accepted set can publish; unfinished units restart, completed units can
be reused. There is no pair-position checkpoint.
- Use MV format 4 for dedup results and format 3 for native pairs;
Python and TypeScript metadata readers recognize both. No source
deletion or source row-ID migration occurs.
Only same-segment/same-partition edges are covered. Quantized scores use
symmetric represented-vector distances. Result materialization requires
V2 data storage and is initially a native MV source, not an ordinary
SELECT table function. Compressed vertex masks stay in memory; edge
sorting spills to local disk. Unreferenced artifacts use normal Lance
retention/cleanup and must be retained for the job's recovery window.
## Validation
AWS Linux validation:
- Native pair/result integration matrix passed: multiple
segments/partitions, exact thresholds, empty output, cross-batch data,
single-process pair parity, source preservation, lost receipts and
publication guards.
- Blob/null-vector/pinned-deletion tests passed with both row-ID modes,
including compaction that changes physical row addresses.
- Representative policy, forced external-sort spill and
conflicting/corrupt artifact tests passed.
- Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust
formatting and Clippy passed.
- Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native
bindings were rebuilt in project-local environments; formatting, lint
and TypeScript docs passed.
- Rust declaration doctests: 3 passed, including the complete local
dedup example.
- Companion host validation: 41 MV job tests passed (1 pre-existing
ignored), full workspace Clippy passed, and the complete process-level
CLI E2E suite passed (3 tests). Recovery exercises both selection and
materialization waves.
- A real CIFAR-10 pHash run used five actual registry-worker owners,
completed 32 selection + 24 materialization tasks, and retained 49,846
of 50,000 rows with identical original values and image payloads.
Host workflow performance measurements are recorded in the companion
host PR. Cross-platform CI is still running. The previous remote-test
bootstrap failed before Cargo tests with `401 Bad credentials`; the
workflow fix described below is being validated by CI. No deployment or
merge is performed.
## Python source validation
AWS tests cover automatic and explicit snapshots, physical/stable row
IDs, local
and remote creation, sync/job/async-connection APIs, delayed refresh,
strict
options and identifier quoting. The native Python extension was rebuilt
through
the project's uv/maturin backend. Ruff, Pyright and the full MkDocs
API-reference
build passed; both new public symbols are rendered.
A real 50,000-image FEv2 run disabled both Python SQL entry points and
supplied an
unreachable SQL endpoint. The new API still materialized the same 49,846
retained
rows and original image payloads. Three recorded creation/refresh jobs
completed
56 tasks each on all five registry-worker owners, and remained on source
snapshot
3 after source latest advanced to 4. No worker or server wire-protocol
changes
were needed for this client entry point.
Previous CI authentication failure:
https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265.
## Review and CI fixes
- Validate all JSON merge source fields before projection, rejecting
unknown and duplicate names at every struct level. Missing/reordered
fields and ordinary JSON merges remain supported. A checked-in 12-case
integration matrix covers root fields, structs, lists, large lists,
fixed-size lists and map values; rejected writes leave the table version
and row count unchanged.
- Match namespace connection MV method signatures to
`MaterializedViewSource` in all four sync/async methods, fixing the
Python 3.10 import failure from runtime override checks.
- Restore the remote-test script's encrypted-CLI path: use the Actions
token for artifact download and pass `SOPHON_READ_TOKEN` under the
variable the decryption script consumes. The previous workflow omitted
that variable and attempted the private release path, failing with HTTP
401. On the final head, remote CI passes: 1,464 Rust unit tests, the
integration targets and doctests pass (3 unit tests ignored).
AWS validation for these fixes: both review reproducers failed before
their fixes and pass afterward; 55 merge tests, 29 cast tests and the
standalone 7-case admission regression pass. Python 3.10.21 imports
successfully; 90 MV/namespace tests, 14 JSON tests and four real
namespace dedup cases pass. After `794b97a0`, the native binding was
rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and
Pyright pass. Sophon's locked test compilation passes against the final
SDK source for all seven affected crates. Extended runners, logs and
benchmark artifacts remain local.
Gatekeeper recommends approval on `794b97a0`, and the `breaking-change`
label is set. Final CI: **49 checks passed, 3 expected skips, no
failures or pending checks**, including remote tests, Python 3.10,
minimum dependencies and Windows ARM Rust tests. All wheel builds pass
after retrying the Linux matrix affected by transient AlmaLinux
repository-metadata 404s. All code-review threads are resolved. The PR
remains open for maintainer review and merge.
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
Co-authored-by: Lance Release <lance-dev@lancedb.com>
|
||
|
Xuanwo
|
a9e350c158 |
ci(python): build Windows release wheel on 8-core runner (#4264)
Since #4241 restored fat LTO / 1 codegen unit for the Windows wheel, the `windows` job in PyPI Publish has been cancelled at its 90-minute timeout on every run, including the v0.40.0-beta.6 and v0.40.0-beta.7 tags, so neither release was published. In those runs dependency compilation finishes after ~26 minutes and the `lancedb` crate compile + fat-LTO step was still running after 63 minutes. Move the job to the org's `windows-2025-8x-x64` larger runner and raise the timeout to 150 minutes. Fat LTO stays, since the thin-LTO wheel exceeds PyPI's 100 MiB limit. The final fat-LTO step is largely single-threaded, so the gain from 8 cores is mainly in the parallel dependency phase plus faster/larger hardware; the PR run of this workflow is the first measurement of fat LTO on this runner. |
||
|
Xuanwo
|
ff757b74fd |
fix(python): reduce Windows release wheel size (#4241)
The Windows wheel for 0.39.0 exceeded PyPI's 100 MiB file-size limit, preventing the original release upload. Restore the repository's release profile (`fat` LTO, 1 codegen unit) by removing the Windows-only `thin`/16 overrides introduced in #3716. Keep `rust-lld` as the linker. This recovers wheel-size headroom at the cost of the longer fat-LTO build. The already-published 0.39.0 Windows wheel was recovered separately by recompressing the original artifact; this change addresses the build configuration for future releases. Fixes #4239. ### Windows size comparison Built the same v0.38.0 source (`8c68e0c619f2b1febe92e51a29d72c968d24a5c9`) twice on one AWS `m7i.4xlarge` Windows Server 2025 machine, changing only the LTO/codegen-unit overrides: | Artifact | thin LTO / 16 units | fat LTO / 1 unit | | --- | ---: | ---: | | Installable Windows wheel | 104,129,447 bytes (99.31 MiB) | 73,875,399 bytes (70.45 MiB) | | Uncompressed native module | 309,969,408 bytes (295.61 MiB) | 206,609,408 bytes (197.04 MiB) | The thin/16 configuration increased wheel size by **40.95%** and native-module size by **50.03%** relative to fat/1. The compressed native module accounts for all but 3 bytes of the wheel increase. Both builds used Rust 1.97.0, maturin 1.12.4, Python 3.13.5, MSVC 14.44.35207, Windows SDK 10.0.26100.0, rust-lld, static CRT, default features, and `maturin build --release --strip --locked --verbose`. They ran sequentially with separate empty target directories and identical locked third-party dependencies. The tag's stale workspace package versions were normalized once before both builds. This controlled pair used VS2022 Build Tools; the historical GitHub runner used VS2026. Both wheels passed ZIP/RECORD integrity checks and installed successfully. Native smoke checks covered import, database creation, row count, nearest-vector query, and reopening the database. This establishes the combined configuration effect; it does not isolate LTO mode from codegen-unit count or establish a runtime-performance difference. The figures above are the v0.38.0 reproduction, **not measurements of this PR head**. The existing PyPI Publish pull-request workflow rebuilds the current revision without publishing; its result is pending. Workflow validation passed with `actionlint -shellcheck=`. Full actionlint reports the same three pre-existing ShellCheck diagnostics in the unchanged repository-selection step as on `main`. |
||
|
Jack YeandXuanwo
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Jack YeandXuanwo
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Will JonesandClaude Sonnet 5
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack Ye
|
e639b1b650 |
feat: add asynchronous remote SQL queries (#4070)
## Summary
Add SQL execution to remote LanceDB connections. On the standard
synchronous connection, `execute_query` waits for the initial result
stream and returns its Arrow reader. `execute_query_async` is called
without Python `await` and immediately returns a query handle for status
inspection, streaming, or cancellation. Local databases report that SQL
is not supported.
The transport and query lifecycle live in Rust. Python exposes
native-backed synchronous and asynchronous connection methods and query
wrappers; it does not use PyArrow's Flight client.
## User experience
The standard synchronous connection supports both direct reads and
background query execution:
```python
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
# Direct execution waits only until the initial result stream is available.
# Later batches continue streaming as the query progresses.
reader = db.execute_query(
"SELECT * FROM events",
default_namespace_path=["production"],
)
for batch in reader:
print(batch.num_rows)
# Background execution returns a query handle immediately. Despite the
# `_async` suffix, no Python `await` is needed on a synchronous connection.
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
description = db.describe_query(query.id)
print(description.status)
print(description.progress)
print(description.expires_at)
# Start reading as soon as the service advertises partial results. The reader
# continues polling and yields newly available record batches until the query
# and all result endpoints are complete.
reader = query.reader()
for batch in reader:
print(batch.num_rows)
# Or cancel a different still-running query. Its status becomes "cancelling"
# while the server is still working, then "cancelled" once confirmed.
cancelled_query = db.execute_query_async("SELECT * FROM large_events")
cancelled_query.cancel()
```
The less commonly used asynchronous connection exposes the same
operations as coroutines:
```python
async_db = await lancedb.connect_async(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
query = await async_db.execute_query_async("SELECT * FROM events")
async for batch in await query.reader():
print(batch.num_rows)
```
The UUIDv7 query id is scoped to the connection that submitted it. The
connection retains lightweight shared query state used by
`query.describe()` and `db.describe_query(query.id)`; the id does not
encode SQL or a Flight continuation token and is not a cross-connection
resume token. Abandoned state has bounded retention, and terminal state
remains available briefly.
Unqualified table names use the connected database and the `public`
namespace by default. `default_namespace_path` accepts a list such as
`["production", "events"]`. SQL can still use qualified names to
reference other databases and namespaces available to the deployment.
## Design
- Uses Arrow Flight `PollFlightInfo` for submission and long polling,
`DoGet` for results, and `CancelFlightInfo` for cancellation. Each
`PollInfo.info` is treated as the cumulative set of currently available
endpoints, so advertised tickets are consumed once and batches can be
delivered before execution is complete.
- Serializes result completion and cancellation into one lifecycle. A
server-accepted request reports `cancelling` and wakes blocked
status/result work; a later retry can confirm `cancelled`. Result
retrieval is rejected after cancellation is accepted, while cancellation
after a result was already delivered is a no-op.
- Assigns a time-ordered UUIDv7 connection-scoped query id and retains
only shared evolving lifecycle state, keeping SQL, Flight continuation
tokens, and Arrow result data out of public ids and the registry.
- Leaves admission control to the server while honoring server
expiration and a local fallback retention window for abandoned entries.
- Retains terminal ids for five minutes so they remain available for
connection-level description.
- Keeps one lazily initialized SQL client on each remote database
connection and attaches fresh authentication, routing, namespace, and
request metadata to every operation.
- Applies the configured overall timeout to each execution, description,
reader, and cancellation operation. A result reader carries one absolute
deadline from `reader()` through the end of streaming; connect and read
timeouts continue to bound their individual phases.
- Returns a bounded, backpressured, single-consumer Arrow stream rather
than collecting the full result in memory. Dropping the reader stops
downloading but does not implicitly cancel the server query.
- Preserves typed schemas for empty result sets through the stream
schema.
- Accepts Flight result messages up to 1 GiB so a valid row containing a
large blob, string, or vector is not rejected by tonic's 4 MiB default
receive limit.
- Supports the Python client first while keeping the authoritative
implementation in the Rust core.
|
||
|
Will JonesandClaude Opus 5
|
a87cada90e |
feat(node)!: require Node >= 22 and drop npm lockfiles (#4074)
The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.
## Dropping npm
`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.
`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.
Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.
The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.
## Node support
Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.
This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.
The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.
## Dependabot
There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.
## Pre-commit biome
The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.
## Testing
`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.
Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.
## Not included
`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.
## Breaking changes
`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Will JonesandClaude Opus 5
|
84f46df876 |
ci(nodejs): fix nightly OOM on the aarch64 publish legs (#4077)
The nightly `NPM Publish` run has failed every night since at least Aug 23, always on the same two legs: `aarch64-unknown-linux-gnu` and `aarch64-unknown-linux-musl`. The other five targets pass. rustc is OOM-killed during the fat-LTO codegen of the cdylib — `signal: 9` with no diagnostic, about 27 minutes in — and on the musl leg that takes the whole runner down with `The runner has received a shutdown signal`. Both legs now pass: | leg | before | peak memory | wall time | | --- | --- | --- | --- | | `aarch64-unknown-linux-gnu` | OOM-killed at ~27 min | 31391 → 22851 MiB | 38m43s → 22m04s | | `aarch64-unknown-linux-musl` | runner killed at ~28 min | >32 GiB → 16516 MiB | ~40 min → 20m50s | **ThinLTO** is most of that. Fat LTO is single-threaded, and its peak is consumed inside rustc's LLVM before any linker process is spawned — which is why it is the whole fix on musl, and why lld alone left the gnu leg still peaking at 31391 MiB against the runner's 32 GiB. Both legs now use the `lto: thin` / `codegen_units: 16` settings that darwin and both Windows legs already use, at a cost of a few percent runtime performance. **lld** covers the rest, on the gnu leg. arm64 Linux otherwise links through GNU `ld` where x86_64 already defaults to `rust-lld`, which is why only the arm64 legs hit this at all; on a comparable arm64 build (`lancedb/sophon#7313`) it cut the largest single linker process from 7.0 to 4.0 GiB and wall time by 35%. The flags live in a small wrapper script used as the linker rather than in `-C link-arg`, because the per-target rustflags variable does not reach every unit that links: dependency crates linking a dylib (`crc-fast`, `lance-arrow`) were invoked as bare `clang`, which targets the x86_64 host and fails with `Relocations in generic ELF (EM: 183)`. Separately, and affecting five legs rather than two: the three ThinLTO targets exported `CARGO_PROFILE_RELEASE_LTO` and `CARGO_PROFILE_RELEASE_CODEGEN_UNITS` from `pre_build`, which runs inside the build step — after the cache step. `Swatinem/rust-cache` computes its key when the action runs, before any step, so step-local values are invisible to it. The result is a loop that never converges: the key never changes, so restores are exact hits, an exact hit makes the post-run save a no-op, and cargo invalidates the restored artifacts anyway because the flags differ. Those legs have been rebuilding cold on every run. Both values move to job-level `env:` ahead of the cache step, driven by new `lto:`/`codegen_units:` matrix fields, and are forwarded into the containers with `-e` since `docker run` inherits nothing. Every leg's cache key shifts once as a result, so expect one cold rebuild. A `Report peak memory` step is added so whether these legs fit is a number rather than an inference from whether the runner survived. It produced the figures above. ## Not included Moving these legs to native arm64 runners. It would retire the zig cross path, the `AT_HWCAP2` workaround and the `TARGET_CC` override, and arm64 runners are billed roughly 37% below x64 at equal core count — but the `lts-debian-aarch64` image exists to link against the manylinux2014 sysroot's glibc 2.17, and building natively on ubuntu-24.04 would raise the minimum glibc for every published aarch64 binary. That is a user-facing decision, not a CI cleanup. Dropping these legs to smaller runners, which is where the real cost saving is — larger runners are billed even on public repos. On these numbers it is not available yet: musl at 16516 MiB is about 130 MiB over what a 16 GB standard runner has. Worth revisiting as a follow-up. ## Testing Cargo's rustflags precedence was checked locally rather than taken from the docs, since getting it wrong would silently change the published binaries. With a throwaway crate carrying both a `target.'cfg(all())'` and a per-target rustflags table: setting `RUSTFLAGS` discards both, and setting it to the empty string discards them too. That rules out routing the linker flag through a job-level `RUSTFLAGS`, because `env:` keys cannot be conditionally omitted and every other leg would then silently lose the `target-cpu`/`target-feature` settings in `.cargo/config.toml` — `+avx2` on x86_64 and `-crt-static` on aarch64-musl. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 5
|
217ea1a799 |
ci: use thin LTO and a larger runner for the Windows wheel build (#3716)
The Windows wheel job is the slowest job in the PyPI release workflow. Fat LTO of the cdylib is single-threaded and the peak-memory step of the build, so it does not get faster with more cores — and it has already caused rustc-LLVM OOM on the Windows runners for the nodejs builds. Switch the job to thin LTO with 16 codegen units on a `windows-2025-8x-x64` runner, trading some runtime performance on our least performance-sensitive platform for build time. This matches what the nodejs Windows builds in `npm-publish.yml` already do. `pypi-publish.yml` is in this workflow's `pull_request` paths filter, so this PR triggers a dry-run build that shows the new timing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 5
|
bacd0e4c3c |
ci: group arrow and datafusion dependabot updates into one PR (#3738)
The arrow-rs and datafusion crates are released in lockstep, but Dependabot has been opening one PR per sub-crate for them — the 58.3.0 to 58.4.0 wave produced four separate PRs for `arrow`, `arrow-array`, `arrow-schema`, and `arrow-buffer`. The existing `rust-minor-patch` group did not catch them because it only filters on `update-types` and declares no patterns. This PR adds an explicit `arrow-datafusion` group matching `arrow*`, `parquet*`, `datafusion*`, and `object_store`, so those bumps arrive as a single PR. It is listed before `rust-minor-patch` because a dependency joins the first group it matches, and it deliberately omits `update-types` so major bumps are grouped too. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
LanceDB RobotandYang Cen
|
cf27f6902e |
chore: update lance dependency to v11.0.0-beta.16 (#3992)
Updates Lance dependencies and Java lance-core to v11.0.0-beta.16. Also narrows the dependency updater's package matching so the local LanceDB crate remains a path dependency. Lance tag: https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.16 --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com> |
||
|
Xuanwo
|
f76ee304b8 |
ci: isolate remote Rust tests (#3998)
The Linux Rust job can exhaust its disk after restoring a large fallback target cache and compiling multiple feature graphs into one target directory. Run remote tests in an independent job with registry-only caching, and run the simple example with all features so it reuses the preceding build artifacts. This preserves remote coverage and fork behavior while preventing all-features and remote-only artifacts from accumulating together. Failure evidence: https://github.com/lancedb/lancedb/actions/runs/32467317540/job/96726650990 |
||
|
Xuanwo
|
4ba2421254 |
refactor(python): require pydantic v2 (#3990)
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2 APIs throughout. This removes dual-version behavior from schema conversion, query serialization, embedding models, and Function wire models while preserving their existing public and canonical-wire behavior. The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared compatibility floor remains covered. |
||
|
Wyatt AltandClaude Opus 5
|
790d0c684c |
docs(ci): clarify tag input on codex-update-lance-dependency (#3924)
Say what resolving "latest" actually does: pick the newest release, preferring stable over pre-release, and skip the run if it is not newer than the version pinned in Cargo.toml. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 LanceDB RobotandYang Cen
|
031c3585a8 |
chore: update lance dependency to v11.0.0-beta.7 (#3925)
Updates the Rust workspace Lance dependencies and Java lance-core dependency to v11.0.0-beta.7. No compatibility fixes were required; full-workspace Clippy passes with warnings denied. Lance tag: https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.7 --------- Co-authored-by: Yang Cen <159225399+BubbleCal@users.noreply.github.com> |
||
|
Xuanwo
|
5acce6782e | ci(docs): report link checker failures through issues (#3909) | ||
|
Xuanwo
|
b1cfe6edb1 |
ci(docs): add scheduled doc link check (#3888)
The docs have no link checking at all, so external links rot silently: a trial run already found `docs/src/python/python.md` pointing at `lancedb.github.io/lance-namespace`, which returns 404 since the repository moved to the lance-format org. Checking external links on the blocking path would be the wrong trade: third-party hosts rate-limit automated clients, reject non-browser user agents, and go down temporarily, so any of them having a bad minute would turn unrelated PRs red. Following lance-format/lance#8315, this adds a daily `lychee` run that reports broken links into a single tracking issue, rewritten in place on each run and closed automatically once every link resolves. The scan job runs the downloaded lychee binary with a read-only token; everything that writes lives in a separate report job, and a non-verdict lychee exit fails the run instead of publishing a bogus report. The check is restricted to http(s) links because much of `docs/src` is generated API reference (the `js/` tree comes from `npm run docs`) and the hand-written pages use mkdocstrings cross-references and nav-relative paths that only resolve in the site mkdocs builds, so relative links would be reported as broken on every run. The one broken link the trial run surfaced is fixed here; after the fix, a local run over all 154 files reports 0 errors across 216 unique links. |