Catalog lookup failures lose their structured resource identity at the
Rust/Python SDK boundaries, leaving callers with generic errors or empty
correlation fields. Decode the existing Flight SQL error metadata and
preserve resource kind/name, namespace, error code, and request/job IDs
in the existing error types.
Direct `get_function` calls unwrap named Function or namespace failures.
Generic SQL job failures remain `JobFailedError` with the typed resource
error as their cause; native deferred commands correctly leave durable
job IDs null. Existing unstructured SQL classifications, including
`FailedPrecondition`, remain supported.
Validation: focused Rust/Python regressions and a native-extension run
against a real Sophon process passed all five Function/Secret
resource-error cases. The final one-commit candidate passes `cargo check
-p lancedb-python` and formatting; earlier runtime evidence applies
after removal of an unused tuple member. Independent deployed acceptance
remains pending for ENT-3373.
Server companion: https://github.com/lancedb/sophon/pull/9804.
Integration validation: all 14 DocLinks pass on the merge candidate with
#4869. The subsequent main documentation routing change was inspected;
it does not alter the typed-error paths. The inherited pin-utils
cargo-deny finding is tracked in a separate dependency repair.
- Removes the stale `/enterprise` redirect that hijacked the real
LanceDB Enterprise page, sending it to the Embedded vs Enterprise
comparison page instead.
- Removes the Merge page from the Data Ingestion nav (confusing parity
story between Embedded and Enterprise); moves RTE Write up to right
below Delete and renames it **RTE Write (Upsert)** to take over that
slot.
- Removes the Early access badge from Tiered Index Cache and restores
its full sidebar title (was shortened to "Index Cache" to fit the
badge).
- Moves Changes below Dedupe in Data Curation.
- Matches Advanced Conflict Resolution's sidebar title to its page title
(previously just "Conflict Resolution" in the sidebar).
- Marks Job Framework as Enterprise-only.
- Distributed RTE Write diagram now shows the `metrics` table split
across multiple shards, illustrating the per-table sharding described in
the page's own text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The SDK documentation-link test fails on `DocLink::BranchDiff` after the
branching page is hidden from navigation. [Mintlify serves hidden MDX
pages by their URL](https://www.mintlify.com/docs/guides/hidden-pages),
so navigation membership is not a route-validity requirement. Remove
that assertion while retaining the file, `.mintignore`, redirect-target,
and anchor checks.
Validated on the development EC2 with `cargo test --profile ci
--features remote --locked --lib
docs::tests::test_all_doc_links_resolve_in_repo_docs -- --exact
--nocapture`: the unchanged base fails on `tables/branching`; this fix
passes. The identical patch also passes against the exact #4867 head,
checking all 14 `DocLink::ALL` entries. Formatting passes. This repairs
the shared docs-test blocker for #4867; its separate `cargo deny`
dependency finding is unchanged.
The refresh guide and SQL reference do not warn that a `WHERE` refresh
recomputes filled rows, and the guide describes resources as per-worker
allocations. Clarify the aggregate Function budget and expression-only
`num_workers`, show the output-null predicate with its stale-value and
successful-null limitations in both references, and document
refresh-level `error_policy='skip'`. Link both guides to the current SQL
options route.
Validation: Vale 3.24.0 found no issues on changed lines; `git diff
--check` and the new link targets passed. Mint 4.2.888 reports the same
four existing `RsExampleBody`/`RsExampleTail` import warnings in
`reranking/index.mdx` and `search/vector-search.mdx` on both pristine
base `5e38120b` and this branch. No additional warnings. Behavior was
checked against the owning Sophon implementation; no runtime code
changed or runtime tests were run.
Companion to https://github.com/lancedb/sophon/pull/9806 for ENT-3279.
Page visibility and saved documentation editions are unchanged. DOC-90
retains publication and matching-shipped-release verification.
## Summary
- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.
## Note
I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
`take_row_ids` used an `IN` filter, so the backend could reorder rows
and collapse duplicate IDs. That can silently misalign training rows
with labels or sample weights.
Restore the requested order and duplicate occurrences in the shared Rust
take path for native and remote tables, and retain that information
through Python query serialization. Missing or deleted IDs are omitted.
Explicit Rust `order_by` overrides request order, and pagination applies
after restoring occurrences. Empty projections and `_rowid` projection
aliases retain their output schema.
RTE tables reject unsupported takes with the existing base-table
guidance before checking the internal row-ID projection used for
ordering. An explicit `use_rte(False)` reads the base table with the
same requested order and occurrences.
A pre-existing fixture defect also blocked CI:
`test_table_add_in_threadpool` responded without reading uploads,
allowing a connection reset during response decoding. This PR includes a
fix confined to that test's handler: consume each request body before
responding and verify one create row plus ten inserted rows. This was
not a regression in the take implementation. Production write retries,
concurrency and HTTP client code are unchanged.
The selective sampling guide scopes the row-ID guarantees to SDK
releases containing this fix, preserves the distinct offset-take
semantics, and explains that ordered takes collect and combine matching
rows before output batching. It recommends smaller ID requests and
narrow projections because `max_batch_length` does not cap working
memory. This uses the existing REST query protocol and requires a new
SDK release; no deployed service was changed.
Validation:
- Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX
compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed
lines. Full-site Mint validation reports four existing snippet-import
warnings on the reranking/vector-search pages, identical with the
original page restored in a baseline comparison.
- The applied threadpool test passed locally after the fixture change.
The identical candidate previously passed 10 trials with the ordinary
reader and 10 with an unbuffered reader; each trial verifies all
uploaded rows. Those checks used macOS arm64/Python 3.14/default
features, not the full failing CI environment. The normal new-head macOS
Python 3.13/experimental-functions job remains the required platform
check.
- Python package Ruff format/check passed. Repository-root Ruff still
reports 59 existing errors and six formatting differences in unchanged
`docs/web-tests` files; none are in changed files.
- At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed;
5 targeted cases and strict workspace Clippy passed. The fixture-only
follow-up changes no production Rust/Python code, so that evidence is
reused.
- Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests
and 24 Python take/getitems tests passed; the focused 49-test suite also
covered offset dataloaders. A local 9,216-row, four-fragment
reproduction returned 200 seeded random batches of 64 rows with zero
ordering or label-alignment mismatches. The ordering and projection
implementation is unchanged by the follow-ups.
- No live Enterprise run was performed.
Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Add SDK coverage for a refined IVF_RQ7 query after deferred compaction
with physical row addresses. The test combines deletion and
prefiltering, verifies the retained index UUID and actual FRI metadata,
requires an ANN plan, and compares IDs/distances before and after
compaction plus recall against a flat oracle.
This is a test-only change on main with the released Lance beta.10
dependency adopted by #4838. It adds no production behavior or
dependency changes. Sophon lazy-executor adoption and integrated
performance acceptance remain separate.
Validation on macOS ARM64 with pinned Rust 1.97.0:
- Five focused SDK regressions passed (0.15 s test runtime), covering
deferred FRI compaction/refinement, stable row IDs, ANN approximation
mode and large query limits.
- `cargo clippy --locked --features remote --tests --examples -- -D
warnings` passed.
- Formatting and diff checks passed.
The native test linker emitted a macOS compact-unwind size warning; all
cases completed successfully without linker-setting changes.
Labels early access features and hides features we aren't documenting
yet.
**Early access label:** Functions (4 pages, plus the GPU refresh,
checkpointing and adaptive execution pages), Secrets (3 pages), Tiered
Index Cache, Blob V2, and merge insert, which is labelled for Enterprise
only. Two sidebar titles are shorter so the tag fits.
**Hidden:** branching, JSON columns and geospatial.
- The JSON functions page and the agent branch-experiments tutorial go
in `.mintignore`.
- `tables/branching` stays reachable but out of the nav, as #4859 left
it, because SDK error messages link to it.
- Other pages' mentions of these features become draft comments, so
they're easy to restore.
**Secrets page:** drops references to manual KMS setup, which the
control plane handles. It keeps the encryption-at-rest facts, states the
Azure value-size limit inline, and points a 501 error to support.
## Summary
- Hides `tools/cli.mdx` from the sidebar and from sitemap/search via
`.mintignore`, unlinking its one reference in `basics/sql.mdx`. No SDK
doc-link depends on this page.
- Removes `tables/branching.mdx` from the Management nav group, but
leaves it reachable (no `.mintignore` entry): the SDK's
`diff_branch()`/`cherry_pick()` error messages link to
`/tables/branching#diff-a-branch` and `#merge-a-branch`, and
`rust/lancedb/src/docs.rs`'s link-resolution test asserts those paths
aren't `.mintignore`'d.
- Adds a Console row to the comparison table, linking to the companion
change's new `enterprise/console` page.
## Companion change
lancedb/sophon#9783 adds `enterprise/console.mdx` and inserts it into
the "Getting Started" nav group this PR touches.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Clarify `FunctionVersion` forward compatibility: unknown top-level
fields are accepted during decoding and omitted from canonical JSON,
while all defined fields and the open `image.descriptor` content retain
their values. Extend the existing Rust and Python canonical round-trip
tests to assert the complete projected object, including future
discriminator values.
This documents and tests the existing behavior; it does not change
runtime serialization or promise lossless retention of unknown top-level
fields.
Related: ENT-2176.
The Search group covers full-text and hybrid search as well as vector
search, and full-text search is lexical, not semantic. This renames the
group to "Search" in the sidebar and in the Embedded vs Enterprise
comparison table.
`mint validate` and `mint broken-links --check-anchors` pass on the
assembled site.
Function refresh failures and skipped-row diagnostics lose the traceback
when decoded by the SDK. Expose the server-provided optional `traceback`
on job failure details and Function error records in Rust, Python, and
Node.js, while keeping existing compact error messages.
Responses from older servers remain supported when the field is absent.
Tracebacks may be truncated by the server. The Rust `Error::JobFailed`
variant now stores `Box<JobFailure>` so the added diagnostics do not
enlarge every `Result` error payload.
## Summary
- Adds a new top-level "Introduction" nav section before "Getting
Started": LanceDB, LanceDB Embedded, LanceDB Enterprise, Embedded vs
Enterprise.
- Trims the landing page's combined
Architecture/Embedded/Enterprise/Compare block down to a short pointer
section (keeping its `#architecture` anchor), moving that content into
two new pages: `basics/lancedb-embedded.mdx` and
`basics/embedded-vs-enterprise.mdx`.
- `basics/embedded-vs-enterprise.mdx` consolidates the
feature-comparison table and the Enterprise overview's own comparison
content (previously duplicated across multiple pages and, within this
page, repeated across a dimension table, several narrative subsections,
and a usage-differences list) into one comparison page with each section
covering a distinct angle.
- Repoints the `/enterprise` and `/enterprise/deployment` redirects at
`/basics/embedded-vs-enterprise`, since the previous destination only
resolves once the Enterprise overlay is applied and fails
`rust/lancedb/src/docs.rs`'s link-resolution test against this repo's
own `docs/web` tree.
## Companion change
lancedb/sophon#9718 trims the Enterprise overview page to drop the
content that moved here, and relocates its nav insertion into the new
Introduction group.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Follow-up to the Mermaid brand-theming PR, addressing review feedback:
- Diagram text was hard to read at the previous 15px font size; bumped
to 18px, light and dark alike.
- The dark-mode orange accent border was brighter and more saturated
than its fill or the default border, reading as too loud against the
dark background. It now reuses the same muted orange already used for
the light-mode accent border, which was fine as is.
## Companion change
lancedb/sophon#9719 makes the same two changes across its own diagrams.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Function catalog responses include `batch_size`, `row_timeout_seconds`,
and `result_stability`, but the Rust and Python `FunctionVersion` models
silently discarded them. Preserve these optional execution controls
through decoding, typed access, and canonical serialization; versions
without the fields retain their existing representation.
The regression exercises raw SQL transport responses and public Python
sync/async get/list paths. Live Sophon catalog acceptance remains
pending under ENT-2073 and the related FunctionVersion round-trip
requirement in ENT-2176.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.10](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.10),
which contains the released MinHash signature scanning API needed by
downstream consumers. Pin Python tests to the published `15.0.0b10`
wheels, including macOS, and refresh both lockfiles. Java remains on
beta.8.
No SDK source compatibility changes were needed. All 25 Lance Git
packages resolve to release commit
`f56a08d40b5630ce1124535c4c3b1406d5ceedf1`; third-party package versions
are unchanged.
Validation on macOS ARM64 with pinned Rust 1.97.0: strict workspace
Clippy with default and all features, Rust formatting, 32 CI-script
tests, three count tests, one WAL/vector regression, and the Python
lockfile check passed. The native test binary emitted a macOS
compact-unwind size warning; both test commands completed successfully
without changing linker settings. Downstream release and integrated
Sophon acceptance remain separate.
Function error records expose physical row locations, which can change
when a table is compacted. Expose the server-provided optional stable
`row_id` through Rust and Python so callers can locate the same row
after compaction. Document stable-ID retries and retain physical
addresses for tables without stable row IDs. Older server responses
continue to deserialize with `row_id=None`.
The Python remote response test and Rust remote deserialization test
passed on verification SDK `4e1725fa352035e2079dfd235b8271cd05adf433`,
including decoding a legacy response without the new field. This branch
applies the same SDK patch to current main; intervening main changes
leave the affected error types and assertions unchanged. Formatting and
lint checks passed after correcting the retry examples to use the
`table.name` property. In a separate Linux server experiment, a real
skipped Function row retained `_rowid=506` after compaction changed its
physical address. The SDK field requires a server that emits stable row
IDs; this PR does not change server dependencies.
Remote SQL failures currently become generic runtime errors, and HTTP
table lookup can discard server error metadata. Preserve transport
status, current request ID, SQL job ID and worker request ID across Rust
and Python. Select typed column, table, reference and schema exceptions
only when an explicit server code matches the transport status.
Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed
SQL jobs retain `JobFailedError` with a typed SQL cause when available.
Missing metadata stays absent; message text never determines the
exception class. Local missing-column errors retain their type and
qualified-field suggestions, and `JobFailure.source` documents
structured remote Flight SQL failures.
This branch integrates main at `a9ad4997`, including its released Lance
15.0.0-beta.9 dependency update and remote optimize support. The only
textual conflict was two independent additions to `test_remote_db.py`;
both test groups and all original assertions are retained. The feature
has no dependency or lockfile delta against current main.
SDK runtime validation on `03ee633a` (runtime source unchanged in
`f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python
3.14 environment:
- Rebuilt the native extension from the integrated source. All 22
targeted Python tests passed: native local errors, HTTP/Flight metadata,
legacy sync/async table-not-found behavior, and the three main
remote-optimize regressions.
- All 52 Rust SQL, metadata/source and nested-field tests passed,
including the new main optimize cases.
- Root and Python-binding Rust Clippy with `-D warnings`, Rust
formatting, Python package Ruff and conflict-file formatting passed.
- Repository-wide Ruff reports 59 existing documentation findings, and
its format check identifies six documentation files. Every affected file
is byte-identical to current main; these checks are not reported as
passing.
- Direct integration review and AST comparison confirm both conflicting
test groups were preserved. Earlier exception-constructor/pickling
evidence remains applicable to the unchanged Python error modules.
The unlocked build exhausted disk again on a runner with the same label
but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing
runner and dependency caches, serialize Cargo compilation, and remove
only completed test/bench executables reported by Cargo. This job never
runs those binaries; normal binaries, libraries and build-script outputs
remain available. Fresh resolution, the ci profile, all features, and
test/bench compilation and linking are retained.
Three probes using Cargo and target-path validation pass with `-D
warnings`: all fixture targets link while ordinary outputs remain
usable; an undefined symbol still produces Cargo exit 101 and its linker
diagnostic; files outside the target directory are never removed. The
[current-head build-no-lock
job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065)
passed on `f3f2d0b8`: all compilation/linking completed, 20 completed
test executables were removed, and `target` retained about 7.3 GiB. This
successful run had a 145 GiB root with about 78 GiB free afterward; it
was not an identical rerun of the failed 72 GiB image. The driver tests
also passed in this job. All previous-head jobs ended naturally before
the push; new-head CI and review gates still apply.
Typed remote input exceptions require the server metadata contract.
Older servers retain generic transport errors and existing missing-table
compatibility. The optimize tests use mock servers and do not establish
downstream server acceptance.
BREAKING CHANGE: Rust callers with exhaustive matches on
`lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants.
Local `Error::ColumnNotFound` failures now raise
`lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass.
Python callers that previously caught only `RuntimeError` for this local
failure must catch `ColumnNotFoundError` or `ValueError`.
Release tags currently point to the version bump before the lockfile
amend, so tagged CLI builds fail with `--locked` even though the branch
has the synced lockfile.
Keep the newly created, unpublished tag on the final amended release
commit before pushing. Existing published tags and packages remain
unchanged.
[Remote preview dry
run](https://github.com/lancedb/lancedb/actions/runs/41764901500) passed
and moved the local tag after the lockfile amend; no tag was pushed.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Update Rust Lance dependencies to
[v15.0.0-beta.9](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.9),
including the scoped stable-row-ID conversion fix. Python tests use
published beta.9 wheels on Linux/Windows while macOS retains beta.4.
Java stays on beta.8, matching the original base, so this update does
not depend on beta.9 Maven publication or macOS Java packaging.
The dependency updater preserves workspace inheritance, the WAL test
wrapper uses the new StoredBatch module, and Linux/remote Rust CI uses
its installed mold linker. The Python lockfile also records the
already-declared OCI extra. CLI CI now caches Cargo downloads under a
separate key instead of restoring compiled targets: the previous runner
exhausted its disk during cache restoration, before compilation. Its
test matrix and commands are unchanged.
This branch includes #4820's row-freshness changes: successful rows
become current while skipped rows retain their values and remain
eligible for retry. Repair identities, replay handling and Restore
fencing protect publication completion.
BREAKING CHANGE: `StalenessPlan::row_dirty` is a new public field. Rust
callers using exhaustive struct literals must initialize it.
Validation:
- All reported non-Java checks passed on
`f5fa66fcbcf31a8eb188935f728735560c77c873`, including Rust
lint/Linux/remote, Python, NodeJS and CLI checks. Current head
`c660bc0ac9fd5920ad4b64be9c19f732ef4c56a5` preserves those product
sources, manifests, lockfiles and compiler/test commands; only the Java
pin restoration and CLI cache configuration differ. Current-head CI is
tracked separately.
- The complete Java directory is identical to the original PR base. The
scope correction reuses [Xuanwo's prepared and validated
commit](https://github.com/lancedb/lancedb/pull/4814#issuecomment-6626088466);
no new Java build was run.
- CLI workflow YAML parses successfully, its inputs match the pinned
rust-cache action schema, and the existing jobs, matrix and commands are
unchanged. Hosted execution must confirm the cache-restoration repair.
- Existing validation includes the updater regressions and 32 CI-script
tests, dependency-policy checks, count/WAL coverage, and #4820's 42
freshness tests, strict core Clippy and Rust doctest. Root Ruff
previously reported unchanged documentation-example issues;
Python-package Ruff passed.
This PR adopts the existing released Lance beta.9. A subsequent official
SDK release and normal Sophon dependency adoption remain separate
delivery steps.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Every Mermaid diagram now renders in the brand palette, with separate
light and dark variants that switch with the site's theme toggle.
Default node styling stays neutral; one accent color marks each
diagram's actual subject, chosen per diagram. A couple of diagrams
changed orientation where their old layout was too wide to read
comfortably.
Companion change in lancedb/sophon uses the same colors and pattern.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Materialized-view creation and refresh can outlive the submitting
connection, but their SQL query IDs cannot be reopened through the Job
Registry. On servers that negotiate early handles, return the durable
registry job ID and use the existing job API to describe, reopen, wait
for, or cancel it. Older servers retain the connection-scoped query
handle. Registry refresh handles forward running and terminal progress
observations and establish the committed-version read-freshness fence
before wait returns.
Preserve a nested SQL failure ID separately from the outer job ID
through Rust and the Python native bindings, so Function failures remain
traceable. Rust callers constructing `JobFailure` directly must supply
the new optional `sql_job_id` field or use `..Default::default()`.
Validation: seven SDK regression cases and the docs-link test pass,
including plain wait and running-to-terminal progress observations; the
observer regression fails before the fix. Formatting and strict Clippy
pass. Earlier accepted coverage includes a real Flight test service
exercising materialized-view creation followed by registry
describe/reopen/cancel, and two Python tests after rebuilding the native
extension. The published product tree matches the validated candidate.
Refs ENT-3562. The matching Sophon server change is being validated
separately before publication.
Python Function declarations can now pin Hugging Face snapshots with
`models=[HuggingFaceModel(repo=..., revision=...)]`. The native Rust
registration path carries those declarations into `CREATE FUNCTION`, and
create/get/list results retain `image.models`. Functions without
declarations preserve their existing payload shape. The API reference
includes a model-loading example that initializes once per Function
instance.
Validation:
- The real SQL mock-service regression fails on the previous native
implementation and passes after the fix. It covers immediate/deferred
execution, requests with/without models, and create/get/list responses.
- Rust Function wire tests, strict Clippy, the public-type doctest, and
formatting.
- Rebuilt Python extension and targeted registration/golden tests.
The service-side snapshot preparation and cache are delivered separately
for ENT-3483.
The removed Enterprise landing page left the `/enterprise` and
`/enterprise/deployment` redirects pointing at a missing page, breaking
the SDK documentation-link test. Point both redirects at the existing
architecture overview, which describes Embedded and Enterprise.
The exact failing Rust test fails on current main and passes with these
redirect changes. This repairs the inherited failure blocking #4821.
Extend the existing job progress regression to cover Function worker
counts, GPU capacity and wait reasons, and known or absent warm-reuse
evidence through `Job.progress()` and `Job.wait()`. The callback
assertions also preserve an unknown warm state instead of turning it
into `false`.
Validation on EC2: all 56 remote SDK tests, Biome lint/format,
TypeScript build, and documentation generation pass. Native code and
dependencies match the previously validated build.
Refs ENT-3486.
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:
**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.
**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.
**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.
**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.
**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.
Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Replace the remaining `pylance` (`take_blobs`) snippet in the LeRobot
integration guide with `Table.fetch_blob_files`, document how to open
`.lance` datasets directly with `lancedb` alongside a
`pylance`-to-`lancedb` call mapping on `/basics/lance`, and clarify in
installation docs that the optional `pylance` extra is only needed for
engine/connector authors or format-level fragment APIs.
TypeScript job handles now expose structured progress through
`progress()` and an optional synchronous callback to `wait()`. Snapshots
preserve unknown fields and missing counters, and job listings include
progress and pause state. Existing `status()` strings and no-argument
`wait()` behavior remain compatible.
Wait callbacks observe the existing backend poller. Observer exceptions
are logged without changing the job outcome. A best-effort ETA is
available only from compatible samples with positive progress, and is
omitted during initialization, queueing, or pause.
Part of ENT-3500.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal>
`pydantic_to_schema` rejected three common field types with `TypeError: Converting Pydantic type to Arrow Type: unsupported type ...`. This maps them to native Arrow types:
| Python | Arrow |
| --- | --- |
| `datetime.time` | `time64("us")` |
| `datetime.timedelta` | `duration("us")` |
| `Literal[...]` | the type of its values, e.g. `Literal["a", "b"]` -> `utf8`, `Literal[1, 2]` -> `int64` |
`Optional[Literal[...]]` and `List[Literal[...]]` use the existing optional/list handling. A `Literal` whose values have different types (`Literal["a", 1]`) raises `TypeError` with a message stating that all values must share one type.
`datetime.time` is stored as timezone-naive wall-clock time. Pydantic model writes reject a `time` with non-null `tzinfo` before Arrow conversion, including direct, list, nested, and Enum values, with a `ValueError` explaining that the offset would be lost.
Tests cover schema mapping including optional and list forms, mixed-type `Literal` rejection, naive time/timedelta round trips, and timezone-aware rejection at the write path.
On remote tables, `create_index` returns as soon as the build is scheduled. Until it finishes, a vector index is missing from `list_indices`, a scalar or FTS index shows 0 indexed rows, and queries don't use the index. The docs didn't say any of this, so it was easy to assume the index was ready.
This adds that to the `create_index` and `list_indices` docs in Python (`Table` and `AsyncTable`) and TypeScript, and points to `wait_timeout` / `waitTimeoutSeconds`, `wait_for_index`, and `create_index_async` for blocking until the index is ready. It also explains that completion tracking is tied to the submitted build only when the server returns a job ID, while legacy name-based waits can return early when replacing an existing index.
Closes#4405
Closes#3059.
`io::object_store::test::test_e2e` has been `#[ignore]`d since Lance started writing local tables through `LocalWriter`. For a `file://` store, `ObjectStore::create()` returns a `LocalWriter` that writes straight to disk, so `MirroringObjectStoreWrapper` never sees the writes and the secondary stays empty.
Lance's `file-object-store://` scheme uses the same local provider, but `create()` only takes the `LocalWriter` path for `file`, so every other scheme writes through the wrapped store. This connects the test to `file-object-store://<dir1>` and drops the `#[ignore]`. The table files still land in `dir1`, so the existing file-list comparison covers the mirroring wrapper on real files again.
The two relative file lists are sorted before comparison so the assertion detects missing and extra files independently of `WalkDir` traversal order. The row and vector-query assertions remain unchanged.
Validation:
- Rust object-store module at `b11f7a9684d94100f40818d1249479c726b2b006`: 3 passed, with the focused `test_e2e` included.
- Hosted Linux test merge: `test_e2e` passed and the library reported 2046 passed.
- Complete applicable CI at the final head: 8 workflows and 39 successful checks passed; 2 conditional jobs were skipped.
Co-authored-by: Bilal Atique <bilalbinatique@gmail.com>
Expand the Linux Python test run without pandas and pylance from two files to
the suite while retaining the regular dependency run for optional-package
modules.
Move optional imports into tests that need them and use Arrow assertions where
pandas is not required, preserving the existing behavioral assertions.
Validation:
- Physical no-pandas/no-pylance broad run: 2884 passed, 316 skipped, 8 existing
marker cases deselected.
- Regular affected suite: 1687 passed, 39 skipped.
- Current-head hosted integration: all 15 checks and 4 workflows passed.
Co-authored-by: importcpp <57519936+importcpp@users.noreply.github.com>
Job callers currently lose the server's structured progress when
describing, listing, or waiting for a job. Expose Rust and Python
progress snapshots and optional wait observers using the existing
backend poller. Python supports callbacks, a caller-owned tqdm-like bar,
or `progress=True`; `status()` still returns a string and the default
wait behavior is unchanged.
Preserve unknown progress fields and phases from newer servers, and
report omitted counters as unknown. Each wait calibrates a caller-owned
bar from its first complete snapshot, then uses a high-water mark within
the same execution epoch; a new epoch or smaller plan resets the bar.
ETA is best-effort and omitted for missing, stale, paused, or
incompatible samples.
Newly submitted index and refresh handles forward observers through
their freshness wrapper while retaining successful-wait read
consistency. The Jobs guide includes draft Python/Rust snapshot and
observer examples for release; TypeScript progress support remains
tracked under ENT-3500.
Validation on the development EC2 host: the initial 11 Python progress
tests, REST fixture caller, and two Rust progress regressions passed.
After fixing freshness-wrapper forwarding, rebuilt the Python extension
and ran two submitted-handle Python tests plus the new Rust refresh and
index regressions (one test each); all passed. Strict Clippy, rustfmt,
and targeted Ruff passed. The final commit only wraps the new guide
section in an MDX draft and clarifies language support. Repository-wide
Ruff reports 48 existing findings in six unchanged docs/web-tests files.
This publishes the SDK portion of ENT-3500. Server-side live-split and
terminal-progress changes and the combined server/SDK end-to-end
scenario remain in progress. Refs ENT-3500.
BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress:
None`, `phase: None`, and `paused: false`. Literals for `JobDescription`
must add `progress: None`, `phase: None`, `queue_reason: None`,
`waiting_for_job_id: None`, and `paused_at_millis: None` when that
metadata is unavailable. Existing Python calls remain compatible.
Rust RRF counted ranks from zero, while Python and the [RRF
paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) count
from one. At the default `k=60`, this reverses the order of two rows
ranked `(9, 6)` and `(2, 14)` by the two searches.
Start ranks at 1 in the Rust reranker, which is also used by Node, and
document the formula. This corrects relevance scores and can change the
order of close results.
Validation:
- Both RRF regression tests fail before the fix and pass afterward,
covering `k=1` and the default `k=60` rank reversal.
- All 8 hybrid tests pass, including actual query scores, an empty FTS
side, empty results, and remote query paths.
- `cargo fmt --all` and `cargo clippy --profile ci --features remote
--tests --examples -- -D warnings` pass.
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
Adds `OCIGenAIEmbeddings`, registered as `"oci-genai"`, embedding text with the Cohere models hosted on Oracle Cloud Infrastructure Generative AI (default `cohere.embed-v4.0`) via the `oci` SDK, alongside the existing Bedrock, Gemini and watsonx functions.
- Auth delegated to the OCI SDK: `API_KEY` / `SECURITY_TOKEN` profiles from the OCI config file, or `INSTANCE_PRINCIPAL` / `RESOURCE_PRINCIPAL`. No credentials are stored on the function or in table metadata; `compartment_id` can come from `OCI_COMPARTMENT_ID` so the OCID need not be baked into metadata either.
- Endpoint derived from `region` (default `us-chicago-1`) or overridden with `service_endpoint`; dedicated AI cluster endpoint OCIDs use `DedicatedServingMode`.
- Asymmetric Cohere input types (`SEARCH_DOCUMENT` for the source column, `SEARCH_QUERY` for queries); `truncate` and `output_dimensions` (embed-v4.0: 256/512/1024/1536) forwarded.
- Batched at the service limit of 96 inputs; empty inputs are not sent and come back as `None`.
- `ndims()` uses `output_dimensions` when set, otherwise a single cached probe request, so there is no model→dims table to keep in sync with the catalog.
- SDK retries disabled in favour of LanceDB's `max_retries` backoff; 401/403 service errors carry `status_code` so they surface without retrying, with an actionable message (expired session token, IAM policy, region, model).
- `oci>=2.165.1` added to the `embeddings` extra (the floor where `EmbedTextDetails.output_dimensions` exists).
Tests: 22 mocked unit tests (auth paths, batching, input types, config validation, retry behaviour, pickling, table-metadata round trip) and a live test skipped unless `OCI_COMPARTMENT_ID` is set, which passes against us-chicago-1 for embed-v4.0 at 1536 and 512 dims and embed-multilingual-v3.0 at 1024. Verified end to end with a real LanceDB table (`LanceModel` with `SourceField`/`VectorField`, 6 rows, two semantic queries ranked correctly). `ruff format --check`, `ruff check`, pyright and typos are clean.
Docs: a page under `docs/src/embeddings/available_embedding_models/text_embedding_functions/` in the shape of the remaining `voyageai_embedding.md`; the function is exported from `lancedb.embeddings` so the API reference picks it up. The hosted docs' available-models table lives outside this repo and would need a separate update.
A synchronous remote table opened in a namespace is displayed as if it
were in the database root. For example, `db.open_table("test",
namespace_path=["analytics", "features"])` shows
`RemoteTable(dev.test)`.
Include the cached namespace path using the existing `$`-delimited table
identifier: `RemoteTable(dev.analytics$features$test)`. Root output and
the leaf `table.name` stay the same. Using cached identity also keeps
repr available after pickle without reopening the native handle or
making a request.
Addresses the repr portion of ENT-3559. The native formatter already
carries the qualified identifier used by Python async and TypeScript;
this is a Python sync wrapper fix with no new API or naming rules.
Validation:
- Before the fix, three namespaced sync cases failed while root and
async controls passed. All six cases now pass through the real SDK and
mock HTTP endpoint.
- Covers default/explicit root, nested paths, valid
dotted/mixed-case/dashed names, and repr after pickle with the server
stopped. Requests remain limited to the original table open.
- Rebuilt current editable bindings in the repository's uv environment
with Python 3.12 and pinned Rust 1.97.0 through mbx.
- Ruff check and format check pass for the Python package (178 files).
- Required repository-wide Ruff check reports 48 pre-existing
diagnostics in six `docs/web-tests` files; the repository-wide format
check reports nine unchanged documentation files. Their source matches
the base commit. No lint suppression or documentation cleanup is
included.
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
The only coverage of the fragment-swap commit's index retention used a
BTree index. Vector is the type a view is usually built for and the one a
rebuild strains hardest, since every fragment the index was trained over
is replaced in the same commit.
The case also pins down what retention does and does not mean: the index
definition survives, but it covers none of the swapped-in rows, so the
view falls back to a flat scan until maintenance rebuilds it. Asserting
`num_unindexed_rows` keeps that from being mistaken for a regression
later.
Running a Function body in a local test or another engine currently
requires discovering undocumented APIs. Document direct calls to
decorated Functions and classes, including initialization fields,
instance reuse, and cleanup, and explain how they differ from registered
`FunctionVersion` applications.
Add examples of `BlobReader(read_range, size)` backed by an in-memory
byte-range callback, including cursor-independent reads. The local
examples and rendered API documentation were validated on EC2 at
`df6076e1ae605f425a380d282c4f25b845a3165f`; integration with current
main leaves these documented call paths unchanged.
Fixes ENT-3561.
Packages passed through `@udf(code=[...])` silently omit non-Python
resources, so registration can appear successful before a worker fails
to open a data file. Warn during function preparation with the omitted
paths and document the source-only boundary next to `code=`. The
diagnostic explains how to install a distribution that includes the data
and remove the package from `code=` to avoid shadowing that
installation.
Python cache files do not produce warnings. Excluded directories are
reported at their boundary, preserving the existing traversal and
Python-source artifact format. This implements the
warning-and-documentation alternative in ENT-3492; it does not add
resource uploads.
On EC2, the regression fails on baseline `f9389f113` because no warning
is emitted; all 60 packaging tests pass at `e23445b96`, using the
matching native extension built from unchanged Rust sources.
CI-scoped Python lint with Ruff 0.9.9 passes. Whole-site Mintlify
validation reports the same four snippet-import warnings on the baseline
and this head; repository-root Ruff likewise reports the same 48
findings under `docs/web-tests`. These checks are not clean passes, but
their failures do not involve this change.
Fixes ENT-3492
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Add a Python regression test for merging metadata-free data into a table whose target schema has metadata. The test calls the native `execute_merge_insert` binding directly so it exercises the native partial-schema comparison without the public merge path normalizing source metadata.
The test verifies that merge insert succeeds with the expected rows and preserves the target schema metadata.
Fixes#2058
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
Xuanwo
Jack YeandClaude Sonnet 5
Yang Cen
Wyatt Alt
Brendan Clement
Lance Release
Bruno RamirezandClaude Sonnet 5
Jay Pokale
Gaétan
Bilal AtiqueandBilal Atique
importcppandimportcpp
Fede Kamelhar
LeoandLeo
lancedb-gatefixer[bot]
Srijan Keshri