"Storage | Yes / No" and an OSS tag on a page called Storage read as if
Enterprise had no storage. The difference the code shows is narrower:
the embedded library configures its storage backend from the
connection, while a deployment sets up its own and a db:// connection
passes none of those options except an Azure storage account name. The
page is now Embedded storage configuration, says so first, and points
deployments to their own setup. The declaration is unchanged.
The page declared OSS unavailable, so the comparison read "Sampling |
No / Varies" and the sidebar tagged the whole page Enterprise, while its
last section described SDK sampling that works in both offerings. It is
now titled SQL sampling, so the row and tag describe TABLESAMPLE, and
the SDK alternative is a section of its own, marked for both offerings,
under its existing anchor.
That section's example passed Permutation.identity(table) to a
DataLoader, which reads rows in stored order. It now builds a seeded
shuffle and a random split with permutation_builder, checked locally
against the baseline SDK, including through a DataLoader.
The consistency page said reads after any write carry a freshness floor,
so you read your own writes without extra configuration, and that the
floor follows the last write. The pinned client records a write's
version as x-lancedb-min-version, which Sophon does not read (neither at
75f4bf28 nor on main); its time floor comes from the interval and from
checkout_latest() and restore(); and its version watermark follows the
x-lancedb-version a response reports. Whether that watermark covers a
write depends on the version the write's response reports, which the
code alone does not settle.
The page now describes those three mechanisms and says how to be sure a
read reflects your write: an interval of zero, or checkout_latest()
after writing. The guarantees page keeps its read-your-writes promise
for the embedded library, on the handle that wrote.
The cleanup and guarantees pages said a tag or branch pointing at a pruned
version stops resolving, while the table page said tagged versions are
exempt and quietly kept. The pinned Lance (ae2903729f02, the commit in
Cargo.lock and in Sophon's Lance submodule) keeps a tagged version
whatever its age, but with its default policy refuses to clean up while a
tag points at a version older than the window, raising an error that
names the tag. optimize() passes that default, so it raises after
compacting, prunes nothing and skips its index update until the tag is
deleted. Sophon's cleanup job uses the same default.
Branches are retained separately: cleaning the parent keeps the version a
branch was forked from and the files the branch reads, while the
parent's other old versions are pruned. The pages now say both, checked
locally against the baseline SDK.
Sophon main now refreshes an eligible materialized view incrementally,
appending what an append-only source added since the last refresh, a day
after the change that made every refresh a full rebuild. So the
materialized view page and the REFRESH reference no longer say recent
versions rebuild in full: they say recent versions run views as SQL jobs,
need no stable row ids and take no refresh options, and that a full or
incremental refresh depends on the version and the view's query. COPY is
described with the same "recent versions" wording.
Four summaries pointed at the table or sections below, which reads wrongly as a row of the comparison. They now stand alone, and insert links to its bulk-ingest section.
The comparison now lists pages across the documentation, grouped as in
the navigation. The page says what Yes, No and Varies mean, that pages
spanning several features are left out, and that the SQL row stands for
the SQL reference.
Storage configuration from the connection is embedded only, since a
deployment configures its own. Training data loading, PyTorch, embedding
functions, pandas, Polars and Pydantic work with local and deployment
tables, and the Hugging Face Hub integration is embedded only.
A local table cannot be pickled for DataLoader workers, so the PyTorch
page says to wrap it in a Permutation; a LazyFrame over a whole table,
to_pandas in the LangChain example and to_arrow in the DuckDB note are
local only; and query text is embedded in the client for both offerings.
SQL is unavailable in the embedded library and varies on a deployment,
which serves it only when its Flight SQL endpoint is enabled and
reachable, and the SDK has to be told where that is. The page's opening
badge goes. The SDKs page declares both offerings.
Neither SDK aggregates or joins, so the parity tables no longer say they
do, and views can be created through the SDKs on a deployment. Pages that
said vector search has no SQL form now point to the version-dependent
SQL form, and COPY, which newer deployment versions refuse, is marked
as such in the reference.
Databases, namespaces, jobs and data evolution work in both offerings.
Materialized views, computed columns and column refresh work in the
embedded library and vary on a deployment, which has to enable them.
Views and functions exist only on a deployment that enables them, and
aggregation, joins, windowing and sampling are SQL, so they need one
with SQL enabled.
Sections now say where they differ: views have SDK methods on a
deployment, dropping a function needs a further setting, a local job
handle runs in-process and has no history, create_index and
refresh_column behave differently against a deployment, and the
overviews no longer call computed columns and materialized views
available everywhere.
Creating, writing, updating, merging and deleting rows, managing tables,
branches and read consistency work in the embedded library and on a
deployment, and those pages now say so. The SQL forms they mention need
a deployment.
Where the offerings differ, the section says how: a deployment reuses an
existing table without comparing schemas, a merge insert can be refused
past a configured number of unindexed rows, space is reclaimed by
optimize() locally and by the deployment's compaction and configured
cleanup otherwise, renaming depends on the catalog, a deployment clones
shallowly only, drops all tables only locally, keeps freshness state per
table handle and defaults freshness to its own setting, and refuses to
delete a branch main depends on. table_exists works only on a namespace
connection.
Guarantees and limits span both offerings, so they carry section notes
rather than a label: reads, retention, index freshness and compaction
differ on a deployment, and the name and job limits are a deployment's.
Indexes are built in the embedded library and on a deployment, which
builds them in the background, keeps most of them current, and builds
only IVF_FLAT, IVF_PQ, IVF_RQ and IVF_HNSW_SQ vector indexes. Compaction
runs on both, by optimize() locally and in the background on a
deployment; cleanup on a deployment runs only where its configuration
enables it, so it varies; background jobs exist only on a deployment.
Sections that described one offering as if it were both now name it:
optimize() is embedded only, the brute-force limit and automatic IVF_PQ
are deployment settings, FM indexes get no background updates, a
remainder below the deployment's threshold can keep wait_for_index from
finishing, GPU builds on a deployment are IVF_PQ with a configured GPU
count, list_indices on a deployment omits num_unindexed_rows, and the
background jobs page no longer says every other job is one you start.
Search, full-text search, hybrid search, filtering, query plans and
reranking all work in the embedded library and against a deployment, and
now say so. Where the two differ, the section says how: a deployment
returns a query's whole result in one response, the synchronous Python
remote table refuses to_pandas, to_arrow and to_lance, probe widening
depends on the deployment's version, full-text indexes build in the
background and are kept current without optimize(), distributed scoring
can differ slightly, model files have to be present on the deployment,
plans carry deployment operators, and hybrid explain_plan is Python only.
Reranking runs in the application either way.
Authorization wrote objects as namespace:db/ns and table:db/ns/table,
which the deployment rejects; it writes namespace:db:ns and
table:db:ns$table, nesting namespaces with $. The page now lists the
view, function and secret objects, CREATE_FUNCTION and CREATE_SECRET,
USAGE and UPDATE where they apply beyond tables, and what OPERATE and
USAGE on system allow. An a: subject names the principal holding a key
rather than a narrower grantee, creation falls back to the parent's
create privilege only for an unowned database or namespace, and a
listing page holds at most 100 entries.
The materialized view page said a deployment's refresh, options and
stable row id needs depend on version and configuration. Newer versions
rebuild the whole view on every create and refresh as a job, need no
stable row ids, take no refresh options, and have no REST route to create
one; earlier ones differed, and their views have to be recreated. The
page and the REFRESH reference now say so, without naming which version
a deployment runs.
Vector search no longer says it has no SQL form: some deployment
versions plan ORDER BY distance(...) LIMIT k as one.
basics/offerings explains the two offerings and holds the comparison the
assembled site generates from the pages' availability. basics/connect
shows local, object storage and Enterprise connections with the existing
tested connection snippets. Both follow the landing page in Getting
Started.
Vector search, materialized view refresh, authentication and authorization
now say in their frontmatter whether LanceDB OSS and LanceDB Enterprise have
what they describe, with one sentence on how. The assembled site renders
that as each page's label, its sidebar tag and its comparison row, so the
hand-written badges and opening paragraphs on the two security pages go.
The availability was checked against the SDK and the Enterprise server:
authorization is enforced only on a deployment configured for it, and a
deployment accepts OAuth tokens only with an identity provider configured.
Sections now say that only the Rust SDK sets approx_mode, that a deployment
may search exactly minimum_nprobes partitions, and that the SDK refuses
with_no_data, like source_version, on a db:// connection.
Lance beta and RC artifacts are moving to `https://maven.lance.org`. Add
that release repository to the Java parent POM so LanceDB can resolve
explicit preview versions from it, while keeping Maven Central available
for historical releases and other dependencies.
Validated resolution of `org.lance:lance-core:14.0.0-beta.3` from a
fresh Maven cache using the actual `lancedb-core` POM. Maven recorded
the new `lance` repository as the source of both the POM and JAR. The
pinned Lance version and LanceDB's own publishing workflow are
unchanged.
Remove scaffold that accidentally caused the post-merge Rust
license-header check to fail. Include `docs/web-tests/**` in the PR
workflow paths so future docs test changes run the license checks before
merge.
Validation: `cargo fmt --all`; `git diff --check`;
license-header-checker v1.5.0 using all four workflow commands on a
clean checkout; Cargo metadata confirms all six Rust example targets
remain unchanged.
Refuse user writes (add, insert, update, delete, merge_insert, column
changes) to a table carrying `mv.definition`, so a materialized view is
written only by refresh. Index, optimize, and computed-column operations
stay allowed. Sophon's REST write paths are handled separately in
ENT-2878.
Fixes ENT-2876.
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Lets a registered-Function column read another computed column (SQL or
Function). Resolves ENT-2780.
Nothing records which rows of a computed column have been filled, so a
refresh treats a null computed input as "not yet computed". It skips the
row and leaves the outputs null until a later refresh, after the input
is filled. Because the Function never receives a null from a computed
input, that input binds a non-nullable parameter.
The trade-off is that a computed input which is null by definition (a
SQL expression that yields null, or a Function row skipped on error)
leaves the downstream row unfilled.
## Changes
- `plan_function_application` and `ensure_binding_matches_schema` now
accept computed inputs and record them as non-nullable. They still
refuse inputs that are:
- unrecognized computed kinds,
- incomplete declarations,
- internal Function assignment columns.
- A new public `function_input_is_computed` reports whether an input
reads a computed column. A new public `computed_function_inputs` lists
the inputs a server refresh must find non-null before it computes a row.
- `ensure_supported_function_metadata` now refuses a schema whose
computed columns read each other in a cycle. A declaration can only read
columns that already exist, so only a schema supplied whole can contain
one.
- Dropping an upstream output while a downstream binding still reads it
is refused by the existing binding guard. This is now covered by a test.
The server-side refresh change (skipping rows that are not ready,
clearing them on recompute) lives in sophon and follows this PR.
## Testing
- `cargo test -p lancedb --features remote --lib -- computed refresh
freshness schema_evolution function`: 255 passed.
- `cargo clippy -p lancedb --features remote --tests -- -D warnings`:
clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sync remote string searches ignored the table's embedding function when
`vector_column_name` was omitted. `auto` silently fell back to full-text
search, while `vector` and `hybrid` failed with `No embedding function
for None`.
Resolve vector columns from the schema before creating the query
builder, using the same inference helper as local search. Automatic
string queries without embedding functions select full-text search
first, so text-only tables and tables with multiple unconfigured vector
columns remain searchable. Vector and hybrid searches still reject
ambiguous columns, infer numeric queries by dimension, and honor
explicit column selection. Structured full-text queries and empty
queries retain their existing behavior.
Regression coverage uses embedding metadata returned by a mocked remote
server rather than a mocked embedding-function lookup. It also checks
automatic and explicit full-text queries on schemas with zero, one, or
multiple vector columns and no embedding functions.
Validation after merging `main` at `87ecfeac`:
- Confirmed the original embedding and ambiguity failures before the
fix, and reproduced the automatic FTS inference failures before
restoring the fallback.
- Remote, query, hybrid, and embedding test files through the uv-managed
environment: 240 passed, 4 skipped, 9 deselected (`not slow and not
s3_test`).
- Python extension built with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `ruff check .` and `ruff format .` from the repository root, using
CI's Ruff version 0.9.9.
- `cargo fmt --all`.
Fixes#4377
<!-- lance-gatekeeper-fix:v1 agent=c167c6c89d857342e5d7415b08142589
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote vector serialization unwrapped `serde_json::Number::from_f64`,
which returns `None` for NaN and infinity, causing a Tokio worker panic.
Validate vectors after conversion to Float32 in the shared `nearest_to`
/ `add_query_vector` builder so local and remote queries consistently
return `Error::InvalidInput` (mapped to `ValueError` by the Python
bindings). This also rejects finite Float64 inputs that overflow
Float32. Make remote serialization fallible to protect requests
constructed directly, including single-vector queries and both remote
batch-query paths.
Validation:
- All 116 query-related Rust tests passed with the `remote` feature,
including the three new regression cases.
- The three regression cases fail with the original production code and
pass with the fix. Coverage includes NaN, positive and negative
infinity, Float16/32/64 Arrow inputs, Float64 overflow, and rejection
before any HTTP request.
- `cargo fmt --all -- --check`, `cargo check --profile ci --locked -p
lancedb --features remote --tests --examples`, and `cargo clippy
--profile ci --locked -p lancedb --features remote --tests --examples --
-D warnings` passed.
Fixes#4375
<!-- lance-gatekeeper-fix:v1 agent=97d6925d4af7cd09767da8b552c9bcf3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Sync remote `create_table(..., embedding_functions=...)` discarded the
supplied configuration and only logged an unsupported-feature warning.
Schema-only creation therefore lost the metadata needed by `add()` to
embed text, while text-only initial data produced no vector column.
Forward the configuration to the existing async creation path, remove
the stale warning, and document the parameter. The shared path stores
the embedding metadata and generates vectors for initial data and
subsequent additions.
Add a regression test against the mocked remote HTTP API for both
schema-only and text-only creation. It checks the outgoing Arrow schema
metadata, the metadata returned by the remote table, inferred vector
types, and generated vectors during creation and `add()`.
Validation:
- Both regression cases failed before the fix because the outgoing
embedding metadata was missing, then passed after the fix.
- `uv run --extra tests --extra dev pytest
python/tests/test_remote_db.py python/tests/test_embeddings.py -q -m
'not slow and not s3_test'`: 82 passed, 4 skipped, 8 deselected.
- Repository-wide `ruff check .` and `ruff format .` passed using the
uv-managed environment; `git diff --check` passed.
Fixes#4376
<!-- lance-gatekeeper-fix:v1 agent=a6816b1f3335438bd8ad192f36395213
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote `create_table` with `exist_ok` opened an existing table without
validating the supplied schema, so Python and TypeScript silently
accepted mismatches.
Capture the request schema before consuming its data stream, compare it
with the opened table's schema, and return the same `Error::Schema` and
message as the local backend when they differ. The existing open-table
callback continues to run before validation.
Regression coverage exercises matching schemas and differences in field
name, type, and nullability, for both empty and data-backed creation
with cached and uncached table opens. All 12 mismatch cases failed
against the original implementation; all 16 cases now pass.
Validation:
- `cargo test --profile ci --features remote -p lancedb --lib
remote::db::tests`: 113 passed.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`: passed.
- `cargo fmt --all` and `git diff --check`: passed.
Fixes#4378
<!-- lance-gatekeeper-fix:v1 agent=5781a39ba124cf9094d36715730342b4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote handles at the latest version have no checkout version, so
`restore()` sent `{"version":null}` and surfaced a server
deserialization error. Validate the checkout version before constructing
the request and return `Error::InvalidInput` with the same message as
local tables. This shared Rust fix covers Python sync/async and
TypeScript callers.
Add a regression test proving that restore sends no request on a fresh
handle, after `checkout_latest()`, or after a successful restore clears
checkout state. It also verifies that a valid restore sends the
checked-out version. Update the schema-cache and branch restore tests to
check out a version before restoring and require success.
Validation:
- The new regression failed against the original implementation when the
mock received a null version.
- `cargo test --locked --profile ci --features remote -p lancedb --lib
restore` — 6 tests passed.
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --quiet --locked --profile ci --features remote -p
lancedb --tests --examples` passed.
Fixes#4381
<!-- lance-gatekeeper-fix:v1 agent=cc67544dfcbb46d631762752d55b39e3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote update responses report their count as `updated_rows`, but the
client only recognized `rows_updated` and silently defaulted the count
to zero. Python sync/async and TypeScript both use this Rust result.
Accept `updated_rows` as a Serde alias while retaining `rows_updated`
and compatibility with older responses. Extend the remote mock test to
cover both keys, zero and missing counts, returned versions, and empty
responses, with and without a predicate.
Validation:
- Both nonzero `updated_rows` regression cases failed before the fix (0
instead of 5); all 11 focused remote tests pass after it.
- All 3 native update tests pass.
- `cargo fmt --all`
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`
Fixes#4443
<!-- lance-gatekeeper-fix:v1 agent=423249073212c9840d96ccf6d4452905
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.
- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.
## Testing
- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
BaseTableAdapter wraps every scan in MetadataEraserExec, which reported
an unknown cardinality effect. Lance's CountPushdown only looks through
nodes that keep every row, so a SQL COUNT(*) over the adapter always
scanned, even with the rule registered. The eraser now reports Equal.
with_new_children kept the old schema and panicked when the new child's
schema differed; it now rebuilds from the child.
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
CatalogConnection.list_databases should return an iterator so that users
don't have to manually re-implement pagination. For those who do want
more control over pagination, add num_page_results and page_token to the
Iterator object, and also allow the iterator object to be created at a
specific page token.
Add Reverie banner to the top of README until event ends.
Additionally, I also removed the "LanceDB Cloud" skinny banner since
we're not promoting that right? Should we modify the larger LanceDB
modify to not mention "Try LanceDB Cloud"?
Closes#4270.
## Summary
Blob v2 columns now accept more of the containers TypeScript code
commonly has, both as the column value and inside `{ data }` / `{ uri
}`:
| Input | Handling |
|---|---|
| `ArrayBuffer` | `new Uint8Array(value)`, no copy |
| `URL` | stored as `value.href` |
| `Blob` / `File` | `await value.arrayBuffer()` in the async write
paths: `createTable`, `add`, `mergeInsert` |
`makeArrowTable` is synchronous, so it can't read a `Blob`. Given one,
it throws an error that names the field and row and points to the async
APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a
null blob.
### Types
`BlobInput` is now the accepted input type. It is widened to the types
above and exported from the package entry point with `BlobData` and
`BlobUri`, so callers can annotate rows and `tsc` catches bad inputs.
The struct forms are a union that requires exactly one of `data` /
`uri`. The old normalized `{ data, uri }` shape is renamed to
`BlobValue` (internal).
### Implementation
- `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`),
`resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes
first.
- `arrow.ts`: `mapBlobInputs` walks only the blob positions in the
schema, including blobs nested in structs and lists. It copies records
and arrays only where a value changed, so caller data is never mutated.
`resolveBlobInputs` runs before `makeArrowTable` in `convertToTable`
(used by `add` / `mergeInsert`) and in `createTable`. Each distinct
`Blob` is read once.
- `makeArrowTable` now coerces every blob value to the full `{ data, uri
}` struct **before** schema inference. This also fixes two bugs that
existed before this change:
- Rows mixing plain bytes with `{ data }` failed inference.
- A `{ uri }`-only struct produced a blob struct missing its `data`
child.
Blob validation errors now include the field path and row.
- Also fixed along the way:
- `add` / `mergeInsert` on a table with an embedding function rejected
most blob values, because `ensureNestedFieldsExist` filled blob columns
in like ordinary structs. It now skips them.
- With a schema, a struct field that is `null` in one row and a record
in a later row failed inference. Primitives and arrays given for a
struct field are now rejected instead of being written as an all-null
struct.
- Blob structs with keys other than `data` / `uri` are rejected, which
catches typos. Class instances with `data` / `uri` are handled like
plain objects.
## Testing
- Unit tests in `__test__/blob.test.ts`: coercion of the new types,
rejection and error messages for `Blob` in the sync path, nested lists
and structs, `resolveBlobInputs` (only blob columns touched, caller data
untouched, shared `Blob` read once).
- Table tests in `__test__/table.test.ts`:
- `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` /
`Blob` / `File` bytes through `fetchBlobs`.
- `URL` and `{ uri: URL }` are added to the external-blob round-trip
test from #4317 (`allowExternalBlobOutsideBases`).
- Full `jest` suite passes locally after the rebase (938 passed). `pnpm
lint` is clean and `pnpm run docs` has been regenerated.
Streaming inputs (`ReadableStream`, Node `Readable`, async iterables)
remain out of scope, as the issue says.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv
---------
Co-authored-by: Claude <noreply@anthropic.com>
Adds a "🚨 Critical Fixes" section to the release notes, directly below
Highlights, for PRs labeled `critical-fix`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4404.
AsyncTable.list_versions split the nanosecond timestamp with float
division/modulo (`ts // 1e9`, `ts % 1e9`).
At ~1.8e18 a float can't hold every integer, so the microseconds drifted
(.274000 came back as .273999).
Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead.
Only the precision fix is included here. Timezone-aware UTC (option 1 in
#4404) is on hold for now,
and I can add it here or in a follow-up depending on what they prefer.
Added test_list_versions_timestamp_precision using the value from the
issue (1790989429274000000), expecting microsecond == 274000.
Function column declarations and refreshes can return HTTP 404 when the
referenced Function has been deleted. The SDK currently translates those
responses into `TableNotFound`, hiding the server diagnostic and making
an existing table appear missing.
Preserve the HTTP error, status, and request ID for these two operations
while retaining schema-cache invalidation. Ordinary table operations
keep their existing `TableNotFound` behavior.
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.
Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.
Validation:
- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.
Fixes#4389
<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.
Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.
Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.
Validation:
- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.
Fixes#4388
BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.
<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Synchronous local Python `create_table` callers using `mode="exist_ok"`
received `ValueError` instead of creating or opening a table.
Related to [#2900](https://github.com/lancedb/lancedb/issues/2900). The
local wrapper rejected this mode before the shared Rust parser could
validate it. Removing that guard delegates mode validation to the
parser, and the sync and async docs now describe `exist_ok`. Remote
option gaps and the TypeScript table type gap remain outside this
change.
The added tests retain both calls from [@wjones127's
comment](https://github.com/lancedb/lancedb/issues/2900#issuecomment-5964167206).
This builds on the code @wjones127 posted there. Three regression tests
fail before the change, pass after it, fail with the original `db.py`
restored, then pass again on HEAD.
Tested on Linux x86_64 with Python 3.14.7. From the Python directory,
`uv run --no-sync --extra tests --extra dev python -m pytest
python/tests/test_db.py -q` reports 53 passed. `uv run --no-sync --extra
tests --extra dev python -m pytest python/tests -q -rs -m "not slow and
not s3_test"` reports 1428 passed, 34 skipped, 68 deselected, and one
failure: the existing `test_get_lsm_write_spec`, also failing on clean
base (1425 passed, one failed). Ruff 0.9.9 `ruff format --check .` and
`ruff check .` pass. Tests used worktree Python code with the released
`lancedb==0.40.0b12` native wheel. No native source build was run.
Co-authored-by: Will Jones <willjones127@gmail.com>
Local `clone_table` passed existing targets to Lance's clone
transaction, which entered the manifest-update path and failed with
`Clone operation should not enter build_manifest`.
Check the target dataset using the connection's storage options and
session before cloning, and return `TableAlreadyExists` when it exists.
Propagate other read errors. Add regression coverage for cloning onto
the source itself and onto a different existing table, with both
manifest layouts, verifying that rows and version histories remain
unchanged.
Validation:
- All four regression cases reproduced the reported internal error
before the fix.
- `cargo test --locked --quiet --profile ci --features remote -p lancedb
--lib test_clone -j 12`: 24 passed.
- `cargo fmt --all -- --check` and `git diff --check` passed.
- `cargo clippy --locked --quiet --profile ci --features remote -p
lancedb --tests --examples -j 12` passed.
Fixes#4387
<!-- lance-gatekeeper-fix:v1 agent=522c80785eeaaef146e8179c848b903e
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4390
Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.
Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.
Validation:
- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.
<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`AsyncTable.search()` and `search(None)` reached vector search without
assigning `vector_query`, raising `UnboundLocalError` on both local and
remote tables. Return a plain `AsyncQuery` before inferring the search
type, equivalent to `table.query()`, and update the overloads and
documentation to describe that return value.
Add six regression cases for omitted and explicit `None` queries on
local tables with and without vectors and on a mocked remote endpoint.
They verify scans, limits, filters, column projection, and the remote
request body.
Validation: all six regressions reproduced the original
`UnboundLocalError`; all 11 focused query tests pass with the fix,
including existing vector, full-text, hybrid, and sync scan coverage.
The uv development bootstrap, locked Ruff formatting/lint checks, and
focused Pyright return-type assertions also pass.
Fixes#4385
<!-- lance-gatekeeper-fix:v1 agent=bb9a7509f6dadde030d8113b172abc9a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Local synchronous `create_table(...,
enable_v2_manifest_paths=True/False)` failed with `TypeError` because
the deprecated argument was forwarded as a boolean in `storage_options`,
while the Rust binding expects string values.
Convert the argument to lowercase `"true"` or `"false"` before
forwarding it, preserving the deprecation warning. Add regression
coverage for both values with populated and schema-only tables, checking
stored data and the manifest mode after reopening.
Validation:
- All four regression cases reproduced the original `TypeError` before
the fix and pass afterward.
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_db.py -q --tb=short -m 'not slow and not s3_test'`: 54
passed.
- Repository-wide `ruff format .` and `ruff check .` completed
successfully.
Fixes#4384
<!-- lance-gatekeeper-fix:v1 agent=9c14955b90d0c12d6a46605ec5ef32c0
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Brendan Clement
Xuanwo
Wyatt Alt
Jack YeandBrendan Clement
Colin Patrick McCabe
Yuval LifshitzandClaude Opus 5.0
Lance Release
lancedb-gatefixer[bot]
XY Zhan
LanceDB RobotandWyatt Alt
ccmao1130
Will JonesandClaude
Aviral Saxena
Chi-Wei LeeandWill Jones