The SDK now sends CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW
to a db:// deployment as SQL, and refuses source_version on a remote
refresh. Current Sophon source accepts both statements, with CASCADE and
WITH (full = true), but only on deployments that enable materialized
views, and whether a refresh rebuilds the view or reconciles changes
depends on configuration. The pages said there is no CREATE MATERIALIZED
VIEW statement, that refreshes are incremental by default and that
CASCADE is not implemented.
The refresh, management and SQL reference pages now describe what the
SDK sends, say that materialized views are experimental and must be
enabled on a deployment, and leave refresh behaviour, CASCADE and the
job type to the deployment's version and configuration. The SQL and
management summaries list CREATE MATERIALIZED VIEW. No deployment was
exercised.
Main now rebuilds a materialized view completely on every refresh
(#4373): there is no incremental mode, no `full` argument and no
stable-row-ID requirement on the source. The example ran against the
SDK built from this branch (0.40.0b13) failed, because a refresh after
adding Bea writes both matching rows, not one.
The example now creates its source with the default options and expects
two rows written by the refresh, and its generated snippet is
regenerated. Separate tests cover `with_no_data` (one matching row on
the first refresh) and refreshing to an earlier source version. The
refresh page explains the full rebuild and keeps the #incremental and
#stable-row-ids anchors for the sections that replace them. Its version
note now says what the published 0.39.0 does differently, as observed
with that release.
The management, table, database, limits, architecture and data-types
pages no longer say a view's source needs stable row ids, and the
feature-engineering and tutorial summaries no longer call refreshes
incremental.
Brings the candidate up to main at bbc4aebe55, which rebuilds materialized views fully on refresh (#4373) and is version 0.40.0-beta.13, so the documentation and its examples can be checked against that SDK.
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.
- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.
## Testing
- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
Mint sends a folder URL to the first sidebar page under it unless that page
is the folder's index. With Security listed first, /enterprise opened
Authentication, while production, whose sidebar starts with the Enterprise
overview, serves the overview at /enterprise. Listing Deployment, which
starts with the overview, before Security restores that. The groups, their
pages and the redirects are otherwise unchanged.
The previous commit restored production's sentence so the example is
attributed to Geneva. This brings back the newer page's customer-facing
framing, that feature engineering costs nothing until it is used and never
copies the dataset or touches the videos, while keeping the Geneva link and
naming the API the code calls: @udf, add_columns and backfill. The code and
its timings, measured with Geneva, are unchanged.
The example calls Geneva's @udf, add_columns and backfill, but the sentence
before it described lancedb computed columns and Functions, which take none
of those parameters and have no backfill. This restores the production
sentence, which names Geneva and links its UDF guide; the code and its
timings are unchanged.
Production served the same LeRobot page at /training/lerobot and
/integrations/lerobotdataset. Only the integrations page carries over, so
the old Training URL now redirects to it instead of returning 404.
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
The TypeScript and Rust Quickstarts read the image from ../../docs/static,
the layout before the documentation moved into this repository. The file is
docs/web/static/assets/images/quickstart/sir-lancelot.jpg, so the TypeScript
test failed with ENOENT and the Rust example panicked at the read. Both now
resolve the image from the example's own directory (__dirname and
CARGO_MANIFEST_DIR), and the two Quickstart exports are regenerated with
docs/web-tests/mdx_snippets_gen.py.
ethnum 1.5.2, which jsonb brings in through lance-arrow, transmutes () into
TryFromIntError and no longer compiles on Rust 1.97.0, the toolchain this
repository pins, so none of the Rust examples built. 1.5.3 builds the error
from a failed conversion instead. Only that lockfile entry changes; lancedb
stays at 0.31.0.
CatalogConnection.list_databases should return an iterator so that users
don't have to manually re-implement pagination. For those who do want
more control over pagination, add num_page_results and page_token to the
Iterator object, and also allow the iterator object to be created at a
specific page token.
Production serves its "Query with SQL" page at /search/sql, and the same
page at /search/sql/index; both URLs return that page on
docs.lancedb.com. Only the /index form was redirected to /sql, where that
guide now lives, so the canonical URL would have 404ed. It now redirects
to the same destination.
Carries over production's #368 (deploy-freeze 113db11c), which landed after
this branch's gallery was split into a page per demo. Each demo gets a page
in that layout with production's card, badges and link, and a row in the
overview table and the sidebar. The two hero images are byte-identical to
production's.
Carries over production's #360 (deploy-freeze 78d55c2) where it was only
partly ported. The navbar's Support link still opened
mailto:contact@lancedb.com, and the home page linked the contact page but
showed the email address as link text. Both now match production. The
other passages #360 changed are already ported at their new paths.
Add Reverie banner to the top of README until event ends.
Additionally, I also removed the "LanceDB Cloud" skinny banner since
we're not promoting that right? Should we modify the larger LanceDB
modify to not mention "Try LanceDB Cloud"?
The note dated the new behavior to "SDK 0.40", but it was verified on the
development SDK 0.40.0b12 built from main, and no stable 0.40 has been
released. The note now names that build and keeps the comparison with
the published 0.39.0, which creates the view empty, has no with_no_data,
and writes both matching rows on the first refresh.
The opt-in Enterprise walkthrough overwrote the table `function_demo`,
registered the Function `double`, then dropped both. On this SDK
drop_function removes the named Function, not only the version this run
created, so a run against a shared database could replace or remove
existing objects.
The walkthrough snippets keep their names. The test registers a copy of
`double` under a name unique to the run, creates the table under such a
name, and drops only those two objects. The table is no longer created
with mode="overwrite", so even a name collision cannot replace an
existing table. The remote test has not been run.
The page said every Function output must be non-nullable. On this SDK
only a scalar output must be. A named-struct output may declare nullable
fields: whether a row has a result at all is the struct's own validity,
which Enterprise keeps in a separate internal column when it expands the
result into sibling columns.
test_function_output_nullability checks both rules against the SDK,
including that a struct declared as a single nullable field is refused.
maintenance/background-jobs.mdx links to /management/function#drop, which
did not exist. The section follows drop_function and drop_function_async
on main: dropping removes the name and its object, and deleting the
content can finish later as a background job.
Found by `mint broken-links --check-anchors` on the assembled site. Three
point at sections that were renamed or moved: the tag-versioning heading,
the branches comparison table, and the table-creation patterns, which now
live on the Create a table page. Three named sections that no longer exist
and now link to the page they were on.
Five pages still said a refresh never revisits a filled row and that a
changed input needs the column dropped and declared again. Since
lancedb#4161 a refresh also recomputes rows whose inputs changed, so the
quickstart, the feature-engineering overview, Guarantees, and the SQL
REFRESH and UPDATE pages now say that.
Retry claims are qualified the same way. A local refresh commits the column
once, and a deployment's job commits finished batches. A retry fills what is
still empty or stale, but can recompute values that were already written (a
refresh whose input record was not saved recomputes next time), so no page
promises that a retry costs only the unfinished rows.
Since lancedb#4180, create_materialized_view runs the initial refresh
before it returns (with_no_data=True skips it), so the page's first manual
refresh now writes 0 rows, not 1. The example now reads the populated view
and refreshes after a source change, and its snippets come from
docs/web-tests/py/test_materialized_view.py. A note scopes the behavior:
0.39 and earlier create the view empty.
Pages that said a refresh computes the rows now say it brings them up to
date, and the SQL note no longer says an incremental refresh covers only
added rows.
Porting the upstream changes replaced snippets/indexing.mdx with a hand
copy from lancedb/docs, which dropped six exports the vector-index page
still imports (`mint validate` warns on each) and added
PyQuantizationCustomParams, which had no source here.
The quantization test region is ported from lancedb/docs and the module
is regenerated, so every export comes from a test again. The regenerated
PyQuantizationCustomParams matches the copy byte for byte. Also fixes
"convetion" and "entires" on the page.
Two pages imported PySchemaAddSetup under the alias SchemaAddSetup, and
Mint resolved only the first, failing `mint validate` on
management/table.mdx. Both pages now import it unaliased.
TsAddProgress was added to the generated tables snippets by hand in
lancedb/docs#243 and dropped by the next regeneration, leaving
ingestion/insert.mdx with a missing import. Its example is restored
unchanged as a test in docs/web-tests/ts/tables.test.ts, which checks
that the callback reports the finished write.
"explictly" and "Additionaly", in snippet sources and their generated
snippets. integrations.mdx is updated in place: regenerating it whole
would also drop PyRerankingWatsonxUsage, which has no source.
`view.table()` and `view.definition()` raise TypeError on a synchronous
view; both are properties. An async view reads its definition with
`await view.definition()`.
Since lancedb#4161 a refresh recomputes rows whose inputs changed, and it
records what it computed from in a second, metadata-only commit, so the
example's version is 4. A local refresh commits once, so a failure keeps
none of its values; the page no longer says each row commits as it
completes.
The guide applied the local decorated callable to `table["value"]`, which
raises TypeError: a table is not subscriptable, and add_columns takes an
application of a registered version, not the callable's result.
The guide now walks one example from definition to filled column: define
`double`, call it locally, connect to Enterprise, create a table whose
`value` column matches the parameter's type and nullability, register the
function, apply `version(value=col("value"))`, refresh, wait, and read the
values back. Its snippets come from docs/web-tests/py/test_function.py.
The local tests in that file check what `@udf` sends for registration, how
a registered version binds a column, and that a local connection refuses
both registration and Function columns. The Enterprise walkthrough runs
only when LANCEDB_URI, LANCEDB_API_KEY and LANCEDB_HOST_OVERRIDE are set.
The page also stops promising that a retry after a failed refresh costs
only the unfinished rows.
Closes#4270.
## Summary
Blob v2 columns now accept more of the containers TypeScript code
commonly has, both as the column value and inside `{ data }` / `{ uri
}`:
| Input | Handling |
|---|---|
| `ArrayBuffer` | `new Uint8Array(value)`, no copy |
| `URL` | stored as `value.href` |
| `Blob` / `File` | `await value.arrayBuffer()` in the async write
paths: `createTable`, `add`, `mergeInsert` |
`makeArrowTable` is synchronous, so it can't read a `Blob`. Given one,
it throws an error that names the field and row and points to the async
APIs or `new Uint8Array(await value.arrayBuffer())`. It does not write a
null blob.
### Types
`BlobInput` is now the accepted input type. It is widened to the types
above and exported from the package entry point with `BlobData` and
`BlobUri`, so callers can annotate rows and `tsc` catches bad inputs.
The struct forms are a union that requires exactly one of `data` /
`uri`. The old normalized `{ data, uri }` shape is renamed to
`BlobValue` (internal).
### Implementation
- `blob.ts`: `normalizeBlobInput` (sync: `ArrayBuffer`/`URL`),
`resolveBlobInput` (async: `Blob`/`File`). `coerceBlobValue` normalizes
first.
- `arrow.ts`: `mapBlobInputs` walks only the blob positions in the
schema, including blobs nested in structs and lists. It copies records
and arrays only where a value changed, so caller data is never mutated.
`resolveBlobInputs` runs before `makeArrowTable` in `convertToTable`
(used by `add` / `mergeInsert`) and in `createTable`. Each distinct
`Blob` is read once.
- `makeArrowTable` now coerces every blob value to the full `{ data, uri
}` struct **before** schema inference. This also fixes two bugs that
existed before this change:
- Rows mixing plain bytes with `{ data }` failed inference.
- A `{ uri }`-only struct produced a blob struct missing its `data`
child.
Blob validation errors now include the field path and row.
- Also fixed along the way:
- `add` / `mergeInsert` on a table with an embedding function rejected
most blob values, because `ensureNestedFieldsExist` filled blob columns
in like ordinary structs. It now skips them.
- With a schema, a struct field that is `null` in one row and a record
in a later row failed inference. Primitives and arrays given for a
struct field are now rejected instead of being written as an all-null
struct.
- Blob structs with keys other than `data` / `uri` are rejected, which
catches typos. Class instances with `data` / `uri` are handled like
plain objects.
## Testing
- Unit tests in `__test__/blob.test.ts`: coercion of the new types,
rejection and error messages for `Blob` in the sync path, nested lists
and structs, `resolveBlobInputs` (only blob columns touched, caller data
untouched, shared `Blob` read once).
- Table tests in `__test__/table.test.ts`:
- `createTable`, `add`, and `mergeInsert` round-trip `ArrayBuffer` /
`Blob` / `File` bytes through `fetchBlobs`.
- `URL` and `{ uri: URL }` are added to the external-blob round-trip
test from #4317 (`allowExternalBlobOutsideBases`).
- Full `jest` suite passes locally after the rebase (938 passed). `pnpm
lint` is clean and `pnpm run docs` has been regenerated.
Streaming inputs (`ReadableStream`, Node `Readable`, async iterables)
remain out of scope, as the issue says.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012U4KdHofT5edv3LcXHXbLv
---------
Co-authored-by: Claude <noreply@anthropic.com>
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.
Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.
Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.
Validation:
- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.
Fixes#4388
BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.
<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Empty local TypeScript queries returned a zero-row Arrow table with no
columns because `toArrow()` built the table solely from record batches.
Local streams can yield no batches, while remote streams often supply an
empty batch carrying the schema.
Expose the executed stream's schema through the native iterator and
decode it into an empty Arrow table when no batches arrive. This
preserves projected and search columns, types, nullability, and metadata
without executing another query. Document the empty-result behavior.
Validation:
- Eleven regression cases failed with the original `toArrow()`
implementation; all twelve now pass, including remote schema-only and
empty-batch responses.
- All 1,075 tests in the query, remote, Arrow, and table suites pass.
- Native build, TypeScript compilation, Rust formatting and Clippy (`-D
warnings`), TypeScript lint/formatting, and documentation generation
pass.
Fixes#4386
<!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Thanks @vincentkoc for starting this! I was just looking into this
myself. I started with your PR and added:
- make arrow 21 the default
- widen the peer range to <22 so if they release 21.3 that'll be
included
- have to do skipLibCheck in tsconfig.json, because of flatbuffers 25
which needs TypeScript 5.7
- fix override in Symbol.hasInstance
- support new types in Arrow 20-21 (LargeList, Utf8View, BinaryView,
IntervalMonthDayNano)
- arrow-compat job in nodejs.yml that runs tests with arrow 19 and 20
(adds 2 sec)
- put the list of arrow types in one place
- update NODEJS_THIRD_PARTY_LICENSES.md
- regenerated typescript docs (sorry about reorderings, one time cost)
- reformat arrow.test.ts (sorry, almost all whitespace, thanks to biome.
git diff -w shows 110 lines edited)
- fixed Sliced Utf8View / BinaryView serialization error, by just
converting them to Utf8 and Binary (as our rust code would anyway)
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Closes#4288.
Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange`
and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an
optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When
the signal aborts, the call rejects with `signal.reason` (an
`AbortError` unless the caller gave a reason) and the native future is
dropped, so the read stops instead of running to completion. A signal
that is already aborted rejects before any read starts.
How it works:
- On the Rust side, these five methods are now sync `#[napi]` methods
that return `env.spawn_future(...)`. The future is wrapped in
`futures::future::abortable`, and the abort handle is registered with
napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so
it can't be an argument of an `async fn`, which is why the methods
changed shape. The TypeScript signatures are unchanged apart from the
new optional argument.
- napi-rs installs its own `onabort` on the signal it receives, which
would overwrite a handler the caller set. So the TypeScript wrapper
passes a private `AbortController` signal to the native call for each
request, forwards the caller's abort to it with `addEventListener(..., {
once: true })`, and removes the listener when the call settles.
Tests in `table.test.ts`:
- an already aborted signal rejects on all five methods, and the handle
still reads afterwards
- a custom abort reason is what the call rejects with
- a read with a signal that never aborts returns normally and leaves no
`abort` listener on the signal
- aborting a 32 MiB `readRange` right after it starts rejects with
`AbortError`, and the same handle reads again afterwards
The Node docs under `docs/src/js` are regenerated.
I didn't add a test for where the `read()` cursor ends up after a
mid-flight abort on a local table. The remote `BlobFile` documents that
a cancelled read leaves the cursor unchanged, but I haven't confirmed
the same for the native Lance handle, so the docs don't claim it.
Closes#4300.
The TypeScript `BlobFile` only had `size()`, `read()` and `readRange()`,
while the Rust `BlobFile` behind it also has `seek`, `tell`,
`read_up_to`, `read_ranges` and `close`. This exposes those on the Node
binding:
- `read(maxBytes?)` reads at most `maxBytes` from the cursor (Rust
`read_up_to`). Without the argument it reads to the end, as before.
- `readRanges([{ start, end }, ...])` reads several half-open ranges and
returns one buffer per range, in order. Like `readRange`, it does not
move the cursor.
- `seek(position)` and `tell()` move and report the cursor.
- `close()` releases the handle, and `isClosed()` reports it. Every
read, `seek` and `tell` after `close()` fails with "blob file is already
closed". A second `close()` does nothing. The binding keeps its own
`closed` flag, the same way the Python `BlobFile` does.
Offsets and lengths are `bigint`, like `readRange` and `size()`.
`BlobRange` is exported as a type.
The new tests in `table.test.ts` cover chunked reads with
`read(maxBytes)`, `seek` and `tell`, a negative seek position,
`readRanges` order and cursor, a `readRanges` end past the blob size,
and reads after `close()`. The Node docs under `docs/src/js` are
regenerated with `pnpm run docs`.
I left the `ReadableStream`/async iterator idea from the issue out of
this PR. The AbortSignal work in #4288 touches the same methods, so it
can follow on top of this.
## Cause
Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding
the server's 1024-row or 64 MiB byte cap. A single blob over the byte
cap could not be read through this method.
## Fix
- Send at most 1024 row IDs per request and split batches further when
the server reports the blob-byte cap, preserving input order and nulls.
- Resolve one exact dataset version before multi-request reads. If an
initial single request hits the byte cap, resolve the version before any
successful split or Range read; use that version for every resulting
request.
- Read an individual oversized blob through the existing Range route.
- Document the per-request caps and transparent handling in the Rust,
Python, and TypeScript APIs and generated JS docs.
## Validation
- Focused Rust `fetch_blobs` tests: 26 passed, including row-count,
byte-cap, Range fallback, and exact-version assertions.
- Rust clippy and formatting passed.
- Focused Python blob tests: 13 passed; Ruff lint and formatting passed.
- Node build, lint, docs generation, and focused `fetchBlobs` test
passed.
Fixes#4275
<!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
SQL now leads with why you'd reach for it beyond analytics: it's served over
Flight SQL, so it's the way in for any language without a native LanceDB SDK,
and broadly every public SDK method has a corresponding SQL statement. Closes
by pointing at the SQL Reference as the spec that defines the precise
behavior behind both the SQL surface and the SDK operations it mirrors.
Moving both into Getting Started (before Agentic Skills) means a reader knows
what the API surface even is before being told how to hand it to an agent.
FAQ content is redistributed into the pages it actually belongs to: license
and Parquet-comparison notes into the Lance format page, scale guidance into
Architecture, index-necessity and refine_factor guidance into the vector
index page, the multiprocessing/fork gotcha into SDKs and SQL, and the
weak-consistency recommendation into Consistency. General query-performance
issues are dropped since Explain and Explain Analyze already cover that
ground.
The new Support page under Deployment carries the OSS Discord and Enterprise
support-email channels that used to live in Troubleshooting.
OSS docs now own the full usage content for features that only run against
an Enterprise deployment, since the client interface itself lives here;
Enterprise docs keep only the parts specific to its own behavior. Sophon's
overlay for these five pages is dropped since the OSS versions are now the
complete reference.
SQL Reference is restructured to match an object-type SQL manual (DDL by
object, then DML, then DQL, then Expressions) instead of the old flat,
loosely-ordered page list, absorbing the SQL client-connection guide and
FTS-via-SQL examples into it.
Indexing loses its wrapper group -- Overview, Vector Index, FTS Index, Scalar
Index and Build & Repair sit directly in Maintenance now, followed by
Compaction, Cleanup and Background Jobs at the same level.
Cleanup is new, split out of Compaction: pruning old versions is a distinct
Enterprise job (cleanup) from merging files (compact), even though optimize()
in OSS bundles both into one call. The retention, disk-usage and
failed-transaction material that used to live under Compaction's umbrella
moves with it.
Background Jobs is new and covers what nothing else does: the job types a
deployment runs on its own -- WAL/SPFresh maintenance, cache prewarming,
registry archiving and metering export, and the object-teardown job that
follows a namespace or view drop. Every entry is grounded against the
job-coordinator's public job_type constants, not the internal two-letter wire
codes used for registry storage. Five existing pages (Build & Repair, Table,
Function, Column Refresh, Materialized View Refresh) gain a one-line job-type
note rather than duplicating a page for job types they already document.
Storage folds into Deployment: storage/index (choosing a backend) and
storage/configuration (configuring one) merge into a single Storage page --
they never overlapped, one is the decision and one is the how-to -- and move
under Deployment along with Monitoring. Cloud Providers loses its nesting;
its two pages plus Azure sit flat, and the former Overview is retitled
Deployment Models so it doesn't collide with the group's own Overview entry.
Redirects cover every moved URL.
## Summary
Materialize a source-preserving dedup result from an indexed, pinned
snapshot with a short declaration. The default greedy policy keeps the
smallest unassigned snapshot row ID and removes only its directly
matching neighbors: For A < B < C, an A–B–C chain without an A–C edge
keeps A and C.
```python
from lancedb import vector_dedup
view = db.create_materialized_view(
"images_clean",
vector_dedup("images", column="phash", distance_threshold=4),
)
cleaned = view.table
```
The Python source expression captures the source version once at
creation and
waits for the materialized result by default. Use `dataset_version=...`
for an
explicit snapshot, `with_no_data=True` for declaration only, or
`create_materialized_view_async(...)` for a job handle. AsyncConnection
supports
the same expression. It uses the existing MV REST endpoints and registry
jobs,
without opening a SQL query connection. Source names are exact
root-namespace
table names. Additional select/where/limit options are rejected for this
source.
The SQL declaration remains available:
```sql
CREATE MATERIALIZED VIEW images_clean AS
SELECT * FROM vector_dedup('images', 7, 'phash', 4);
REFRESH MATERIALIZED VIEW images_clean;
```
This is the host-facing SQL entry point. The MV builder now accepts a
typed native vector source; the Rust SDK also supports the same
definition through `MaterializedViewDefinition::from_sql` and
`prepare_definition`; `view.refresh().execute()` executes it locally.
Hosts distribute its units using their existing job infrastructure. The
raw `vector_duplicate_pairs` source remains available separately when
callers want the pair relation.
## Implementation and contract
- Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native
partition primitive computes pairs from index representations without
source-vector aggregation or a Python pair loop.
- Extend partitioned MV refresh with two dependent waves. Physical
segment/partition tasks stream pairs through a bounded external sort,
select direct representatives and write per-source-fragment bitmap
artifacts. Source-fragment tasks start after every selection receipt
completes and stream retained original rows, including blob payloads,
into uncommitted output fragments.
- Reuse normal blob write coercion and row-mask scans, including stable
row-ID translation after compaction. Null vectors, isolated rows and
source deletions are preserved correctly. Conflicting representative
decisions across overlapping index scopes fail before publication.
- Expose `dependency_units()` and `write_refresh_partition_with_inputs`
alongside the existing plan/commit API. Receipts bind
snapshot/configuration/policy/destination generation. Only a complete
accepted set can publish; unfinished units restart, completed units can
be reused. There is no pair-position checkpoint.
- Use MV format 4 for dedup results and format 3 for native pairs;
Python and TypeScript metadata readers recognize both. No source
deletion or source row-ID migration occurs.
Only same-segment/same-partition edges are covered. Quantized scores use
symmetric represented-vector distances. Result materialization requires
V2 data storage and is initially a native MV source, not an ordinary
SELECT table function. Compressed vertex masks stay in memory; edge
sorting spills to local disk. Unreferenced artifacts use normal Lance
retention/cleanup and must be retained for the job's recovery window.
## Validation
AWS Linux validation:
- Native pair/result integration matrix passed: multiple
segments/partitions, exact thresholds, empty output, cross-batch data,
single-process pair parity, source preservation, lost receipts and
publication guards.
- Blob/null-vector/pinned-deletion tests passed with both row-ID modes,
including compaction that changes physical row addresses.
- Representative policy, forced external-sort spill and
conflicting/corrupt artifact tests passed.
- Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust
formatting and Clippy passed.
- Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native
bindings were rebuilt in project-local environments; formatting, lint
and TypeScript docs passed.
- Rust declaration doctests: 3 passed, including the complete local
dedup example.
- Companion host validation: 41 MV job tests passed (1 pre-existing
ignored), full workspace Clippy passed, and the complete process-level
CLI E2E suite passed (3 tests). Recovery exercises both selection and
materialization waves.
- A real CIFAR-10 pHash run used five actual registry-worker owners,
completed 32 selection + 24 materialization tasks, and retained 49,846
of 50,000 rows with identical original values and image payloads.
Host workflow performance measurements are recorded in the companion
host PR. Cross-platform CI is still running. The previous remote-test
bootstrap failed before Cargo tests with `401 Bad credentials`; the
workflow fix described below is being validated by CI. No deployment or
merge is performed.
## Python source validation
AWS tests cover automatic and explicit snapshots, physical/stable row
IDs, local
and remote creation, sync/job/async-connection APIs, delayed refresh,
strict
options and identifier quoting. The native Python extension was rebuilt
through
the project's uv/maturin backend. Ruff, Pyright and the full MkDocs
API-reference
build passed; both new public symbols are rendered.
A real 50,000-image FEv2 run disabled both Python SQL entry points and
supplied an
unreachable SQL endpoint. The new API still materialized the same 49,846
retained
rows and original image payloads. Three recorded creation/refresh jobs
completed
56 tasks each on all five registry-worker owners, and remained on source
snapshot
3 after source latest advanced to 4. No worker or server wire-protocol
changes
were needed for this client entry point.
Previous CI authentication failure:
https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265.
## Review and CI fixes
- Validate all JSON merge source fields before projection, rejecting
unknown and duplicate names at every struct level. Missing/reordered
fields and ordinary JSON merges remain supported. A checked-in 12-case
integration matrix covers root fields, structs, lists, large lists,
fixed-size lists and map values; rejected writes leave the table version
and row count unchanged.
- Match namespace connection MV method signatures to
`MaterializedViewSource` in all four sync/async methods, fixing the
Python 3.10 import failure from runtime override checks.
- Restore the remote-test script's encrypted-CLI path: use the Actions
token for artifact download and pass `SOPHON_READ_TOKEN` under the
variable the decryption script consumes. The previous workflow omitted
that variable and attempted the private release path, failing with HTTP
401. On the final head, remote CI passes: 1,464 Rust unit tests, the
integration targets and doctests pass (3 unit tests ignored).
AWS validation for these fixes: both review reproducers failed before
their fixes and pass afterward; 55 merge tests, 29 cast tests and the
standalone 7-case admission regression pass. Python 3.10.21 imports
successfully; 90 MV/namespace tests, 14 JSON tests and four real
namespace dedup cases pass. After `794b97a0`, the native binding was
rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and
Pyright pass. Sophon's locked test compilation passes against the final
SDK source for all seven affected crates. Extended runners, logs and
benchmark artifacts remain local.
Gatekeeper recommends approval on `794b97a0`, and the `breaking-change`
label is set. Final CI: **49 checks passed, 3 expected skips, no
failures or pending checks**, including remote tests, Python 3.10,
minimum dependencies and Windows ARM Rust tests. All wheel builds pass
after retrying the Linux matrix affected by transient AlmaLinux
repository-metadata 404s. All code-review threads are resolved. The PR
remains open for maintainer review and merge.
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
Co-authored-by: Lance Release <lance-dev@lancedb.com>
Brendan Clement
XY Zhan
Lance Release
Jack Ye
Colin Patrick McCabe
ccmao1130
Will JonesandClaude
lancedb-gatefixer[bot]
Bilal Atique
ningsh7
