SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
Remote materialized-view creation currently expects a REST response
containing a registry job ID, while refresh already submits SQL. Route
both through the SQL query backend and preserve Python's synchronous and
asynchronous return types.
### Query to Job mapping
`SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through
the same backend as `execute_query_async()`. Once submission succeeds,
it retains the returned `Query` and starts one client-side task that
consumes its result reader to completion. `Job::from_query_task()` wraps
that task, its cached outcome, a single ID and a cancellation callback.
Python exposes this as `Job[T]` or `AsyncJob[T]`.
The existing `job_type` property maps SQL-backed handles to the new
`"query"` kind. The shared `id` contains that query's UUID string, so
Python callers can inspect it with `db.describe_query(job.id)` on the
submitting connection. No separate `query_id`, backend enum or `type()`
API is added.
| Handle | `job_type` | `id` | Available lookup |
| --- | --- | --- | --- |
| SQL-backed | `"query"`, known at submission | Connection-scoped query
UUID string | `describe_query(job.id)` |
| Registry-backed | Existing server-reported kind, after `status()` or
`refresh()` | Registry job ID string | Registry job APIs, descriptions
and events |
| Local | `None` | `None` | The current in-process handle |
| Job concept | SQL-backed behavior |
| --- | --- |
| Submission | Await initial SQL submission before returning the handle.
Submission errors propagate immediately. |
| `status()` | Tracks the adapter task: running during result
consumption/finalization, finished after both succeed, failed if either
fails, or cancelled after cancellation. |
| `wait()` | Returns the operation's typed terminal result. One task
consumes the query reader once; repeated waits reuse its cached outcome
without resubmitting SQL or reopening the reader. |
| `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported
cancellation leaves the task running; an accepted request stops the
task. Cancelling a finished operation is a no-op. |
| Registry details | Query-backed handles expose no registry
description, event history, pause/resume or reattachment after a client
restart. `refresh()` updates cached lifecycle state. |
The Job tracks command completion rather than exposing SQL batches. A
wait timeout does not cancel the query. Query IDs remain scoped to the
submitting connection and cannot be used with registry APIs or as
durable resume tokens. Python `describe_query()` accepts both UUID
objects and the string `id` exposed by query-backed jobs. Rust, Python
and TypeScript expose the same classification through their existing
job-type accessors.
### Materialized-view operations
- **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace
and optional auto-refresh settings. After command completion, resolve
the job to `None`. Synchronous creation waits on the job, then opens and
returns the materialized view.
- **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same
adapter. After command completion, read the refreshed table's version
and row count to construct the existing typed refresh result.
Synchronous refresh waits on the same path.
Remote creation requires the SQL service. Execution placement remains a
server decision. REST endpoints are unchanged. The job guide documents
the ID/job-type contract and query-backed limitations.
Regression coverage exercises SQL submission, namespaces, refresh
options, job-type/ID preservation through typed result mapping, repeated
waits, failures and cancellation rejection. No local builds or tests
were run.
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
OpenAI shuts down `gpt-4-turbo` and its snapshots on 2026-10-23
(https://developers.openai.com/api/docs/deprecations), so
`OpenaiReranker` fails for every user who relies on the default
`gpt-4-turbo-preview` after that date.
This changes the default to `gpt-5.6-sol`, the replacement OpenAI names
for `gpt-4-turbo` on that page, in the class, its docstring and the
integration docs page. `_rerank` also stops sending `temperature=0`: the
GPT-5 models only accept the default temperature, so keeping it would
fail the new default on its first call. The reranker still asks for
`json_object` output and the rest of the request is unchanged.
The new test calls `_rerank` with a mocked client and checks that no
`temperature` is sent. It fails on main and passes with the change. No
request was made to OpenAI.
This change was made with Claude Code.
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4248.
The preview index still contains pylance 1.0.0b6. uv's default
first-index resolution can select it, or backtrack to an old LanceDB
release when installing the extra. Meanwhile, the extra still permits
pylance 5 even though current main bundles Lance 14.
Raise the optional dependency to `pylance>=14.0.0b12`, the version
already pinned in the test extra, and keep the lockfile metadata
consistent. Document both Fury indexes and uv's `unsafe-best-match`
strategy in the Python README and the source-owned quickstart
(`docs/web/quickstart.mdx`). Explain the index trust requirement and
recommend installing the extra to enforce the version floor.
Add CI checks that reject older Lance majors and ensure the pylance and
tests extras remain compatible. Run these checks when either dependency
manifest changes, so a future Lance major bump cannot silently leave the
Python minimum behind.
### Validation
- Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with
uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects
`pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and
the unexpected `blob_handling` scanner argument using a real blob table.
- Resolved the modified `python/pyproject.toml` with the pylance extra
against both indexes: selects `pylance==14.0.0b18`, matching the current
Rust pin.
- Ran both documented pip and uv installations in isolated environments:
`lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a
blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both
package dependency checks pass.
- New regression fails on the original dependency bound (27
rejected-version cases incorrectly accepted), then passes with the fix.
Full CI-script suite: 3 tests pass.
- `uv lock --check` with both indexes passes; focused Ruff checks pass;
`git diff --check` passes.
- Repository-wide Ruff reports 49 existing errors in untouched
`docs/web-tests` files. Unrelated formatting changes were discarded.
Runtime smoke checks used released native wheels; source dependency
resolution was tested directly from the modified pyproject. No Rust or
Python runtime code changes.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Adds the Enterprise reference page at
`docs/web/enterprise/distributed-search.mdx`, following the same
arrangement as Enterprise architecture. The page describes distributed
search and the embedded execution model, with links to shared search
guides. Enterprise architecture links to it, and the official sidebar
lists it immediately after Architecture in the existing Deployment
group.
Adds stable anchors for the RaBitQ and `approx_mode` sections linked by
the Enterprise guide.
Companion: https://github.com/lancedb/sophon/pull/8986 supplies the
detailed Enterprise overlay at the same path. Merge this reference-page
PR first so the overlay has an existing route to replace.
Validation:
- Combined-site assembly with both PR branches passed (309 files).
- Mintlify build validation and link, anchor, and redirect checks
passed.
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.
In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
`blobMode: "bytes"` left blobs inside structs as descriptors because
projection matching ignored parent structs and byte replacement only
rebuilt top-level fields.
Expand direct struct projections and aliases to their blob leaves, then
recursively replace the corresponding Arrow fields and data with
`LargeBinary`. Track literal output names separately from descendant
paths so scalar aliases stay intact and every overlapping blob
projection is converted. Preserve null structs, null and empty blobs,
sibling fields, schema metadata, and the existing row ID and snapshot
behavior. Blobs inside lists retain their descriptors. Update the API
docs to describe struct support.
Regression coverage includes full and parent-struct projections, leaf
and struct aliases, dotted output names, overlapping projections in both
orders, nested structs, nulls, empty blobs and results, multiple
batches, list siblings, query reuse, row IDs, and concurrent writes.
Regression tests reproduce the nested-blob and projection-collision
failures before the corresponding changes.
Validation:
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test __test__/nested_blob.test.ts __test__/blob.test.ts
__test__/query.test.ts --runInBand --verbose=false` — 142 passed
- `pnpm test __test__/table.test.ts --runInBand --verbose=false -t 'when
dealing with blob columns'` — 60 passed
Fixes#4609
<!-- lance-gatekeeper-fix:v1 agent=76edaac77d044c468d569b58efa2104a
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Closes#4609.
`blobMode: "bytes"` only resolved top-level blob columns, so a blob
nested in a struct came back as its descriptor. Now struct projections
that contain blob columns are resolved too: the bytes are fetched per
blob column and the struct is rebuilt with `LargeBinary` leaves,
preserving struct validity so a null struct stays null. Sibling fields
pass through untouched, aliases work for both `info.image AS image` and
`info AS details`, and blobs nested in lists still keep their
descriptors.
Tested with `npx tsc`, `biome check`, and the full `table.test.ts` suite
(580 tests, including new regressions for select-all, struct selection,
aliases, null structs, null and empty blobs, sibling fields, empty
results, and single-row batches). Not run against remote tables.
`OllamaEmbeddings.compute_source_embeddings(np.array(["hello",
"world"]))` raises a validation error before sending a request. The
embedding interface accepts NumPy arrays, but the adapter passes the
array directly to Ollama's string-or-sequence request field.
Convert NumPy text arrays to lists before the batched request. This
preserves input order, the single request, model options and
`keep_alive`. The tests cover Unicode/object/sliced arrays, empty input,
existing list/Arrow/query inputs and API error propagation.
The added input-contract tests also cover two-dimensional arrays and
byte strings, with equivalent list inputs as controls. The real Ollama
SDK rejects 2-D inputs and invalid UTF-8 locally, before transport.
Valid UTF-8 bytes are decoded normally. No extra adapter validation is
needed for these cases.
Validation on Python 3.12.14 / macOS arm64:
- Built the Rust/PyO3 bindings using the repository's uv/maturin
environment.
- The initial regression run had 5 failures and 4 passing controls on
the base; all 9 passed with the fix. With the 6 additional
input-contract cases, the related embedding and trust suites now total
55 passed / 11 skipped.
- All Python Ruff lint and formatting checks pass. Root checks still
report the same 49 lint errors and 11 documentation formatting files
reproduced on the untouched base.
- Separate review exercised 13 public API cases. Another 7 groups passed
through a real loopback HTTP transport, including local validation,
UTF-8 serialization and empty-batch model-error propagation.
The regression tests use the real Ollama 0.6.2 SDK with HTTPX
MockTransport. The loopback check uses an HTTP surrogate, not an Ollama
daemon. No live model inference was run.
A live Ollama server was unavailable locally. [The current server
handler](https://github.com/ollama/ollama/blob/2cd63f7a15039ccd790e97b31edc091fa1707c56/server/routes.go#L1212)
explicitly returns HTTP 200 with empty embeddings for empty input after
model lookup and runner setup. The adapter preserves the existing
empty-list request and its model/setup errors. This is a source check,
not a live-server test.
Developed with OpenAI Codex, with separate agent review and executed
local checks.
PIL images, PNG bytes and local image paths in VoyageAI source
embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA)
and palette (P) images therefore raise `OSError: cannot write mode ...
as JPEG` before the provider is called.
Convert to RGB at all three JPEG encoding sites. The caller’s original
PIL image remains unchanged. Document that alpha is discarded and
callers should composite onto a chosen background when it matters.
Validation:
- New image regressions + existing VoyageAI tests: **18 failed / 26
passed before → 44 passed after**. Tests cover both multimodal models,
PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images.
- Independent local LanceDB table ingestion smoke: three PNG modes
stored with 1024-dimensional vectors, original bytes preserved. Uses the
freshly built current native extension.
- Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key),
9 deselected. The separate Flickr download test failed with
`RemoteDisconnected`; it is excluded from the local result.
- Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting
passed.
External VoyageAI calls are stubbed; real Pillow conversion and LanceDB
ingestion execute locally. No live provider or full-repository test
result is claimed. AI assistance was used for implementation and
validation.
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]`
and NumPy text batches before reaching the provider, although both
belong to the declared `TEXT` input type. For example, `["first",
"second"]` raises `ValueError: Input type <class 'list'> not allowed
with text model.`
Normalize these containers to a list before the existing string-entry
validation. The regression cases exercise the public embedding API with
a stub provider: input order, list/NumPy batches, Arrow controls, empty
batches, and rejection of non-text entries and ordinary two-dimensional
arrays. The VoyageAI integration page now documents the accepted text
containers.
Verified on current main `7b3ba4c` with a locally compiled native
extension (0.41.0b7, Python 3.12.12):
- VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed
afterward.
- Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases
deselected.
- Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured
Pyright reports zero errors/warnings; all applicable pre-commit hooks
pass.
The provider is stubbed; no live Voyage API request or model-quality
evaluation was performed.
AI disclosure: I used Codex for investigation, implementation, tests,
review, and drafting this description.
Addresses option 2 of #4440.
tokenize() builds the FTS analyzer on the client. For jieba/* and
lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and
when they’re missing you only got lance’s raw Invalid directory path:
... error, even though server-side search works.
Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps
the original cause and adds that tokenize() runs on the client, which
directory the files need to be in, and that for remote tables they must
match the server’s. I also updated the tokenize docs (Rust, Python, TS,
generated JS docs) to say a mismatch can’t be detected, so tokens may
differ from the server’s index.
Behavior when the files exist is unchanged. Detecting a mismatch needs a
server-side tokenize endpoint (option 1), so I left it out. The hint
isn’t limited to remote tables, since Table::as_native is slated for
removal and the wording fits local tables too.
Tests: Rust test with mocked jieba/default index_details and an empty
model dir, one checking other tokenizer errors don’t get the hint, and
Python sync/async tests.
TypeScript CreateTableOptions.mode omitted exist_ok even though
createTable already forwards that mode to the native client.
closes: #2900
---------
Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com>
Preserve the compaction option tests and upstream CUDA tests by retaining both Any and MagicMock imports.
Validation: rebuilt the Python extension against Lance v15; 29 optimize/CUDA tests and 7 storage documentation tests passed. Rust formatting and Python source checks passed. Full repository Ruff still reports 49 inherited documentation example errors.
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
## Summary
- Rework `docs/web/training/torch.mdx` for people who already know the
PyTorch `DataLoader`, built around `lancedb.streaming.StreamingDataset`.
- The quickstart trains a state-to-action policy on the real
`lance-format/lerobot-xvla-soft-fold` dataset loaded via `hf://`, using
a zero-copy Arrow-to-tensor transform.
- Add a short table mapping PyTorch idioms (`IterableDataset`,
`DistributedSampler`, `collate_fn`, `random_split`,
`StatefulDataLoader`) to their LanceDB equivalents.
- Add an Optimizations section: Transform (including an `ExtractFrames`
example using blob v2 and torchcodec), Distributed Training, and the
existing Output Formats content moved underneath.
## Notes
- The quickstart depends on #4602 (permutation building over tables with
schema metadata), which is merged.
- `ExtractFrames` takes a URI and opens the table lazily because a local
`LanceTable` cannot yet be pickled into DataLoader workers.
- The Permutation-based "Selecting columns" and "Using multiple
DataLoader workers" sections further down are unchanged from main.
## Test plan
- [x] Ran the quickstart, transform, `ExtractFrames` (main process and
forkserver workers) and distributed snippets locally.
- [x] Rendered the page with `mint dev` and checked for console errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
Derive a row target from estimated row size, link the options reference, and retain first-pass and source-budget caveats. Keep the section hidden until release availability is confirmed.
Validate the example across initial compaction and a later append; all seven storage documentation tests pass. Guide, draft, and snippet MDX compilation and targeted Ruff checks pass.
Document explicit Managed Blob v2 opt-in, descriptor queries and payload
fetching, Function inputs and outputs, and whole-object cleanup. Explain
null versus empty values, snapshot ownership, packed-object retention,
and the distinction from the legacy large-binary Blob API.
The local ingestion and Enterprise Function examples have been executed
end to end. The Function lifecycle description depends on
lance-format/lance#10176 and the corresponding Enterprise migration. The
existing Functions page remains outside the published navigation until
launch.
Validation: the 347-file assembled site has no broken links. Mintlify
4.2.888 build validation reports four existing
`RsExampleBody`/`RsExampleTail` import warnings in the reranking and
vector-search pages; the same diagnostics reproduce with the unchanged
base versions of all three pages edited here.
The existing docs have 254 findings from our shared Vale rules.
Contributors would otherwise inherit these when editing the same lines.
Clean up the prose across 80 pages: correct product names and a repeated
word, simplify a few phrases, and replace dashes with punctuation that
fits each sentence. The pinned Vale scan now reports zero findings, with
no suppressions.
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The curation, SQL and operations pages contain outdated claims and
examples that do not reliably work as written.
This corrects those pages against the reviewed SDK and local server
evidence, including prepared statements, background jobs, cleanup and
monitoring. Integration guides are grouped with their examples in #4533.
Function and Secret-object documentation stays unpublished until its
release. The two Function pages are listed in `.mintignore` and left out
of the navigation, and the sections about them on other pages are MDX
comments.
Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after
#4533 is squash-merged.
Local assembly, Mint validation and link checks pass, and the assembled
site shows no Function or Secret-object content. A local release
rehearsal that uncomments the drafts and restores the navigation also
passes. Hold publication until this PR and lancedb/sophon#8688 are on
`main`.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
Ingestion, search and integration examples are incomplete or
inconsistent across languages, and some guidance misstates SDK behavior.
This brings the guides and their example sources together. It includes
complete Python, TypeScript and Rust workflows, isolated provider and
reranker examples, the Superlinked corrections, and guidance on retries
that can duplicate writes.
Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after
#4491 is squash-merged.
Local assembly, Mint validation and link checks pass. Example code is
unchanged from the previously validated package. Publication stays held
through integration.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The introductory pages are inconsistent, and the old PR split asked
reviewers to review some pages twice.
This brings the final quickstart, connection guide, architecture and
Concepts pages together. It clarifies OSS and Enterprise, simplifies
naming rules and includes the navigation and diagram fixes from #4489.
Function and Secret details on the Concepts pages are MDX comments until
their release, and CONTRIBUTING.md explains how to write docs before a
release.
Includes #4532. Ingestion, search and integrations are in #4533; data
workflows, SQL and operations are in #4534. Each MDX file has one PR
owner.
Local assembly, Mint validation and link checks pass. The Function pages
stay published from `main` until #4534 hides them, so hold publication
until the complete stack is integrated.
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
Gatefixer
Jack Ye
alexcastgonzandXuanwo
KarthikSandXuanwo
LuQQiu
Colin Patrick McCabe
Ishita Singh
Captain
Aviral Saxena
Preston ZhaoandPreston Zhao
Lance Release
Lei XuandClaude Sonnet 5.5
Brendan Clement
Will JonesandClaude Opus 5.5
Bilal Atique
Drew