`OllamaEmbeddings.compute_source_embeddings(np.array(["hello",
"world"]))` raises a validation error before sending a request. The
embedding interface accepts NumPy arrays, but the adapter passes the
array directly to Ollama's string-or-sequence request field.
Convert NumPy text arrays to lists before the batched request. This
preserves input order, the single request, model options and
`keep_alive`. The tests cover Unicode/object/sliced arrays, empty input,
existing list/Arrow/query inputs and API error propagation.
The added input-contract tests also cover two-dimensional arrays and
byte strings, with equivalent list inputs as controls. The real Ollama
SDK rejects 2-D inputs and invalid UTF-8 locally, before transport.
Valid UTF-8 bytes are decoded normally. No extra adapter validation is
needed for these cases.
Validation on Python 3.12.14 / macOS arm64:
- Built the Rust/PyO3 bindings using the repository's uv/maturin
environment.
- The initial regression run had 5 failures and 4 passing controls on
the base; all 9 passed with the fix. With the 6 additional
input-contract cases, the related embedding and trust suites now total
55 passed / 11 skipped.
- All Python Ruff lint and formatting checks pass. Root checks still
report the same 49 lint errors and 11 documentation formatting files
reproduced on the untouched base.
- Separate review exercised 13 public API cases. Another 7 groups passed
through a real loopback HTTP transport, including local validation,
UTF-8 serialization and empty-batch model-error propagation.
The regression tests use the real Ollama 0.6.2 SDK with HTTPX
MockTransport. The loopback check uses an HTTP surrogate, not an Ollama
daemon. No live model inference was run.
A live Ollama server was unavailable locally. [The current server
handler](https://github.com/ollama/ollama/blob/2cd63f7a15039ccd790e97b31edc091fa1707c56/server/routes.go#L1212)
explicitly returns HTTP 200 with empty embeddings for empty input after
model lookup and runner setup. The adapter preserves the existing
empty-list request and its model/setup errors. This is a source check,
not a live-server test.
Developed with OpenAI Codex, with separate agent review and executed
local checks.
PIL images, PNG bytes and local image paths in VoyageAI source
embeddings are saved directly as JPEG. RGBA, grayscale-with-alpha (LA)
and palette (P) images therefore raise `OSError: cannot write mode ...
as JPEG` before the provider is called.
Convert to RGB at all three JPEG encoding sites. The caller’s original
PIL image remains unchanged. Document that alpha is discarded and
callers should composite onto a chosen background when it matters.
Validation:
- New image regressions + existing VoyageAI tests: **18 failed / 26
passed before → 44 passed after**. Tests cover both multimodal models,
PIL/bytes/Path, RGBA/LA/P/RGB, decoded JPEG pixels and invalid images.
- Independent local LanceDB table ingestion smoke: three PNG modes
stored with 1024-dimensional vectors, original bytes preserved. Uses the
freshly built current native extension.
- Adjacent local embedding tests: 15 passed, 1 skipped (no OpenAI key),
9 deselected. The separate Flickr download test failed with
`RemoteDisconnected`; it is excluded from the local result.
- Pyright: 0 errors; CI-pinned Ruff 0.9.9 lint and Python formatting
passed.
External VoyageAI calls are stubbed; real Pillow conversion and LanceDB
ingestion execute locally. No live provider or full-repository test
result is claimed. AI assistance was used for implementation and
validation.
Direct VoyageAI `compute_source_embeddings()` calls reject `list[str]`
and NumPy text batches before reaching the provider, although both
belong to the declared `TEXT` input type. For example, `["first",
"second"]` raises `ValueError: Input type <class 'list'> not allowed
with text model.`
Normalize these containers to a list before the existing string-entry
validation. The regression cases exercise the public embedding API with
a stub provider: input order, list/NumPy batches, Arrow controls, empty
batches, and rejection of non-text entries and ordinary two-dimensional
arrays. The VoyageAI integration page now documents the accepted text
containers.
Verified on current main `7b3ba4c` with a locally compiled native
extension (0.41.0b7, Python 3.12.12):
- VoyageAI suite: 7 failed / 20 passed before the fix; 27 passed
afterward.
- Adjacent embedding suite: 13 passed, 4 skipped, 8 slow cases
deselected.
- Python CI's pinned Ruff 0.9.9 lint/format checks pass; configured
Pyright reports zero errors/warnings; all applicable pre-commit hooks
pass.
The provider is stubbed; no live Voyage API request or model-quality
evaluation was performed.
AI disclosure: I used Codex for investigation, implementation, tests,
review, and drafting this description.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
## Summary
- Rework `docs/web/training/torch.mdx` for people who already know the
PyTorch `DataLoader`, built around `lancedb.streaming.StreamingDataset`.
- The quickstart trains a state-to-action policy on the real
`lance-format/lerobot-xvla-soft-fold` dataset loaded via `hf://`, using
a zero-copy Arrow-to-tensor transform.
- Add a short table mapping PyTorch idioms (`IterableDataset`,
`DistributedSampler`, `collate_fn`, `random_split`,
`StatefulDataLoader`) to their LanceDB equivalents.
- Add an Optimizations section: Transform (including an `ExtractFrames`
example using blob v2 and torchcodec), Distributed Training, and the
existing Output Formats content moved underneath.
## Notes
- The quickstart depends on #4602 (permutation building over tables with
schema metadata), which is merged.
- `ExtractFrames` takes a URI and opens the table lazily because a local
`LanceTable` cannot yet be pickled into DataLoader workers.
- The Permutation-based "Selecting columns" and "Using multiple
DataLoader workers" sections further down are unchanged from main.
## Test plan
- [x] Ran the quickstart, transform, `ExtractFrames` (main process and
forkserver workers) and distributed snippets locally.
- [x] Rendered the page with `mint dev` and checked for console errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
Document explicit Managed Blob v2 opt-in, descriptor queries and payload
fetching, Function inputs and outputs, and whole-object cleanup. Explain
null versus empty values, snapshot ownership, packed-object retention,
and the distinction from the legacy large-binary Blob API.
The local ingestion and Enterprise Function examples have been executed
end to end. The Function lifecycle description depends on
lance-format/lance#10176 and the corresponding Enterprise migration. The
existing Functions page remains outside the published navigation until
launch.
Validation: the 347-file assembled site has no broken links. Mintlify
4.2.888 build validation reports four existing
`RsExampleBody`/`RsExampleTail` import warnings in the reranking and
vector-search pages; the same diagnostics reproduce with the unchanged
base versions of all three pages edited here.
The existing docs have 254 findings from our shared Vale rules.
Contributors would otherwise inherit these when editing the same lines.
Clean up the prose across 80 pages: correct product names and a repeated
word, simplify a few phrases, and replace dashes with punctuation that
fits each sentence. The pinned Vale scan now reports zero findings, with
no suppressions.
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The curation, SQL and operations pages contain outdated claims and
examples that do not reliably work as written.
This corrects those pages against the reviewed SDK and local server
evidence, including prepared statements, background jobs, cleanup and
monitoring. Integration guides are grouped with their examples in #4533.
Function and Secret-object documentation stays unpublished until its
release. The two Function pages are listed in `.mintignore` and left out
of the navigation, and the sections about them on other pages are MDX
comments.
Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after
#4533 is squash-merged.
Local assembly, Mint validation and link checks pass, and the assembled
site shows no Function or Secret-object content. A local release
rehearsal that uncomments the drafts and restores the navigation also
passes. Hold publication until this PR and lancedb/sophon#8688 are on
`main`.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
Ingestion, search and integration examples are incomplete or
inconsistent across languages, and some guidance misstates SDK behavior.
This brings the guides and their example sources together. It includes
complete Python, TypeScript and Rust workflows, isolated provider and
reranker examples, the Superlinked corrections, and guidance on retries
that can duplicate writes.
Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after
#4491 is squash-merged.
Local assembly, Mint validation and link checks pass. Example code is
unchanged from the previously validated package. Publication stays held
through integration.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The introductory pages are inconsistent, and the old PR split asked
reviewers to review some pages twice.
This brings the final quickstart, connection guide, architecture and
Concepts pages together. It clarifies OSS and Enterprise, simplifies
naming rules and includes the navigation and diagram fixes from #4489.
Function and Secret details on the Concepts pages are MDX comments until
their release, and CONTRIBUTING.md explains how to write docs before a
release.
Includes #4532. Ingestion, search and integrations are in #4533; data
workflows, SQL and operations are in #4534. Each MDX file has one PR
owner.
Local assembly, Mint validation and link checks pass. The Function pages
stay published from `main` until #4534 hides them, so hold publication
until the complete stack is integrated.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
Captain
Xuanwo
Jack Ye
Colin Patrick McCabe
Lei XuandClaude Sonnet 5.5
Brendan Clement
Will JonesandClaude Opus 5.5
Drew