Resolve the docs Rust example lockfile: keep lance-namespace 8.0.0 and
the remote client's windows-result 0.4.1 and windows-strings 0.5.1, which
main's dependency update dropped, and take main's other updated versions.
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.11).
Adapts the LSM resident-index guards to Lance's query-based index API
and adds regression coverage for index presence and type checks.
Validation: workspace Clippy with all features and warnings denied, Rust
formatting, and targeted LSM query unit tests.
test_function_enterprise ran against a deployment whenever LANCEDB_URI,
LANCEDB_HOST_OVERRIDE and LANCEDB_API_KEY were set. Like the quickstart
tests, it now also needs LANCEDB_DOCS_TEST_DEPLOYMENT=1, and fails
rather than skips when that is set without the settings.
New tests drive it with lancedb.connect stubbed: without the exact
opt-in it skips before connecting, with the opt-in but missing settings
it fails naming them, and with both it reaches the connection. The
examples are unchanged.
## Summary
The Java SDK pinned `jackson-databind` 2.17.1, and `jackson-core` and
the other Jackson modules came in transitively via `arrow-vector` and
`lance-namespace-apache-client` at 2.16.x–2.17.x, all affected by known
CVEs.
This imports `jackson-bom` 2.18.11 in the parent POM, and declares
`jackson-core`, `jackson-annotations`, `jackson-datatype-jsr310`, and
`jackson-jaxrs-json-provider` as direct (BOM-managed) dependencies of
`lancedb-core`. The direct declarations are needed because the parent's
dependency management doesn't apply to a consumer's transitive
resolution; without them, Maven's nearest-wins mediation still hands
downstream apps the older versions from `arrow-vector`.
## Evidence
`mvn dependency:list` in a separate consumer project depending only on
`lancedb-core`:
| Artifact | Before | After |
|---|---|---|
| jackson-core | 2.16.0 | 2.18.11 |
| jackson-annotations | 2.16.0 | 2.18.11 |
| jackson-databind | 2.17.1 | 2.18.11 |
| jackson-datatype-jsr310 | 2.16.0 | 2.18.11 |
| jackson-jaxrs-base / json-provider | 2.17.1 | 2.18.11 |
| jackson-module-jaxb-annotations | 2.17.1 | 2.18.11 |
## Merge Risk
**Door:** Two-way. Dependency versions only, and they stay within
Jackson 2.x.
**Blast Radius:** `java` dependencies. Downstream users of
`lancedb-core` now get Jackson 2.18.11 transitively.
Fixes OSS-2360, OSS-2337
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Fixes the Dependabot alerts for Rust crates that have a patched release.
- `Cargo.lock`: aws-smithy-json 0.62.5 → 0.62.7, quinn-proto 0.11.14 →
0.11.19, xxhash-rust 0.8.15 → 0.8.19.
- `docs/web-tests/rs/Cargo.lock`: rustls 0.23.35 → 0.23.45,
rustls-webpki 0.103.13 → 0.103.15, serde_with 3.16.1 → 3.22.0, rand
0.8.5 → 0.8.8 and 0.9.2 → 0.9.5, xxhash-rust 0.8.15 → 0.8.19.
- `rust/lancedb/Cargo.toml`: the AWS SDK crates' default features
include the legacy `rustls` feature, which pulls in rustls 0.21 and the
vulnerable rustls-webpki 0.101.7 via hyper 0.14.
`aws-sdk-bedrockruntime` (behind the optional `bedrock` feature) and the
`aws-sdk-s3`/`dynamodb`/`kms` dev-dependencies now use `default-features
= false` with `default-https-client` + `rt-tokio`, the same rustls 0.23
client `aws-config` already uses. Minimum versions are raised to the
first releases that have `default-https-client`. This removes rustls
0.21, rustls-webpki 0.101, hyper 0.14 and h2 0.3 from the lockfile.
Not fixed: fast-float 0.2.0 has no patched release. It comes from
polars-arrow via the optional `polars` feature, pinned to `<0.40`;
removing it needs a polars upgrade.
## Merge Risk
**Door:** two-way; lockfile and dependency-feature changes only.
**Blast Radius:** `bedrock` embedding function's HTTP client, and
S3/DynamoDB/KMS integration tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
LanceDB's remote HTTP client (reqwest) and gRPC client (tonic) used
rustls with the `ring` crypto provider, while the AWS SDK and
object_store use aws-lc-rs. This moves both clients to aws-lc-rs:
- `tonic`: `tls-ring` → `tls-aws-lc`.
- `reqwest`: 0.12 → 0.13. reqwest 0.12 has no aws-lc-rs feature; 0.13's
`rustls` feature uses it. 0.13 was already in the lockfile through
object_store. No code changes were needed.
- `macos-system-configuration` is replaced by its 0.13 name,
`system-proxy`, which also reads the system proxy on Windows.
- `query` is now an opt-in feature in 0.13; it's used for request query
parameters.
- Certificate verification moves from `rustls-native-certs` to
`rustls-platform-verifier`, which uses the OS verifier on macOS and
Windows.
`ring` is still compiled in, because `lance-namespace-impls` and
`lance-namespace-reqwest-client` enable it through reqwest 0.12
(lance-format/lance#9855, lance-format/lance-namespace#369). Both
clients here pick aws-lc-rs explicitly, so they don't depend on which
providers are compiled in.
## Evidence
A temporary example that connects the remote client to real HTTPS hosts:
```
https://example.com: Http error: ... 404 Not Found: <!doctype html>... # TLS handshake OK
https://expired.badssl.com: Hit retry limit ... connect_failures=3/3 # bad cert rejected
```
## Merge Risk
**Door:** two-way; dependency feature changes only.
**Blast Radius:** `remote` TLS and certificate verification (Python,
Node, and Rust LanceDB Cloud clients).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The TypeScript connection example only defined a helper that nothing
called, so every later step failed with `db is not defined`. It now
connects, as the Python one does. The Rust one kept its connection
inside a helper, the Enterprise one read its settings without
connecting, and the later steps relied on imports and a record-batch
helper the page never showed.
The quickstart now says how the steps make one program: an ES module
run with Node in TypeScript, and the body of an async `main` in Rust,
whose dependencies the install step lists. Each Rust step brings its own
imports, the data step shows the helper that turns the records into a
reader, and Arrow comes from lancedb's re-exports, so no Arrow crate is
needed. The row structs no longer derive serde traits nothing used.
The connection examples are shared, so the connect, create-table and
branching pages gain the import line their connection code now needs.
The Rust docs crate pinned lance-namespace 7.0.0 while lancedb 0.31.0
uses 8.0.0, so the connection example, which passes lance-namespace
request types to lancedb, did not compile. It now pins 8.0.0, the
version lancedb 0.31.0 itself depends on.
It also enables lancedb's `remote` feature, without which the crate
cannot compile an example that connects to LanceDB Enterprise.
lancedb stays at 0.31.0 and no locked version changes: the lockfile
loses the lance 7.0.0 crates the old pin pulled in and gains the
`remote` feature's dependencies.
The Python and TypeScript quickstart tests ran against a deployment
whenever LANCEDB_URI, LANCEDB_HOST_OVERRIDE and LANCEDB_API_KEY were set,
and replaced its `characters` and `character_images` tables. They now
also need LANCEDB_DOCS_TEST_DEPLOYMENT=1.
On a deployment, each test gives the tables it creates a suffix of its
own, never replaces a table that was already there, and drops only what
it created, whether or not it passes. The examples are unchanged.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
One page states the characters and lengths each kind of name can have,
following the rules lancedb #4490 centralizes, with examples. It says
what enforces them today: the embedded library checks table characters
but not length, a directory namespace accepts more, the SDKs check
db:// names only from the first release that includes #4490 (none yet),
and a deployment can accept a longer name at creation and fail on it
later. Readers are told to keep to the rules regardless. Limits, table
creation and namespaces link to it.
Two pages said list_materialized_views reads every table's schema and
that SHOW MATERIALIZED VIEWS is cheaper. That is the local
implementation; on a db:// connection the SDK asks the deployment for
its views, as SHOW MATERIALIZED VIEWS does.
The computed-column page now says that a literal must fit the column it
meets, and that a deployment's refresh can fail on an expression reading
a struct field, which the embedded library computes, so an expression
for a deployment should use top-level columns.
- CREATE INDEX: the example left out USING, which the server requires,
and the page said SQL cannot choose the algorithm or name the index.
USING names the algorithm, the page lists them, and a name before ON
is taken where the deployment's configuration allows it.
- The search join passed fts() a bare string; fts() takes JSON.
- Sampling a subquery's result is refused, so the sampling and subquery
pages sample the table inside the subquery, and the refusals list it.
- Two unaliased casts in one projection produce the same column name;
the example aliases them.
- A json_get nested in another json_get is refused. The page shows one
level of nesting and json_extract with a JSONPath for deeper values,
and corrects what json_extract and json_extract_with_type return.
- Full-text search: the multiline query chain was not parenthesized, so
pasting it raised an IndentationError.
- Pandas async: the example chained .select onto the coroutine an async
search returns; it now awaits the search first, and its test asserts
the result.
- Pydantic: the example searched with a string, which needs a full-text
index or an embedding function the table does not have; it searches
with a vector, as the page describes, and asserts the model returned.
- PyTorch: the column-selection example selected columns the page's
table lacks and printed a schema from a batch that is a dict. It now
builds its own table and prints the batch's columns.
- Data evolution: the setup used the test's tmp_db fixture, an optional
data argument and an unimported numpy. It now needs only `db`, which
the page says how to open.
A grant listing was said to hold at most 100 entries a page. The server
counts 100 stored (object, subject) tuples a page and then expands each
into its privileges, so a page can return more; the page now says so
and to keep requesting with the token.
The consistency page said a write's version is never used to choose
what a read sees. The client sends two version headers: the write
result in x-lancedb-min-version, which deployments ignore, and the
highest version any response reported in x-lancedb-min-read-version,
which they check. A successful write's response advances the second.
The page now separates them and keeps the advice to read with a zero
interval or after checkout_latest when a read must reflect a write.
- Python's table_exists was said to work on namespace connections; on
one from connect_namespace("dir" or "rest") it fails with an
AttributeError. The page now shows the supported checks: open the
table and handle the error each connection raises, or read every page
of a listing.
- Dropping a missing table was said to raise by default. A local
database raises; a deployment may report success, depending on its
catalog, and its Python connection takes no ignore_missing.
- A deployment's catalog may refuse cloning altogether; where cloning
works, it is shallow.
- A dropped column was called unrecoverable. Restoring a version from
before the drop brings it back until cleanup prunes that version.
- The namespace page stored data in a `data/` directory that does not
exist, presented a recommended character set as enforced, and its
Python listing comments showed lists where the calls return response
objects. Its namespace examples now run as tests.
- The storage tip offered the table's reported options as the effective
ones; they omit what the table inherits from its connection.
The quickstart called a refresh that raises on a deployment before it
showed the alternative, trained on columns only optional steps created,
and read its image from a path inside the documentation repository.
The core journey is now install, connect, create, add, search and
filter, and its code is tested unchanged against a local database and,
when connection variables are set, a deployment. Enterprise readers get
the settings they need before the connection code. Computed columns,
images and training are optional extensions that each start from the
core table: the computed column shows the embedded refresh beside the
deployment's job, and says that a deployment must enable computed
columns; the image step downloads the picture into ./data first and the
code reads it from there in every language; the training step derives
the features it trains on.
The Enterprise connection examples now read the database URI from
LANCEDB_URI and the endpoint from LANCEDB_HOST_OVERRIDE, with the API
key and region beside them, the convention the rest of the site uses.
Without an endpoint they stop rather than fall back to one. Their tests
check both.
The REST API reference used LANCEDB_URI for the HTTP endpoint and listed
and created tables at routes the server does not serve. It now uses
LANCEDB_HOST_OVERRIDE, lists tables at /v1/table/, creates `words` from
an Arrow IPC stream, and describes it to confirm the write.
The SDK, architecture and database pages said application code moves
to Enterprise by changing the URI, and the SQL page said every SDK
operation has a SQL form and that the SQL reference defines the SDKs.
The pages now promise what holds: start locally, then connect to
Enterprise by changing the URI, and keep the same core table and search
code. A new section on the SDK page lists what else changes: connection
settings and credentials, missing or different methods, maintenance the
deployment runs on its own configuration, and work submitted as jobs.
SQL Reference is described as the dialect's specification, with links
to each SDK's reference.
The feature engineering overview said that every mechanism separates
declaring from filling and that every refresh leaves current rows alone.
That holds for computed columns only: creating a materialized view
computes its rows, and a local refresh replaces the whole view even when
nothing changed. The overview now compares the two, keeps the
incremental explanation for computed columns, explains the view
lifecycle with its deployment qualification, and says that a deployment
does not refresh a computed column by itself. Its chooser lists Geneva
for Python work until Functions launch.
The two Function pages declare `launch: functions`, and every other page
marks what it says about Functions, so the assembled site leaves them out
until the launch. Before it, the pages that offered Functions for Python
work point Enterprise readers at Geneva instead.
The SQL Function example now uses Arrow type names, which the server
requires, and states the 3.12 default for `python_version`.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Centralize the entity name validation for tables, views, functions,
namespaces, secrets, and databases. This helps in several ways. One is
that percent-encoding is no longer needed in most cases (the exception
is database names, which can contain slashes). Another good thing is
that LLMs looking at this repository can now understand what the naming
rules are, since they will be the same here as on the server side.
Finally, this will help with the upcoming authZ code, since it also
needs to serialize these names, and needs to know that they don't
contain (among other things) colons.
Moves Function timestamps on the wire from RFC 3339 strings to epoch
milliseconds.
## What
- `FunctionVersion.created_at` (an RFC 3339 string) becomes
`created_at_millis` (milliseconds since the epoch, as an integer).
That's the shape Secrets, jobs and `FunctionErrorRecord` already use.
- Rust: `FunctionVersion::created_at() -> &str` becomes
`created_at_millis() -> i64`.
- Python: `FunctionVersion.created_at: str` becomes `created_at_millis:
int`.
- Fixtures and the mock server now send `1787270400000`, the old
`2026-08-21T00:00:00Z`.
Node.js doesn't expose `FunctionVersion`, so it doesn't change.
## Compatibility
Breaking, and it must ship together with the matching server change: a
server that still sends `created_at` fails to deserialize on this
client. Functions aren't deployed yet, so nothing is migrated.
## Tests
- `cargo test --features remote --test first_class_function_slice1`: 10
passed
- `cargo test --features remote --lib function`: 51 passed
- `pytest python/tests/test_first_class_function*.py`: 100 passed
- clippy, `cargo fmt --check`, ruff check and ruff format are clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BY5xcEGEAa1E11VG55Kd1w
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The cleanup page said timedelta(0) prunes everything except the latest
version, and the three pages said any tag on a version older than the
window stops cleanup. In the pinned Lance, cleanup also keeps the
versions branches were forked from, and only a tagged version that is
older than the window and is not the latest stops it; a tag on the
latest version is kept without an error. optimize() can still turn
such a tag into a blocking one by committing a compaction first.
The pages now say so, with moving the tag inside the window or widening
the window as remedies alongside deleting it. Each case was checked
locally against the baseline SDK.
Covers lance-format/lance#9750's `TopKLateMaterialization` rule through
lancedb's DataFusion `TableProvider`, now that main is on lance
v14.0.0-beta.7: SQL `ORDER BY ... LIMIT` over `BaseTableAdapter` with
the rule registered returns the same rows as without it, the plan
contains a `Take`, and the output schema metadata stays erased. No
certification of `MetadataEraserExec` is needed because DataFusion
pushes the top-k sort below it.
ENT-2777
The compare-and-merge section documented branches.merge, mergeable,
mergeBlockers and merged/rejected statuses, none of which the pinned SDK
has, and read TypeScript's branches() as a property. The SDK exposes
diff and cherry_pick (cherryPick in TypeScript, reached through await
table.branches()), returns camelCase results whose diff lists errors,
answers a refused cherry-pick with status "failed" rather than an
exception, and refuses both calls on local tables.
The section now documents those calls, the diff and result fields,
dry-run behaviour, statuses, error codes, async Python and the Rust
names, under its existing anchors. The Python examples were run from
the page against a mocked server and a local table, and the TypeScript
examples were type-checked against the SDK's declarations and run the
same way. No deployment was contacted.
The basic example iterated a Table, which is not iterable; it now reads
batches through an identity Permutation. The two-phase shuffle passed
block_size, window_blocks and max_shuffle_distance, which
StreamingDataset does not accept: its block-wise option is
shuffle_clump_size, which shuffles contiguous groups of rows as units.
That section, the shuffling introduction and the split explanation now
describe that, under the old anchors, and the fetching example uses
io_queue_depth in place of the deprecated prefetch_batches. Each changed
example was run locally against the baseline SDK.
"Storage | Yes / No" and an OSS tag on a page called Storage read as if
Enterprise had no storage. The difference the code shows is narrower:
the embedded library configures its storage backend from the
connection, while a deployment sets up its own and a db:// connection
passes none of those options except an Azure storage account name. The
page is now Embedded storage configuration, says so first, and points
deployments to their own setup. The declaration is unchanged.
The page declared OSS unavailable, so the comparison read "Sampling |
No / Varies" and the sidebar tagged the whole page Enterprise, while its
last section described SDK sampling that works in both offerings. It is
now titled SQL sampling, so the row and tag describe TABLESAMPLE, and
the SDK alternative is a section of its own, marked for both offerings,
under its existing anchor.
That section's example passed Permutation.identity(table) to a
DataLoader, which reads rows in stored order. It now builds a seeded
shuffle and a random split with permutation_builder, checked locally
against the baseline SDK, including through a DataLoader.
The consistency page said reads after any write carry a freshness floor,
so you read your own writes without extra configuration, and that the
floor follows the last write. The pinned client records a write's
version as x-lancedb-min-version, which Sophon does not read (neither at
75f4bf28 nor on main); its time floor comes from the interval and from
checkout_latest() and restore(); and its version watermark follows the
x-lancedb-version a response reports. Whether that watermark covers a
write depends on the version the write's response reports, which the
code alone does not settle.
The page now describes those three mechanisms and says how to be sure a
read reflects your write: an interval of zero, or checkout_latest()
after writing. The guarantees page keeps its read-your-writes promise
for the embedded library, on the handle that wrote.
The cleanup and guarantees pages said a tag or branch pointing at a pruned
version stops resolving, while the table page said tagged versions are
exempt and quietly kept. The pinned Lance (ae2903729f02, the commit in
Cargo.lock and in Sophon's Lance submodule) keeps a tagged version
whatever its age, but with its default policy refuses to clean up while a
tag points at a version older than the window, raising an error that
names the tag. optimize() passes that default, so it raises after
compacting, prunes nothing and skips its index update until the tag is
deleted. Sophon's cleanup job uses the same default.
Branches are retained separately: cleaning the parent keeps the version a
branch was forked from and the files the branch reads, while the
parent's other old versions are pruned. The pages now say both, checked
locally against the baseline SDK.
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.
JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.
Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.
Refs #3891.
Sophon main now refreshes an eligible materialized view incrementally,
appending what an append-only source added since the last refresh, a day
after the change that made every refresh a full rebuild. So the
materialized view page and the REFRESH reference no longer say recent
versions rebuild in full: they say recent versions run views as SQL jobs,
need no stable row ids and take no refresh options, and that a full or
incremental refresh depends on the version and the view's query. COPY is
described with the same "recent versions" wording.
Four summaries pointed at the table or sections below, which reads wrongly as a row of the comparison. They now stand alone, and insert links to its bulk-ingest section.
The comparison now lists pages across the documentation, grouped as in
the navigation. The page says what Yes, No and Varies mean, that pages
spanning several features are left out, and that the SQL row stands for
the SQL reference.
Storage configuration from the connection is embedded only, since a
deployment configures its own. Training data loading, PyTorch, embedding
functions, pandas, Polars and Pydantic work with local and deployment
tables, and the Hugging Face Hub integration is embedded only.
A local table cannot be pickled for DataLoader workers, so the PyTorch
page says to wrap it in a Permutation; a LazyFrame over a whole table,
to_pandas in the LangChain example and to_arrow in the DuckDB note are
local only; and query text is embedded in the client for both offerings.
SQL is unavailable in the embedded library and varies on a deployment,
which serves it only when its Flight SQL endpoint is enabled and
reachable, and the SDK has to be told where that is. The page's opening
badge goes. The SDKs page declares both offerings.
Neither SDK aggregates or joins, so the parity tables no longer say they
do, and views can be created through the SDKs on a deployment. Pages that
said vector search has no SQL form now point to the version-dependent
SQL form, and COPY, which newer deployment versions refuse, is marked
as such in the reference.
Databases, namespaces, jobs and data evolution work in both offerings.
Materialized views, computed columns and column refresh work in the
embedded library and vary on a deployment, which has to enable them.
Views and functions exist only on a deployment that enables them, and
aggregation, joins, windowing and sampling are SQL, so they need one
with SQL enabled.
Sections now say where they differ: views have SDK methods on a
deployment, dropping a function needs a further setting, a local job
handle runs in-process and has no history, create_index and
refresh_column behave differently against a deployment, and the
overviews no longer call computed columns and materialized views
available everywhere.
Creating, writing, updating, merging and deleting rows, managing tables,
branches and read consistency work in the embedded library and on a
deployment, and those pages now say so. The SQL forms they mention need
a deployment.
Where the offerings differ, the section says how: a deployment reuses an
existing table without comparing schemas, a merge insert can be refused
past a configured number of unindexed rows, space is reclaimed by
optimize() locally and by the deployment's compaction and configured
cleanup otherwise, renaming depends on the catalog, a deployment clones
shallowly only, drops all tables only locally, keeps freshness state per
table handle and defaults freshness to its own setting, and refuses to
delete a branch main depends on. table_exists works only on a namespace
connection.
Guarantees and limits span both offerings, so they carry section notes
rather than a label: reads, retention, index freshness and compaction
differ on a deployment, and the name and job limits are a deployment's.
Indexes are built in the embedded library and on a deployment, which
builds them in the background, keeps most of them current, and builds
only IVF_FLAT, IVF_PQ, IVF_RQ and IVF_HNSW_SQ vector indexes. Compaction
runs on both, by optimize() locally and in the background on a
deployment; cleanup on a deployment runs only where its configuration
enables it, so it varies; background jobs exist only on a deployment.
Sections that described one offering as if it were both now name it:
optimize() is embedded only, the brute-force limit and automatic IVF_PQ
are deployment settings, FM indexes get no background updates, a
remainder below the deployment's threshold can keep wait_for_index from
finishing, GPU builds on a deployment are IVF_PQ with a configured GPU
count, list_indices on a deployment omits num_unindexed_rows, and the
background jobs page no longer says every other job is one you start.
Brendan Clement
Lance Release
Yang Cen
LanceDB Robot
Will JonesandClaude Opus 5.5
Kevin Dong
Jack Ye
Colin Patrick McCabe
Jonathan HsiehandClaude Opus 5.5
Wyatt Alt
Xuanwo