rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-30 23:36:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)

Browse Source 
## Summary

Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.

- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.

Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.

## Evidence

- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
  - Vale.
  - The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.

## Merge Risk

**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.

Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-10-28 14:04:41 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent b4cbaf2f67
commit 4d7a64712f
148 changed files with 981 additions and 755 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/python/lancedb/__init__.py
+27 -23
View File
@@ -137,18 +137,18 @@ def connect(
The uri of the database. When ``namespace_client_impl`` is provided you may
omit ``uri`` and connect through a namespace client instead.
api_key: str, optional
If presented, connect to LanceDB cloud.
If presented, connect to LanceDB Enterprise or LanceDB Cloud (legacy).
Otherwise, connect to a database on file system or cloud storage.
Can be set via environment variable `LANCEDB_API_KEY`.
OAuth configuration is currently supported only by ``connect_async``;
synchronous LanceDB Cloud connections require an API key.
synchronous ``db://`` connections require an API key.
region: str, default "us-east-1"
The region to use for LanceDB Cloud.
The region to use for LanceDB Cloud (legacy).
host_override: str, optional
The override url for LanceDB Cloud.
The URL of the LanceDB Enterprise deployment to connect to.
sql_host_override: str, optional
The remote SQL service endpoint override. The client connects lazily when SQL
is first executed and retains that connection.
The LanceDB Enterprise SQL service endpoint override. The client
connects lazily when SQL is first executed and retains that connection.
read_consistency_interval: timedelta, default None
The interval at which to check for updates to the table from other
processes. If None, then consistency is not checked. For performance
@@ -164,18 +164,19 @@ def connect(
often each read pays the cost of checking for updates against object
storage, raising per-read latency and cost.
client_config: ClientConfig or dict, optional
Configuration options for the LanceDB Cloud HTTP client. If a dict, then
Configuration options for the HTTP client used by ``db://`` connections.
If a dict, then
the keys are the attributes of the ClientConfig class. If None, then the
default configuration is used.
storage_options: dict, optional
Additional options for the storage backend. See available options at
<https://docs.lancedb.com/storage/>
manifest_enabled : bool, default False
When true for local/native connections, use directory namespace
When true for LanceDB Embedded connections, use directory namespace
manifests as the source of truth for table metadata. Existing
directory-listed root tables are migrated into the manifest on access.
session: Session, optional
(For LanceDB OSS only)
(For LanceDB Embedded only)
A session to use for this connection. Sessions allow you to configure
cache sizes for index and metadata caches, which can significantly
impact memory use and performance. They can also be re-used across
@@ -231,7 +232,7 @@ def connect(
In-memory databases are not persisted. Tables are dropped when the last
connection or table handle referencing them is closed.
Connect to LanceDB cloud:
Connect to LanceDB Enterprise or LanceDB Cloud (legacy):
>>> db = lancedb.connect("db://my_database", api_key="ldb_...",
... client_config={"retry_config": {"retries": 5}})
@@ -289,15 +290,17 @@ def connect(
if unknown_kwargs:
raise ValueError(f"Unknown keyword arguments: {unknown_kwargs}")
if session is not None:
raise NotImplementedError("session is not supported for remote connections")
raise NotImplementedError(
"session is not supported for LanceDB Distributed connections"
)
if manifest_enabled:
raise NotImplementedError(
"manifest_enabled is not supported for remote connections"
"manifest_enabled is not supported for LanceDB Distributed connections"
)
if api_key is None:
api_key = os.environ.get("LANCEDB_API_KEY")
if api_key is None:
raise ValueError(f"api_key is required to connect to LanceDB cloud: {uri}")
raise ValueError(f"api_key is required to connect to {uri}")
if isinstance(request_thread_pool, int):
request_thread_pool = ThreadPoolExecutor(request_thread_pool)
return RemoteDBConnection(
@@ -480,16 +483,16 @@ async def connect_async(
uri: str or Path
The uri of the database.
api_key: str, optional
If present, connect to LanceDB cloud.
If present, connect to LanceDB Enterprise or LanceDB Cloud (legacy).
Otherwise, connect to a database on file system or cloud storage.
Can be set via environment variable `LANCEDB_API_KEY`.
region: str, default "us-east-1"
The region to use for LanceDB Cloud.
The region to use for LanceDB Cloud (legacy).
host_override: str, optional
The override url for LanceDB Cloud.
The URL of the LanceDB Enterprise deployment to connect to.
sql_host_override: str, optional
The remote SQL service endpoint override. The client connects lazily when SQL
is first executed and retains that connection.
The LanceDB Enterprise SQL service endpoint override. The client
connects lazily when SQL is first executed and retains that connection.
read_consistency_interval: timedelta, default None
The interval at which to check for updates to the table from other
processes. If None, then consistency is not checked. For performance
@@ -505,27 +508,28 @@ async def connect_async(
often each read pays the cost of checking for updates against object
storage, raising per-read latency and cost.
client_config: ClientConfig or dict, optional
Configuration options for the LanceDB Cloud HTTP client. If a dict, then
Configuration options for the HTTP client used by ``db://`` connections.
If a dict, then
the keys are the attributes of the ClientConfig class. If None, then the
default configuration is used.
storage_options: dict, optional
Additional options for the storage backend. See available options at
<https://docs.lancedb.com/storage/>
session: Session, optional
(For LanceDB OSS only)
(For LanceDB Embedded only)
A session to use for this connection. Sessions allow you to configure
cache sizes for index and metadata caches, which can significantly
impact memory use and performance. They can also be re-used across
multiple connections to share the same cache state.
manifest_enabled : bool, default False
When true for local/native connections, use directory namespace
When true for LanceDB Embedded connections, use directory namespace
manifests as the source of truth for table metadata. Existing
directory-listed root tables are migrated into the manifest on access.
namespace_client_properties : dict, optional
Additional directory namespace client properties to use with
``manifest_enabled=True``.
oauth_config : OAuthConfig, optional
OAuth configuration for LanceDB Cloud/Enterprise. This is supported by
OAuth configuration for LanceDB Enterprise. This is supported by
``connect_async`` only; synchronous ``connect`` uses API key
authentication for ``db://`` URIs.
@@ -551,7 +555,7 @@ async def connect_async(
... )
... # For tests and temporary data, use an in-memory database
... db = await lancedb.connect_async("memory://")
... # Connect to LanceDB cloud
... # Connect to LanceDB Enterprise or LanceDB Cloud (legacy)
... db = await lancedb.connect_async("db://my_database", api_key="ldb_...",
... client_config={
... "retry_config": {"retries": 5}})
python/python/lancedb/db.py
+43 -38
View File
@@ -626,9 +626,9 @@ class DBConnection(EnforceOverrides):
) -> MaterializedView:
"""Define a materialized view named ``name`` over the table ``source``.
Local creation and refresh require a source build with the
LanceDB Embedded creation and refresh require a source build with the
``experimental-functions`` Cargo feature. Published packages leave it
disabled; remote Cloud/Enterprise operations remain available.
disabled; LanceDB Distributed operations remain available.
The view is populated before creation returns. Pass
``with_no_data=True`` to create only its definition and empty backing
@@ -652,11 +652,11 @@ class DBConnection(EnforceOverrides):
with_no_data: bool, default False
Skip the initial refresh and leave the backing table empty.
auto_refresh_enabled: bool, optional
Enable or disable automatic refresh on Cloud and Enterprise.
Enable or disable automatic refresh in LanceDB Distributed.
When omitted, the server determines whether automatic refresh is enabled.
Embedded databases do not support automatic refresh.
LanceDB Embedded does not support automatic refresh.
auto_refresh_interval_seconds: float, optional
Positive, finite refresh interval in seconds on Cloud and Enterprise.
Positive, finite refresh interval in seconds in LanceDB Distributed.
When omitted, the server determines when to refresh.
Returns
@@ -681,7 +681,8 @@ class DBConnection(EnforceOverrides):
) -> Job[None]:
"""Submit materialized-view creation and return its job.
The job may already be complete for a local database. Remote creation
The job may already be complete in LanceDB Embedded. In LanceDB
Distributed, creation
submits CREATE MATERIALIZED VIEW through the SQL server. Its job has
``job_type == "query"`` and ``id`` identifies the connection-scoped
query. Registry event history is unavailable.
@@ -721,7 +722,8 @@ class DBConnection(EnforceOverrides):
) -> None:
"""Drop a materialized view.
Remote drops wait for the SQL command to remove the catalog binding.
In LanceDB Distributed, drops wait for the SQL command to remove the
catalog binding.
Physical cleanup continues in the background.
"""
raise NotImplementedError(
@@ -733,7 +735,8 @@ class DBConnection(EnforceOverrides):
) -> Job[None]:
"""Start dropping a materialized view and return its job.
The job may already be complete for a local database. Remote jobs have
The job may already be complete in LanceDB Embedded. In LanceDB
Distributed, jobs have
type ``query`` and use the SQL query ID as ``id``. Waiting confirms
catalog removal; physical cleanup continues in the background.
"""
@@ -809,7 +812,7 @@ class DBConnection(EnforceOverrides):
"""
Drop all tables from the database
Remote connections do not currently support this operation. Use
LanceDB Distributed does not currently support this operation. Use
``list_tables()`` and ``drop_table()`` to drop tables individually.
Parameters
@@ -854,7 +857,7 @@ class DBConnection(EnforceOverrides):
For namespace connections, this returns the backing namespace client.
For enterprise (remote) connections, this returns a RestNamespace
For LanceDB Enterprise connections, this returns a RestNamespace
with the same URI and authentication headers.
Returns
@@ -890,7 +893,7 @@ class DBConnection(EnforceOverrides):
The server builds the OCI image and registers the completed artifact.
This is the blocking counterpart of :meth:`create_function_async`.
Local connections raise ``NotImplementedError``.
LanceDB Embedded connections raise ``NotImplementedError``.
Parameters
----------
@@ -934,7 +937,7 @@ class DBConnection(EnforceOverrides):
The server-side job builds the OCI image, then registers the completed
artifact. Waiting on the job returns the immutable Function version.
Local connections raise ``NotImplementedError``.
LanceDB Embedded connections raise ``NotImplementedError``.
"""
raise NotImplementedError(
"Function catalog operations are not supported for this connection type"
@@ -962,7 +965,7 @@ class DBConnection(EnforceOverrides):
"""List every published immutable Function version in a namespace.
Functions in child namespaces are not included. Results are ordered by
Function name then version. Local connections raise
Function name then version. LanceDB Embedded connections raise
``NotImplementedError``.
Examples
@@ -1012,7 +1015,7 @@ class DBConnection(EnforceOverrides):
The name is unbound before this returns; the object's content may still
be being deleted. Call :meth:`Job.wait` to wait for that to finish. When
the server deletes inline, or when nothing was bound, the returned job
is already finished and has no id. Local connections raise
is already finished and has no id. LanceDB Embedded connections raise
NotImplementedError.
"""
raise NotImplementedError(
@@ -1041,7 +1044,7 @@ class DBConnection(EnforceOverrides):
Fails if it does not exist. Every Function bound to the Secret uses the
new value from its next job, and no new Function version is created --
which is how a rotation reaches columns pinned to a version registered
before it. Local connections raise ``NotImplementedError``.
before it. LanceDB Embedded connections raise ``NotImplementedError``.
"""
raise NotImplementedError(
"Secret operations are not supported for this connection type"
@@ -1057,7 +1060,8 @@ class DBConnection(EnforceOverrides):
"""The names of every Secret in this database.
Names only. No method returns a stored credential, by construction
rather than by policy. Local connections raise ``NotImplementedError``.
rather than by policy. LanceDB Embedded connections raise
``NotImplementedError``.
Returns a lazy iterator. ``page_limit`` limits each request, not the total;
``page_token`` resumes from a saved token. Requests and errors occur during
@@ -1075,7 +1079,7 @@ class DBConnection(EnforceOverrides):
Functions bound to it fail at their next job, naming the Secret; that
is the revocation path. The name becomes free to reuse, and a new
Secret under it is picked up by everything still bound to that name.
Local connections raise ``NotImplementedError``.
LanceDB Embedded connections raise ``NotImplementedError``.
"""
raise NotImplementedError(
"Secret operations are not supported for this connection type"
@@ -1103,7 +1107,7 @@ class DBConnection(EnforceOverrides):
its readers see its sources as they are at read time.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create. Local connections raise
view is a drop followed by a create. LanceDB Embedded connections raise
``NotImplementedError``.
>>> import lancedb
@@ -1125,7 +1129,7 @@ class DBConnection(EnforceOverrides):
schema that query resolved to.
The schema is the one recorded at creation; a source altered since then
shows up when the view is read. Local connections raise
shows up when the view is read. LanceDB Embedded connections raise
``NotImplementedError``.
"""
raise NotImplementedError(
@@ -1139,7 +1143,7 @@ class DBConnection(EnforceOverrides):
The tables it reads are untouched: a view holds no rows of its own. Use
:meth:`drop_view_async` to get the cleanup job instead of waiting on it.
Local connections raise ``NotImplementedError``.
LanceDB Embedded connections raise ``NotImplementedError``.
"""
raise NotImplementedError(
"View operations are not supported for this connection type"
@@ -1286,7 +1290,7 @@ class DBConnection(EnforceOverrides):
cancellation. Open ``reader()`` directly: a streamed query needs its
reader to make progress and reports ``running`` until the stream ends.
Local connections do not support SQL.
LanceDB Embedded connections do not support SQL.
"""
raise NotImplementedError("SQL is not supported for this connection type")
@@ -1764,9 +1768,9 @@ class LanceDBConnection(DBConnection):
) -> MaterializedView:
"""Define a materialized view named ``name`` over the table ``source``.
Local creation and refresh require a source build with the
LanceDB Embedded creation and refresh require a source build with the
``experimental-functions`` Cargo feature. Published packages leave it
disabled; remote Cloud/Enterprise operations remain available.
disabled; LanceDB Distributed operations remain available.
See
[DBConnection.create_materialized_view][lancedb.DBConnection.create_materialized_view].
@@ -2261,12 +2265,12 @@ class AsyncConnection(object):
To obtain a connection you can use the [connect_async][lancedb.connect_async]
function.
This could be a native connection (using lance) or a remote connection (e.g. for
connecting to LanceDb Cloud)
This could be a LanceDB Embedded connection (using lance) or a LanceDB Distributed
connection (e.g. for connecting to LanceDB Enterprise)
Local connections do not currently hold any open resources but they may do so in the
future (for example, for shared cache or connections to catalog services) Remote
connections represent an open connection to the remote server. The
Embedded connections do not currently hold any open resources but they may do so in
the future (for example, for shared cache or connections to catalog services)
Distributed connections represent an open connection to the server. The
[close][lancedb.db.AsyncConnection.close] method can be used to release any
underlying resources eagerly. The connection can also be used as a context manager.
@@ -2854,9 +2858,9 @@ class AsyncConnection(object):
) -> AsyncMaterializedView:
"""Define a materialized view named ``name`` over the table ``source``.
Local creation and refresh require a source build with the
LanceDB Embedded creation and refresh require a source build with the
``experimental-functions`` Cargo feature. Published packages leave it
disabled; remote Cloud/Enterprise operations remain available.
disabled; LanceDB Distributed operations remain available.
See
[DBConnection.create_materialized_view][lancedb.DBConnection.create_materialized_view].
"""
@@ -2890,8 +2894,8 @@ class AsyncConnection(object):
) -> AsyncJob[None]:
"""Submit materialized-view creation and return its job.
Wait for the returned job before opening or querying the view. Remote
creation uses SQL; ``job_type`` is ``"query"`` and ``id``
Wait for the returned job before opening or querying the view. In LanceDB
Distributed, creation uses SQL; ``job_type`` is ``"query"`` and ``id``
identifies that query on this connection.
"""
source, native_options = await self._materialized_view_source(
@@ -2945,7 +2949,8 @@ class AsyncConnection(object):
) -> None:
"""Drop a materialized view.
Remote drops wait for the SQL command to remove the catalog binding.
In LanceDB Distributed, drops wait for the SQL command to remove the
catalog binding.
Physical cleanup continues in the background.
"""
if namespace_path is None:
@@ -2960,7 +2965,7 @@ class AsyncConnection(object):
) -> AsyncJob[None]:
"""Start dropping a materialized view and return its job.
For remote connections, the job has type ``query`` and its ID identifies
In LanceDB Distributed, the job has type ``query`` and its ID identifies
the SQL query. Await :meth:`AsyncJob.wait` for catalog removal; physical
cleanup continues in the background.
"""
@@ -3135,7 +3140,7 @@ class AsyncConnection(object):
[EnvVarSecret][lancedb.secrets.EnvVarSecret], each naming a Secret and
the environment variable its value arrives in. ``namespace_path`` is
the namespace the Function is registered in; None or an empty list is
the root. Local connections raise ``NotImplementedError``.
the root. LanceDB Embedded connections raise ``NotImplementedError``.
"""
if not isinstance(definition, UdfDefinition):
raise TypeError("create_function_async requires a @udf definition")
@@ -3167,7 +3172,7 @@ class AsyncConnection(object):
"""List every published immutable Function version in a namespace.
Functions in child namespaces are not included. Results are ordered by
Function name then version. Local connections raise
Function name then version. LanceDB Embedded connections raise
``NotImplementedError``.
Returns a lazy iterator. ``page_limit`` limits each request, not the total;
@@ -3422,7 +3427,7 @@ class AsyncConnection(object):
references. The namespace defaults to ``["public"]``. ``parameters``
binds values to ``$1`` / ``$name`` placeholders; see
[DBConnection.execute_query_async][lancedb.db.DBConnection.execute_query_async].
Local connections raise ``NotImplementedError``.
LanceDB Embedded connections raise ``NotImplementedError``.
"""
return AsyncSqlQuery(
await self._inner.execute_query_async(
@@ -3450,7 +3455,7 @@ class AsyncConnection(object):
For namespace connections, this returns the backing namespace client.
For enterprise (remote) connections, this returns a RestNamespace
For LanceDB Enterprise connections, this returns a RestNamespace
with the same URI and authentication headers.
Returns
python/python/lancedb/functions.py
+4 -4
View File
@@ -4,7 +4,7 @@
"""Canonical Function values exchanged with LanceDB Enterprise services.
These immutable models contain client/wire state only. Catalog persistence,
environment bake, secret resolution, and execution are owned by Sophon.
environment bake, secret resolution, and execution are owned by LanceDB Enterprise.
``RefreshColumnResult`` is also the backend-neutral result of a local
expression-backed refresh job.
"""
@@ -275,7 +275,7 @@ class FunctionSignature(_RemoteValue):
class PythonEnvironmentSpec(_RemoteValue):
"""One Sophon-managed Python environment source."""
"""One LanceDB Enterprise-managed Python environment source."""
kind: str
packages: tuple[str, ...] = ()
@@ -653,8 +653,8 @@ class FunctionBinding(_RemoteValue):
class RefreshColumnResult(_RemoteValue):
"""Terminal result of an expression-backed or Function-backed refresh Job.
Local jobs produce this value in process. LanceDB Cloud and Enterprise
decode the same value from the durable server-job terminal payload.
LanceDB Embedded jobs produce this value in process. LanceDB Distributed
decodes the same value from the durable server-job terminal payload.
"""
rows_assigned: _UInt64
python/python/lancedb/index.py
+5 -5
View File
@@ -145,9 +145,9 @@ class FTS:
Lance tokenizer can panic on malformed triples; see
https://github.com/lance-format/lance/issues/8812.
Remote tables use the same ``FTS()`` configuration and require a server
that supports JSON FTS. Wait for ``table.wait_for_index(["doc_idx"])``
before searching a newly created remote index with its default name.
LanceDB Distributed uses the same ``FTS()`` configuration and requires a
server that supports JSON FTS. Wait for ``table.wait_for_index(["doc_idx"])``
before searching a newly created index there with its default name.
Examples
--------
@@ -210,12 +210,12 @@ class FTS:
memory_limit : int, optional
The total memory limit in MiB for the local FTS build stage. The limit
is divided evenly among indexing workers. This build-only setting is
not persisted with the index and does not apply to remote tables.
not persisted with the index and does not apply to LanceDB Distributed.
num_workers : int, optional
The number of workers for a local FTS build. By default Lance uses
roughly half of the available CPU cores. The effective value is
limited by the available compute capacity. This build-only setting is
not persisted with the index and does not apply to remote tables.
not persisted with the index and does not apply to LanceDB Distributed.
document_granularity : DocumentGranularity, default ROW
``ROW`` treats the selected text in one table row as one document.
``LIST_ELEMENT`` treats each element of the deepest list on the indexed
python/python/lancedb/materialized_view.py
+14 -11
View File
@@ -4,9 +4,10 @@
"""Materialized views: tables defined by a query over a source table and
maintained by refresh. See ``DBConnection.create_materialized_view``.
Local creation and refresh require the ``experimental-functions`` Cargo feature
in a source build. Published packages disable it. Opening, reading, listing and
dropping existing views and remote Cloud/Enterprise operations remain available.
LanceDB Embedded creation and refresh require the ``experimental-functions``
Cargo feature in a source build. Published packages disable it. Opening, reading,
listing and dropping existing views and LanceDB Distributed operations remain
available.
"""
from __future__ import annotations
@@ -188,13 +189,14 @@ class AsyncMaterializedView:
) -> "RefreshMaterializedViewResult":
"""Recompute the view from its source.
``full=True`` forces a remote rebuild; otherwise the server chooses
the refresh mode. Local views always rebuild, and ``source_version``
selects a source version instead of the latest. Remote SQL refreshes
do not support source-version pinning.
In LanceDB Distributed, ``full=True`` forces a rebuild; otherwise the
server chooses the refresh mode. LanceDB Embedded views always rebuild,
and ``source_version`` selects a source version instead of the latest.
LanceDB Distributed SQL refreshes do not support source-version pinning.
Remote results leave ``mode``, ``rows_written``, and ``source_version``
as ``None`` because the SQL job adapter does not expose these metrics.
LanceDB Distributed results leave ``mode``, ``rows_written``, and
``source_version`` as ``None`` because the SQL job adapter does not expose
these metrics.
"""
return await self._table._inner.refresh_materialized_view(
source_version=source_version, full=full
@@ -205,8 +207,9 @@ class AsyncMaterializedView:
) -> "AsyncJob[RefreshMaterializedViewResult]":
"""Submit a refresh and return its job without waiting.
The job may already be complete for a local view. Remote refreshes are
submitted through the SQL service. Options and result metrics follow
The job may already be complete in LanceDB Embedded. In LanceDB
Distributed, refreshes are submitted through the SQL service. Options and
result metrics follow
[refresh][lancedb.materialized_view.AsyncMaterializedView.refresh].
"""
from ._lancedb import RefreshMaterializedViewResult
python/python/lancedb/namespace.py
+1 -1
View File
@@ -1675,7 +1675,7 @@ class AsyncLanceNamespaceDBConnection:
) -> AsyncSqlQuery:
"""Start executing SQL when supported by the underlying connection.
Namespace-backed local connections do not support SQL.
Namespace-backed LanceDB Embedded connections do not support SQL.
"""
return await self._inner.execute_query_async(
query,
python/python/lancedb/permutation.py
+2 -2
View File
@@ -548,7 +548,7 @@ class Permutation:
picklable classes implementing ``__call__``. Lambdas and closures over
local variables don't pickle with the default protocol.
A factory is optional for normal local and remote LanceDB connections:
A factory is optional for normal LanceDB Embedded and Distributed connections:
if not set, ``__getstate__`` captures the table's own picklable reopen
state. Use a factory when that default state is not enough, for example
when credentials should be loaded from the worker environment instead
@@ -587,7 +587,7 @@ class Permutation:
{"root": "/data/lance_db"},
)
LanceDB Cloud, reading credentials from env vars at worker startup
LanceDB Enterprise, reading credentials from env vars at worker startup
so secrets aren't pickled into the dataset::
import os, lancedb
python/python/lancedb/query.py
+10 -7
View File
@@ -125,7 +125,7 @@ def _supports_local_blob_fetch(
def _blob_bytes_without_snapshot_error() -> NotImplementedError:
return NotImplementedError(
"blob_mode='bytes' cannot safely materialize blob columns without a "
"stable table snapshot. Remote tables and namespace QueryTable pushdown "
"stable table snapshot. LanceDB Distributed and namespace QueryTable pushdown "
"run the query on a server, so the blob fetch could read another table "
"version. Use blob_mode='descriptions'."
)
@@ -1147,7 +1147,8 @@ class LanceQueryBuilder(ABC):
mode and for legacy blob columns. Other local blob v2
queries return descriptors in ``"lazy"`` and ``"descriptions"``
modes, or payloads in ``"bytes"`` mode, through :meth:`to_arrow`.
Remote blob queries support only ``"descriptions"`` until a stable
LanceDB Distributed blob queries support only ``"descriptions"`` until a
stable
table snapshot can bind the query and blob fetches.
**kwargs
Forwarded to pyarrow.Table.to_pandas after query execution and
@@ -3483,7 +3484,7 @@ class AsyncQueryBase(object):
``fetch_blobs`` reads. ``"descriptions"`` returns them without it.
``"bytes"`` returns payload bytes and raises for legacy blob columns
and blobs inside lists.
Remote tables and namespace query pushdown refuse ``"bytes"``.
LanceDB Distributed and namespace query pushdown refuse ``"bytes"``.
"""
blob_sources = await self._validated_blob_sources(blob_mode)
batch_iter = await self.to_batches(timeout=timeout)
@@ -3562,7 +3563,8 @@ class AsyncQueryBase(object):
mode and for legacy blob columns. Other local blob v2
queries return descriptors in ``"lazy"`` and ``"descriptions"``
modes, or payloads in ``"bytes"`` mode, through :meth:`to_arrow`.
Remote blob queries support only ``"descriptions"`` until a stable
LanceDB Distributed blob queries support only ``"descriptions"`` until a
stable
table snapshot can bind the query and blob fetches.
**kwargs
Forwarded to pyarrow.Table.to_pandas after query execution and
@@ -4209,9 +4211,10 @@ class AsyncVectorQueryBase:
your actual data to find the smallest possible value that will still give
you the desired recall.
LanceDB retains this as `nprobes` through local and remote request
construction. Lance sets both probe bounds to this value. Explicit minimum
or maximum settings can override their respective bounds.
LanceDB retains this as `nprobes` through LanceDB Embedded and LanceDB
Distributed request construction. Lance sets both probe bounds to this
value. Explicit minimum or maximum settings can override their respective
bounds.
"""
self._inner.nprobes(nprobes)
return self
python/python/lancedb/remote/__init__.py
+2 -2
View File
@@ -163,7 +163,7 @@ class TlsConfig:
@dataclass
class ClientConfig:
"""Configuration for the LanceDB Cloud HTTP client.
"""Configuration for the HTTP client used to connect to LanceDB Enterprise.
Attributes
----------
@@ -190,7 +190,7 @@ class ClientConfig:
Provider for dynamic headers to be added to each request.
user_id: Optional[str]
User identifier for tracking purposes. This is sent as the
`x-lancedb-user-id` header in requests to LanceDB Cloud/Enterprise.
`x-lancedb-user-id` header in requests to LanceDB Enterprise.
This can also be set via the `LANCEDB_USER_ID` environment variable.
Alternatively, set `LANCEDB_USER_ID_ENV_KEY` to specify another
python/python/lancedb/remote/db.py
+16 -12
View File
@@ -101,7 +101,7 @@ def _tls_config_to_dict(config: Optional[TlsConfig]) -> Optional[dict[str, Any]]
def _client_config_to_dict(config: ClientConfig) -> dict[str, Any]:
if config.header_provider is not None:
raise ValueError(
"Cannot serialize a remote connection with a header_provider. "
"Cannot serialize a LanceDB Distributed connection with a header_provider. "
"Use static api_key/extra_headers or provide a worker-side "
"connection factory instead."
)
@@ -119,7 +119,7 @@ def _client_config_to_dict(config: ClientConfig) -> dict[str, Any]:
class RemoteDBConnection(DBConnection):
"""A connection to a remote LanceDB database."""
"""A connection to a LanceDB Enterprise database."""
def __init__(
self,
@@ -135,7 +135,7 @@ class RemoteDBConnection(DBConnection):
read_consistency_interval: Optional[timedelta] = None,
sql_host_override: Optional[str] = None,
):
"""Connect to a remote LanceDB database."""
"""Connect to a LanceDB Enterprise database."""
if isinstance(client_config, dict):
client_config = ClientConfig(**client_config)
elif client_config is None:
@@ -246,14 +246,17 @@ class RemoteDBConnection(DBConnection):
@property
def session(self) -> Optional["Session"]:
"""Remote connections do not expose a local cache session."""
raise NotImplementedError("session is not supported by remote connections")
"""LanceDB Distributed connections do not expose a local cache session."""
raise NotImplementedError(
"session is not supported by LanceDB Distributed connections"
)
@property
def read_consistency_interval(self) -> Optional[timedelta]:
"""Remote connections do not expose a local read consistency interval."""
"""LanceDB Distributed connections do not expose a read consistency interval."""
raise NotImplementedError(
"read_consistency_interval is not supported by remote connections"
"read_consistency_interval is not supported by LanceDB Distributed "
"connections"
)
@override
@@ -631,11 +634,11 @@ class RemoteDBConnection(DBConnection):
The embedding functions to store in the table schema and use to
generate vectors when creating the table or adding data.
storage_options: dict, optional
Not supported for remote table creation. Raises NotImplementedError.
Not supported in LanceDB Distributed. Raises NotImplementedError.
data_storage_version: str, optional
Not supported for remote table creation. Raises NotImplementedError.
Not supported in LanceDB Distributed. Raises NotImplementedError.
enable_v2_manifest_paths: bool, optional
Not supported for remote table creation. Raises NotImplementedError.
Not supported in LanceDB Distributed. Raises NotImplementedError.
Returns
-------
@@ -708,7 +711,8 @@ class RemoteDBConnection(DBConnection):
):
if value is not None:
raise NotImplementedError(
f"{option} is not supported for remote table creation"
f"{option} is not supported for table creation in "
"LanceDB Distributed"
)
if exist_ok:
if mode == "create":
@@ -1145,7 +1149,7 @@ class RemoteDBConnection(DBConnection):
default_namespace_path: Optional[List[str]] = None,
parameters: Optional[QueryParameters] = None,
) -> SqlQuery:
"""Start executing SQL through this remote connection.
"""Start executing SQL through this LanceDB Distributed connection.
Unqualified tables use this connection's database and the
``["public"]`` namespace by default. Fully qualified table names may
python/python/lancedb/remote/header.py
+1 -1
View File
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Header providers for LanceDB remote connections.
"""Header providers for LanceDB Enterprise connections.
This module provides a flexible header management framework for LanceDB remote
connections, allowing users to implement custom header strategies for
python/python/lancedb/remote/table.py
+21 -21
View File
@@ -126,8 +126,8 @@ class RemoteTable(Table):
def _serialized_connection_state(self) -> str:
if self._connection_state is None:
raise RuntimeError(
"Cannot reopen this remote table because it does not carry "
"serialized connection state"
"Cannot reopen this LanceDB Distributed table because it does not "
"carry serialized connection state"
)
if callable(self._connection_state):
self._connection_state = self._connection_state()
@@ -273,32 +273,32 @@ class RemoteTable(Table):
return LOOP.run(self._table.list_versions())
def to_arrow(self, *, blob_mode: BlobMode = "lazy") -> pa.Table:
"""to_arrow() is not supported for remote tables."""
"""to_arrow() is not supported in LanceDB Distributed."""
validate_blob_mode(blob_mode)
raise NotImplementedError(
"to_arrow() is not supported for remote tables. "
"to_arrow() is not supported in LanceDB Distributed. "
"Use table.search().to_arrow() instead."
)
def to_pandas(self, blob_mode: BlobMode = "lazy", **kwargs):
"""to_pandas() is not supported for remote tables."""
"""to_pandas() is not supported in LanceDB Distributed."""
validate_blob_mode(blob_mode)
raise NotImplementedError(
"to_pandas() is not supported for remote tables. "
"to_pandas() is not supported in LanceDB Distributed. "
"Use table.search().to_pandas() instead."
)
def to_lance(self, **kwargs) -> "lance.LanceDataset":
"""Direct Lance dataset access is not supported for remote tables.
"""Direct Lance dataset access is not supported in LanceDB Distributed.
Use ``table.search().to_arrow()`` to read query results through the server.
"""
return LOOP.run(self._table.to_lance(**kwargs))
def to_polars(self, batch_size=None) -> "pl.LazyFrame":
"""Use ``table.search().to_polars()`` to read remote query results."""
"""Use ``table.search().to_polars()`` to read query results."""
raise NotImplementedError(
"to_polars() is not supported for remote tables. "
"to_polars() is not supported in LanceDB Distributed. "
"Use table.search().to_polars() instead."
)
@@ -331,7 +331,7 @@ class RemoteTable(Table):
"""Tokenize a query using the tokenizer configured on an FTS index.
Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
rebuilt in the client process from index metadata. For remote tables,
rebuilt in the client process from index metadata. In LanceDB Distributed,
this means the same tokenizer model files must also exist locally
(under ``LANCE_LANGUAGE_MODEL_HOME``) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
@@ -638,7 +638,7 @@ class RemoteTable(Table):
if accelerator is not None:
logging.warning(
"GPU accelerator is not supported for remote tables "
"GPU accelerator is not supported in LanceDB Distributed "
"and will be ignored."
)
@@ -713,7 +713,7 @@ class RemoteTable(Table):
) -> AddResult:
"""Add more data to the [Table][lancedb.table.Table].
It has the same API signature as the OSS version.
It has the same API signature as the LanceDB Embedded version.
Parameters
----------
@@ -743,7 +743,7 @@ class RemoteTable(Table):
capped at the number of CPU cores. Lower this if bulk ingestion is
using too much memory.
allow_external_blob_outside_bases: bool, default False
Not supported for remote tables. Setting this raises.
Not supported in LanceDB Distributed. Setting this raises.
Returns
-------
@@ -1048,20 +1048,20 @@ class RemoteTable(Table):
delete_unverified: bool = False,
):
"""
cleanup_old_versions() is a no-op for remote tables.
cleanup_old_versions() is a no-op in LanceDB Distributed.
"""
warnings.warn(
"cleanup_old_versions() is a no-op for remote tables.",
"cleanup_old_versions() is a no-op in LanceDB Distributed.",
stacklevel=2,
)
pass
def compact_files(self, *args, **kwargs):
"""
compact_files() is a no-op for remote tables.
compact_files() is a no-op in LanceDB Distributed.
"""
warnings.warn(
"compact_files() is a no-op for remote tables.",
"compact_files() is a no-op in LanceDB Distributed.",
stacklevel=2,
)
pass
@@ -1074,7 +1074,7 @@ class RemoteTable(Table):
retrain: bool = False,
compaction_options: Optional[CompactionOptions] = None,
):
"""optimize() is not supported for remote tables."""
"""optimize() is not supported in LanceDB Distributed."""
return LOOP.run(
self._table.optimize(
cleanup_older_than=cleanup_older_than,
@@ -1184,7 +1184,7 @@ class RemoteTable(Table):
)
def close_rte_writers(self) -> None:
"""No-op for remote tables (no local shard writers)."""
"""No-op in LanceDB Distributed (no local shard writers)."""
return LOOP.run(self._table.close_rte_writers())
@_deprecated_keyword("index_name", "name")
@@ -1213,7 +1213,7 @@ class RemoteTable(Table):
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
Currently only supported in LanceDB Distributed.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
@@ -1242,7 +1242,7 @@ class RemoteTable(Table):
def uri(self) -> str:
"""The table URI (storage location).
For remote tables, this fetches the location from the server via describe.
In LanceDB Distributed, this fetches the location from the server via describe.
"""
return LOOP.run(self._table.uri())
python/python/lancedb/sql.py
+1 -1
View File
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Handles to SQL queries running on a remote database."""
"""Handles to SQL queries running on a LanceDB Distributed database."""
from typing import Any, Mapping, Optional, Sequence, Union
from uuid import UUID
python/python/lancedb/table.py
+58 -51
View File
@@ -1572,9 +1572,10 @@ class Table(ABC):
raise NotImplementedError("to_arrow is not supported for this table type")
def to_lance(self, **kwargs) -> lance.LanceDataset:
"""Return a local table as a lance.LanceDataset.
"""Return a LanceDB Embedded table as a lance.LanceDataset.
Remote tables cannot be opened as Lance datasets by the client. Use
LanceDB Distributed tables cannot be opened as Lance datasets by the
client. Use
``table.search().to_arrow()`` to read them through the server.
Returns
@@ -1586,7 +1587,7 @@ class Table(ABC):
def to_polars(self, batch_size=None) -> "pl.LazyFrame":
"""Return the table as a Polars LazyFrame.
For remote tables, use ``table.search().to_polars()`` to convert query
In LanceDB Distributed, use ``table.search().to_polars()`` to convert query
results to a Polars DataFrame.
Parameters
@@ -2530,7 +2531,7 @@ class Table(ABC):
) -> pa.LargeBinaryArray:
"""Materialize full blob bytes for ``column`` at the given rows.
Local tables support both legacy blob v1 and blob v2 columns.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
The result has the same length and order as ``row_ids``. Null blobs
produce null slots; valid empty blobs produce ``b""``.
@@ -2542,7 +2543,7 @@ class Table(ABC):
Blobs inside lists cannot be fetched by row id yet.
Remote servers limit each request to 1024 row IDs and 64 MiB of blob
LanceDB Distributed limits each request to 1024 row IDs and 64 MiB of blob
bytes. The client splits requests automatically and reads an individual
larger blob through the Range route. This method still materializes all
bytes in memory; for large values use :meth:`fetch_blob_files`.
@@ -2556,7 +2557,7 @@ class Table(ABC):
) -> pa.LargeBinaryArray:
"""Materialize row-specific byte ranges from a blob column.
Local tables support both legacy blob v1 and blob v2 columns.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
Each request is a three-item ``(row_id, offset, length)`` sequence.
Requests may be repeated or reordered, including multiple ranges for
the same blob.
@@ -2569,7 +2570,7 @@ class Table(ABC):
results are returned.
Row IDs can be obtained from a query with ``with_row_id(True)``.
Remote tables read the requested ranges through version-pinned blob
LanceDB Distributed reads the requested ranges through version-pinned blob
handles using HTTP byte-range requests.
"""
@@ -2581,12 +2582,12 @@ class Table(ABC):
) -> "list[Optional[BlobFile]]":
"""Open lazy, seekable :class:`~lancedb._blob.BlobFile` handles.
Local tables support both legacy blob v1 and blob v2 columns.
LanceDB Embedded supports both legacy blob v1 and blob v2 columns.
Prefer this over :meth:`fetch_blobs` for large payloads. ``row_ids`` is
a ``list[int]``, an integer Arrow array or chunked array, or a query
``pyarrow.Table`` carrying row identity via ``_rowid`` or a
``_lance_row_id`` field on the blob descriptor. Null blob values return
``None``. Remote tables require LanceDB Cloud server 0.5.0 or newer.
``None``. LanceDB Distributed requires server version 0.5.0 or newer.
``_rowid`` values stay valid after compaction when the table has stable
row ids.
@@ -2823,8 +2824,8 @@ class Table(ABC):
Notes
-----
This function is not available in LanceDb Cloud (since LanceDB
Cloud manages cleanup for you automatically)
This function is not available in LanceDB Distributed, which manages
cleanup for you automatically.
"""
@abstractmethod
@@ -2845,8 +2846,8 @@ class Table(ABC):
Notes
-----
This function is not available in LanceDB Cloud (since LanceDB
Cloud manages compaction for you automatically)
This function is not available in LanceDB Distributed, which manages
compaction for you automatically.
"""
@abstractmethod
@@ -2932,7 +2933,7 @@ class Table(ABC):
Specify exactly one of ``column`` or ``index_name``.
Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
rebuilt in the client process from index metadata. For remote tables,
rebuilt in the client process from index metadata. In LanceDB Distributed,
this means the same tokenizer model files must also exist locally
(under ``LANCE_LANGUAGE_MODEL_HOME``) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
@@ -2988,12 +2989,12 @@ class Table(ABC):
or named-struct result in the named table column. A bare
named-struct application expands its ordered result fields as one
atomic binding; aliases come from ``rename(columns=...)``.
Function columns are supported only on LanceDB Cloud and
Enterprise.
Function columns are supported only in LanceDB
Distributed.
computed: Dict[str, str], optional
Local declaration and refresh require a source build with the
LanceDB Embedded declaration and refresh require a source build with the
``experimental-functions`` Cargo feature, which published packages
do not enable. Remote Cloud/Enterprise operations remain available.
do not enable. LanceDB Distributed operations remain available.
A mapping from output column names to SQL expressions derives each
output field from its expression. A direct projection of a Blob v2
@@ -3012,14 +3013,15 @@ class Table(ABC):
While a declaration reads a column, that column cannot be renamed,
retyped or dropped.
On LanceDB Cloud and Enterprise the expression is planned by the
In LanceDB Distributed the expression is planned by the
server, and the refresh runs as a server job -- see
[`refresh_column_async`][lancedb.table.Table.refresh_column_async].
Cannot be combined with ``transforms``.
auto_refresh_enabled: bool, optional
Enable automatic refresh for each declared computed output. Defaults
to True. Local tables store this option; scheduling requires a server.
to True. LanceDB Embedded stores this option; scheduling requires a
server.
auto_refresh_interval_seconds: int, optional
Positive minimum interval after a successful refresh, in seconds.
When omitted, the server determines when to refresh. Applies to every
@@ -3059,7 +3061,8 @@ class Table(ABC):
by the next one, and rows whose inputs changed since they were computed
are recomputed; everything else is left as it is.
Local tables only: a remote refresh runs as a server job, through
LanceDB Embedded only. In LanceDB Distributed, a refresh runs as a
server job, through
[`refresh_column_async`][lancedb.table.Table.refresh_column_async].
Parameters
@@ -3080,14 +3083,15 @@ class Table(ABC):
Like :meth:`refresh_column`, but returns a handle to the refresh job
instead of blocking until it completes.
Local refresh requires a source build with ``experimental-functions``.
Remote Cloud/Enterprise jobs do not require this feature.
LanceDB Embedded refresh requires a source build with
``experimental-functions``.
LanceDB Distributed jobs do not require this feature.
The job may already be complete when returned; callers must not assume
the column is filled until :meth:`Job.wait` returns. Invalid input --
an unknown column, or one that is not computed -- raises here rather
than failing the job. On local tables the job runs in-process; on
LanceDB Cloud and Enterprise it is the server's backfill job.
than failing the job. In LanceDB Embedded the job runs in-process; in
LanceDB Distributed it is the server's backfill job.
Returns
-------
@@ -3123,7 +3127,7 @@ class Table(ABC):
A refresh running under a skip policy records each row it skipped
with the input that failed and the error. This lists those records,
newest job first, plus a summary for any fragment whose per-row
detail was capped. LanceDB Cloud and Enterprise only; reading errors
detail was capped. LanceDB Distributed only; reading errors
needs read access to the table, since a message carries the value
that failed.
@@ -4186,7 +4190,7 @@ class LanceTable(Table):
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
Currently only supported in LanceDB Distributed.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
@@ -4435,7 +4439,7 @@ class LanceTable(Table):
using too much memory.
allow_external_blob_outside_bases: bool, default False
Allow blob URIs outside registered bases. See :meth:`Table.add`.
Local tables only.
LanceDB Embedded only.
Returns
-------
@@ -5077,7 +5081,7 @@ class LanceTable(Table):
Specify exactly one of ``column`` or ``index_name``.
Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
rebuilt in the client process from index metadata. For remote tables,
rebuilt in the client process from index metadata. In LanceDB Distributed,
this means the same tokenizer model files must also exist locally
(under ``LANCE_LANGUAGE_MODEL_HOME``) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
@@ -6071,9 +6075,10 @@ class AsyncTable:
return AsyncQuery(self._inner.query(), self)
async def to_lance(self, **kwargs) -> lance.LanceDataset:
"""Return the Lance dataset backing a local table.
"""Return the Lance dataset backing a LanceDB Embedded table.
Remote tables cannot be opened as Lance datasets by the client. Use
LanceDB Distributed tables cannot be opened as Lance datasets by the
client. Use
[to_arrow][lancedb.table.AsyncTable.to_arrow] or
[query][lancedb.table.AsyncTable.query] to read them through the server.
@@ -6094,7 +6099,7 @@ class AsyncTable:
"""
if not self._inner._is_native():
raise NotImplementedError(
"to_lance() is not supported for remote tables; "
"to_lance() is not supported in LanceDB Distributed; "
"query the server instead"
)
@@ -6127,7 +6132,7 @@ class AsyncTable:
Parameters
----------
blob_mode: str, default "lazy"
Controls how Lance blob columns are returned. Remote tables support
Controls how Lance blob columns are returned. LanceDB Distributed supports
"descriptions"; "bytes" and "lazy" are not yet supported.
**kwargs
Forwarded to PyArrow / Lance pandas conversion.
@@ -6371,7 +6376,7 @@ class AsyncTable:
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
Currently only supported in LanceDB Distributed.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
@@ -6415,8 +6420,8 @@ class AsyncTable:
"""
Get the table URI (storage location).
For remote tables, this fetches the location from the server via describe.
For local tables, this returns the dataset URI.
In LanceDB Distributed, this fetches the location from the server via describe.
In LanceDB Embedded, this returns the dataset URI.
Returns
-------
@@ -6499,7 +6504,7 @@ class AsyncTable:
using too much memory.
allow_external_blob_outside_bases: bool, default False
Allow blob URIs outside registered bases. See :meth:`Table.add`.
Local tables only.
LanceDB Embedded only.
"""
schema = await self.schema()
@@ -7203,12 +7208,12 @@ class AsyncTable:
or named-struct result in the named table column. A bare
named-struct application expands its ordered result fields as one
atomic binding; aliases come from ``rename(columns=...)``.
Function columns are supported only on LanceDB Cloud and
Enterprise.
Function columns are supported only in LanceDB
Distributed.
computed: Dict[str, str], optional
Local declaration and refresh require a source build with the
LanceDB Embedded declaration and refresh require a source build with the
``experimental-functions`` Cargo feature, which published packages
do not enable. Remote Cloud/Enterprise operations remain available.
do not enable. LanceDB Distributed operations remain available.
A mapping from output column names to SQL expressions derives each
output field from its expression. A direct projection of a Blob v2
@@ -7226,13 +7231,13 @@ class AsyncTable:
While a declaration reads a column, that column cannot be renamed,
retyped or dropped.
On LanceDB Cloud and Enterprise the expression is planned by
In LanceDB Distributed the expression is planned by
the server. Cannot be combined with ``transforms``.
auto_refresh_enabled: bool, optional
Enable automatic refresh for each computed output declared by this
call. Defaults to True. Local tables store the setting; automatic
scheduling requires Cloud or Enterprise.
call. Defaults to True. LanceDB Embedded stores the setting; automatic
scheduling requires LanceDB Distributed.
auto_refresh_interval_seconds: int, optional
Positive minimum seconds after a successful refresh before another
automatic refresh is eligible. When omitted, the server determines when
@@ -7337,7 +7342,8 @@ class AsyncTable:
by the next one, and rows whose inputs changed since they were computed
are recomputed; everything else is left as it is.
Local tables only: a remote refresh runs as a server job, through
LanceDB Embedded only. In LanceDB Distributed, a refresh runs as a
server job, through
[`refresh_column_async`][lancedb.table.Table.refresh_column_async].
Parameters
@@ -7364,7 +7370,7 @@ class AsyncTable:
A refresh running under a skip policy records each row it skipped
with the input that failed and the error. This lists those records,
newest job first, plus a summary for any fragment whose per-row
detail was capped. LanceDB Cloud and Enterprise only; reading errors
detail was capped. LanceDB Distributed only; reading errors
needs read access to the table, since a message carries the value
that failed.
@@ -7402,14 +7408,15 @@ class AsyncTable:
Like :meth:`refresh_column`, but returns a handle to the refresh job
instead of blocking until it completes.
Local refresh requires a source build with ``experimental-functions``.
Remote Cloud/Enterprise jobs do not require this feature.
LanceDB Embedded refresh requires a source build with
``experimental-functions``.
LanceDB Distributed jobs do not require this feature.
The job may already be complete when returned; callers must not assume
the column is filled until :meth:`AsyncJob.wait` resolves. Invalid
input -- an unknown column, or one that is not computed -- raises here
rather than failing the job. On local tables the job runs
in-process; on LanceDB Cloud and Enterprise it is the server's
rather than failing the job. In LanceDB Embedded the job runs
in-process; in LanceDB Distributed it is the server's
backfill job.
Returns
@@ -7831,7 +7838,7 @@ class AsyncTable:
Specify exactly one of ``column`` or ``index_name``.
Model-backed tokenizers such as ``jieba/*`` and ``lindera/*`` are
rebuilt in the client process from index metadata. For remote tables,
rebuilt in the client process from index metadata. In LanceDB Distributed,
this means the same tokenizer model files must also exist locally
(under ``LANCE_LANGUAGE_MODEL_HOME``) and must match the files the
server uses. A mismatch cannot be detected: the returned tokens may
python/python/tests/test_materialized_views.py
+3 -3
View File
@@ -726,7 +726,7 @@ def test_embedded_create_refuses_automatic_refresh_before_creating_view(
tmp_path, options
):
db = make_db(tmp_path)
with pytest.raises(NotImplementedError, match="Cloud or Enterprise"):
with pytest.raises(NotImplementedError, match="requires LanceDB Distributed"):
db.create_materialized_view("adults", "people", **options)
assert "adults" not in db.table_names()
@@ -743,7 +743,7 @@ def test_remote_create_rejects_invalid_interval_without_request(seconds):
@pytest.mark.experimental_functions
def test_namespace_forwards_automatic_refresh_request(tmp_path):
db = lancedb.connect_namespace("dir", {"root": str(tmp_path)})
with pytest.raises(NotImplementedError, match="Cloud or Enterprise"):
with pytest.raises(NotImplementedError, match="requires LanceDB Distributed"):
db.create_materialized_view("mv", "source", auto_refresh_enabled=True)
@@ -751,7 +751,7 @@ def test_namespace_forwards_automatic_refresh_request(tmp_path):
@pytest.mark.asyncio
async def test_async_namespace_forwards_automatic_refresh_request(tmp_path):
db = lancedb.connect_namespace_async("dir", {"root": str(tmp_path)})
with pytest.raises(NotImplementedError, match="Cloud or Enterprise"):
with pytest.raises(NotImplementedError, match="requires LanceDB Distributed"):
await db.create_materialized_view("mv", "source", auto_refresh_enabled=True)
python/python/tests/test_remote_db.py
+11 -9
View File
@@ -910,7 +910,7 @@ def test_table_unimplemented_functions():
message = str(exc_info.value)
assert method in message
assert "not" in message and "supported" in message
assert "remote tables" in message
assert "LanceDB Distributed" in message
assert "cloud" not in message.lower()
assert f"search().{method}()" in message
@@ -977,7 +977,7 @@ async def test_remote_to_lance_rejection(
await table.to_lance(**options)
assert requests == opened_requests
assert "to_lance() is not supported for remote tables" in str(sync_error.value)
assert "to_lance() is not supported in LanceDB Distributed" in str(sync_error.value)
assert "query the server" in str(sync_error.value)
assert str(sync_error.value) == str(async_error.value)
@@ -1764,7 +1764,8 @@ def test_remote_create_index_new_api(caplog, rq_kwargs):
assert received_requests[5]["replace"] is False
assert "replace is not supported" not in caplog.text
assert caplog.messages == [
"GPU accelerator is not supported for remote tables and will be ignored."
"GPU accelerator is not supported in LanceDB Distributed "
"and will be ignored."
]
@@ -1822,7 +1823,7 @@ def test_remote_unsupported_operations_sync(method, args, kwargs, operation):
with pytest.raises(NotImplementedError) as error:
getattr(table, method)(*args, **kwargs)
expected = f"LanceDBError: not supported: {operation} is not supported "
expected += "for remote tables."
expected += "in LanceDB Distributed."
if operation == "IVF_HNSW_PQ":
expected += " Please use IVF_HNSW_SQ instead."
assert str(error.value) == expected
@@ -1840,7 +1841,7 @@ async def test_remote_unsupported_operations_async(method, args, kwargs, operati
with pytest.raises(NotImplementedError) as error:
await getattr(table, method)(*args, **kwargs)
expected = f"LanceDBError: not supported: {operation} is not supported "
expected += "for remote tables."
expected += "in LanceDB Distributed."
if operation == "IVF_HNSW_PQ":
expected += " Please use IVF_HNSW_SQ instead."
assert str(error.value) == expected
@@ -1852,7 +1853,7 @@ def test_remote_unsupported_legacy_hnsw_pq():
with pytest.warns(DeprecationWarning, match="create_index"):
with pytest.raises(
NotImplementedError,
match="IVF_HNSW_PQ is not supported for remote tables. "
match="IVF_HNSW_PQ is not supported in LanceDB Distributed. "
"Please use IVF_HNSW_SQ instead.",
):
table.create_index(
@@ -1870,14 +1871,15 @@ def test_remote_python_only_messages():
with pytest.raises(NotImplementedError) as error:
getattr(table, method)()
assert str(error.value) == (
f"{method}() is not supported for remote tables. "
f"{method}() is not supported in LanceDB Distributed. "
f"Use table.search().{method}() instead."
)
for method in ["compact_files", "cleanup_old_versions"]:
with pytest.warns(UserWarning) as warning:
getattr(table, method)()
assert (
str(warning[0].message) == f"{method}() is a no-op for remote tables."
str(warning[0].message)
== f"{method}() is a no-op in LanceDB Distributed."
)
@@ -3842,7 +3844,7 @@ async def test_async_remote_to_lance_rejects_storage_access(monkeypatch):
raise AssertionError("remote table URI must not be requested")
monkeypatch.setattr(table, "uri", fail_uri)
with pytest.raises(NotImplementedError, match="remote tables"):
with pytest.raises(NotImplementedError, match="LanceDB Distributed"):
await table.to_lance()
python/python/tests/test_remote_options.py
+7 -5
View File
@@ -86,7 +86,9 @@ async def test_remote_create_rejects_write_options(storage_options, empty):
if not empty:
kwargs["data"] = [{"value": 1}]
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="write_options.*remote"):
with pytest.raises(
NotImplementedError, match="write_options.*LanceDB Distributed"
):
await db.create_table("t", storage_options=storage_options, **kwargs)
assert requests == []
@@ -108,7 +110,7 @@ def test_sync_remote_create_rejects_write_options(options):
request.end_headers()
with mock_lancedb_connection(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
with pytest.raises(NotImplementedError, match="LanceDB Distributed"):
db.create_table("t", [{"value": 1}], **options)
assert requests == []
@@ -129,11 +131,11 @@ async def test_remote_open_rejects_local_options(options, asynchronous):
if asynchronous:
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
with pytest.raises(NotImplementedError, match="LanceDB Distributed"):
await db.open_table("t", **options)
else:
with mock_lancedb_connection(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
with pytest.raises(NotImplementedError, match="LanceDB Distributed"):
db.open_table("t", **options)
assert requests == []
@@ -149,6 +151,6 @@ async def test_remote_open_rejects_location():
request.wfile.write(b'{"version": 1, "schema": {"fields": []}}')
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="location.*remote"):
with pytest.raises(NotImplementedError, match="location.*LanceDB Distributed"):
await db.open_table("t", location="/tmp/zzz")
assert requests == []
python/python/tests/test_table.py
+3 -1
View File
@@ -5552,7 +5552,9 @@ async def test_computed_column_async(tmp_path):
def test_function_errors_are_remote_only(tmp_path):
db = lancedb.connect(tmp_path)
table = db.create_table("t", [{"x": 1}])
with pytest.raises(NotImplementedError, match="LanceDB Cloud and Enterprise"):
with pytest.raises(
NotImplementedError, match="recorded only in LanceDB Distributed"
):
table.function_errors()
python/src/table.rs
+4 -2
View File
@@ -1922,7 +1922,8 @@ impl Table {
inner
.as_native()
.ok_or_else(|| lancedb::Error::NotSupported {
message: "uses_v2_manifest_paths is not supported for remote tables.".into(),
message: "uses_v2_manifest_paths is not supported in LanceDB Distributed."
.into(),
})
.infer_error()?
.uses_v2_manifest_paths()
@@ -1937,7 +1938,8 @@ impl Table {
inner
.as_native()
.ok_or_else(|| lancedb::Error::NotSupported {
message: "migrate_manifest_paths_v2 is not supported for remote tables.".into(),
message: "migrate_manifest_paths_v2 is not supported in LanceDB Distributed."
.into(),
})
.infer_error()?
.migrate_manifest_paths_v2()