mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-26 13:36:13 +00:00
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run.
This commit is contained in:
Jack Ye
@@ -320,6 +320,8 @@ class Job:
|
||||
@property
|
||||
def id(self) -> Optional[str]: ...
|
||||
@property
|
||||
def job_type(self) -> Optional[str]: ...
|
||||
@property
|
||||
def _state(self) -> Optional[str]: ...
|
||||
@property
|
||||
def _description(self) -> Optional[JobDescription]: ...
|
||||
|
||||
@@ -674,10 +674,11 @@ class DBConnection(EnforceOverrides):
|
||||
) -> Job[None]:
|
||||
"""Submit materialized-view creation and return its job.
|
||||
|
||||
The job may already be complete for a local database. On LanceDB
|
||||
Cloud and Enterprise, its ``id`` is the server job identifier from
|
||||
the ``202 Accepted`` create response. Wait for the job before opening
|
||||
or querying the view.
|
||||
The job may already be complete for a local database. Remote creation
|
||||
submits CREATE MATERIALIZED VIEW through the SQL server. Its job has
|
||||
``job_type == "query"`` and ``id`` identifies the connection-scoped
|
||||
query. Registry event history is unavailable.
|
||||
Wait for the job before opening or querying the view.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"materialized views are not supported on this connection type"
|
||||
@@ -1265,7 +1266,7 @@ class DBConnection(EnforceOverrides):
|
||||
"""
|
||||
raise NotImplementedError("SQL is not supported for this connection type")
|
||||
|
||||
def describe_query(self, query_id: UUID) -> QueryDescription:
|
||||
def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
|
||||
"""Describe a submitted SQL query by its connection-scoped id."""
|
||||
raise NotImplementedError("SQL is not supported for this connection type")
|
||||
|
||||
@@ -2869,7 +2870,9 @@ class AsyncConnection(object):
|
||||
) -> AsyncJob[None]:
|
||||
"""Submit materialized-view creation and return its job.
|
||||
|
||||
Wait for the returned job before opening or querying the view.
|
||||
Wait for the returned job before opening or querying the view. Remote
|
||||
creation uses SQL; ``job_type`` is ``"query"`` and ``id``
|
||||
identifies that query on this connection.
|
||||
"""
|
||||
source, native_options = await self._materialized_view_source(
|
||||
source, select, where, limit
|
||||
@@ -3409,8 +3412,14 @@ class AsyncConnection(object):
|
||||
)
|
||||
)
|
||||
|
||||
async def describe_query(self, query_id: UUID) -> QueryDescription:
|
||||
"""Describe a submitted SQL query by its connection-scoped id."""
|
||||
async def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
|
||||
"""Describe a submitted SQL query by its UUID or string ID.
|
||||
|
||||
For a handle with ``job_type == "query"``, pass its ``id`` on the
|
||||
connection that submitted it. Registry job IDs are not query IDs.
|
||||
"""
|
||||
if isinstance(query_id, str):
|
||||
query_id = UUID(query_id)
|
||||
return await self._inner.describe_query(query_id)
|
||||
|
||||
async def namespace_client(self) -> LanceNamespace:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
|
||||
|
||||
"""Handles to operations a server may run asynchronously."""
|
||||
"""Handles to asynchronous registry jobs, SQL queries, and local operations."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
@@ -44,12 +44,12 @@ class AsyncJob(Generic[T]):
|
||||
|
||||
@property
|
||||
def id(self) -> Optional[str]:
|
||||
"""Identifies the operation on the server that is running it.
|
||||
"""The operation's opaque ID, interpreted according to :attr:`job_type`.
|
||||
|
||||
Returned for correlating with server logs or the jobs API. Operations
|
||||
that run in this process have no server id and return `None`. The value
|
||||
is opaque: parsing it or storing it to resume the job later is not
|
||||
supported.
|
||||
Registry jobs expose a registry job ID; SQL-backed jobs expose a
|
||||
connection-scoped query ID as a string. Local operations return ``None``.
|
||||
Query IDs cannot be used with registry job APIs or to resume after a
|
||||
client restart.
|
||||
"""
|
||||
return self._inner.id if self._inner is not None else None
|
||||
|
||||
@@ -94,9 +94,11 @@ class AsyncJob(Generic[T]):
|
||||
"""Ask the backend for this job's current state, and for a server-side
|
||||
job its full record, then cache it for the properties below.
|
||||
|
||||
The properties are all `None` until this runs, because submitting an
|
||||
operation returns only a job id. `status` fetches the whole record too;
|
||||
`wait` records only the terminal state it establishes.
|
||||
Registry details are `None` until this runs, because submitting an
|
||||
operation does not fetch its full registry record. Query-backed jobs
|
||||
expose only lifecycle state through this method. For registry jobs,
|
||||
`status` fetches the whole record too; `wait` records only the terminal
|
||||
state it establishes.
|
||||
"""
|
||||
if self._inner is None:
|
||||
return
|
||||
@@ -114,11 +116,13 @@ class AsyncJob(Generic[T]):
|
||||
|
||||
@property
|
||||
def job_type(self) -> Optional[str]:
|
||||
"""The job's type, as the server names it.
|
||||
"""The job's kind of work, without contacting the backend.
|
||||
|
||||
`None` for an in-process job, which has no server-side record.
|
||||
SQL-backed handles report ``"query"`` immediately. Registry handles
|
||||
report the server's job type after :meth:`refresh` or :meth:`status`.
|
||||
In-process handles return ``None``.
|
||||
"""
|
||||
return self._field("job_type")
|
||||
return self._inner.job_type if self._inner is not None else None
|
||||
|
||||
@property
|
||||
def creation_ms(self) -> Optional[int]:
|
||||
@@ -294,7 +298,7 @@ class Job(Generic[T]):
|
||||
@property
|
||||
def job_type(self) -> Optional[str]:
|
||||
"""The job's type. See :attr:`AsyncJob.job_type`."""
|
||||
return self._field("job_type")
|
||||
return self._inner.job_type if self._inner is not None else None
|
||||
|
||||
@property
|
||||
def creation_ms(self) -> Optional[int]:
|
||||
|
||||
@@ -1683,7 +1683,7 @@ class AsyncLanceNamespaceDBConnection:
|
||||
parameters=parameters,
|
||||
)
|
||||
|
||||
async def describe_query(self, query_id: UUID) -> QueryDescription:
|
||||
async def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
|
||||
"""Describe a submitted SQL query when supported."""
|
||||
return await self._inner.describe_query(query_id)
|
||||
|
||||
|
||||
@@ -1134,7 +1134,7 @@ class RemoteDBConnection(DBConnection):
|
||||
)
|
||||
|
||||
@override
|
||||
def describe_query(self, query_id: UUID) -> QueryDescription:
|
||||
def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
|
||||
"""Describe a submitted SQL query by its connection-scoped id."""
|
||||
return LOOP.run(
|
||||
self._conn.describe_query(
|
||||
|
||||
@@ -1729,7 +1729,11 @@ def test_remote_registration_job_and_exact_version_reopen_round_trip():
|
||||
)
|
||||
registration = db.create_function_async(normalize_score)
|
||||
assert registration.id == "job-register"
|
||||
assert registration.job_type is None
|
||||
assert registration.status() == "finished"
|
||||
assert registration.job_type == "create_function"
|
||||
created = registration.wait()
|
||||
assert registration.job_type == "create_function"
|
||||
reopened = db.get_function("normalize_score", version=created.version)
|
||||
|
||||
assert created == reopened
|
||||
|
||||
@@ -8,6 +8,7 @@ import threading
|
||||
|
||||
import lancedb
|
||||
import pytest
|
||||
from lancedb.db import AsyncConnection
|
||||
from lancedb.materialized_view import MaterializedViewDefinition
|
||||
from lancedb.remote.db import RemoteDBConnection
|
||||
|
||||
@@ -56,7 +57,7 @@ def mock_remote_materialized_views():
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def mock_remote_materialized_view_create():
|
||||
def mock_remote_materialized_view_drop():
|
||||
requests = []
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
@@ -67,7 +68,8 @@ def mock_remote_materialized_view_create():
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
body = json.loads(self.rfile.read(length) or b"{}")
|
||||
requests.append((self.path, body))
|
||||
job_id = "mv-drop-123" if self.path.endswith("/drop") else "mv-create-123"
|
||||
assert self.path.endswith("/drop")
|
||||
job_id = "mv-drop-123"
|
||||
encoded = json.dumps({"job_id": job_id}).encode()
|
||||
self.send_response(202)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
@@ -98,7 +100,7 @@ def test_remote_list_uses_namespace_route():
|
||||
|
||||
|
||||
def test_remote_drop_async_returns_server_job():
|
||||
with mock_remote_materialized_view_create() as (host, requests):
|
||||
with mock_remote_materialized_view_drop() as (host, requests):
|
||||
db = lancedb.connect(
|
||||
"db://dev",
|
||||
api_key="fake",
|
||||
@@ -192,11 +194,13 @@ def test_create_and_refresh_jobs(tmp_path):
|
||||
"adults", "people", where="age >= 18", with_no_data=True
|
||||
)
|
||||
assert create_job.id is None
|
||||
assert create_job.job_type is None
|
||||
assert create_job.wait() is None
|
||||
|
||||
view = db.open_materialized_view("adults")
|
||||
refresh_job = view.refresh_async()
|
||||
assert refresh_job.id is None
|
||||
assert refresh_job.job_type is None
|
||||
result = refresh_job.wait()
|
||||
assert result.mode == "rebuild"
|
||||
assert result.rows_written == 2
|
||||
@@ -337,11 +341,13 @@ async def test_async_create_and_refresh_jobs(tmp_path):
|
||||
"adults", "people", with_no_data=True
|
||||
)
|
||||
assert create_job.id is None
|
||||
assert create_job.job_type is None
|
||||
assert await create_job.wait() is None
|
||||
|
||||
view = await db.open_materialized_view("adults")
|
||||
refresh_job = await view.refresh_async()
|
||||
assert refresh_job.id is None
|
||||
assert refresh_job.job_type is None
|
||||
result = await refresh_job.wait()
|
||||
assert result.mode == "rebuild"
|
||||
assert result.rows_written == 1
|
||||
@@ -567,6 +573,27 @@ def test_local_materialized_view_listing_pages_and_resume(tmp_path):
|
||||
assert list(resumed) == []
|
||||
|
||||
|
||||
class QueryBackedCreateJob:
|
||||
id = "0198f1b2-c3d4-7e5f-8123-456789abcdef"
|
||||
|
||||
job_type = "query"
|
||||
|
||||
async def wait(self):
|
||||
return None
|
||||
|
||||
async def status(self):
|
||||
return "finished"
|
||||
|
||||
|
||||
class CapturingCreateConnection:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
async def create_materialized_view_async(self, name, source, **options):
|
||||
self.calls.append((name, source, options))
|
||||
return QueryBackedCreateJob()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"options",
|
||||
[
|
||||
@@ -576,16 +603,28 @@ def test_local_materialized_view_listing_pages_and_resume(tmp_path):
|
||||
],
|
||||
)
|
||||
def test_remote_create_forwards_refresh_options_without_interval_default(options):
|
||||
with mock_remote_materialized_view_create() as (host, requests):
|
||||
db = lancedb.connect("db://dev", api_key="fake", host_override=host)
|
||||
job = db.create_materialized_view_async("adults", "people", **options)
|
||||
assert job.id == "mv-create-123"
|
||||
assert requests == [
|
||||
native = CapturingCreateConnection()
|
||||
db = RemoteDBConnection.__new__(RemoteDBConnection)
|
||||
db._conn = AsyncConnection(native)
|
||||
job = db.create_materialized_view_async("adults", "people", **options)
|
||||
assert job.id == QueryBackedCreateJob.id
|
||||
assert job.job_type == "query"
|
||||
assert not hasattr(job, "query_id")
|
||||
assert job.wait() is None
|
||||
assert job.status() == "finished"
|
||||
assert native.calls == [
|
||||
(
|
||||
"/v1/materialized_view/adults/create",
|
||||
"adults",
|
||||
"people",
|
||||
{
|
||||
"query": 'SELECT * FROM "people"',
|
||||
**options,
|
||||
"projections": None,
|
||||
"filter": None,
|
||||
"limit": None,
|
||||
"with_no_data": False,
|
||||
"auto_refresh_enabled": options.get("auto_refresh_enabled"),
|
||||
"auto_refresh_interval_seconds": options.get(
|
||||
"auto_refresh_interval_seconds"
|
||||
),
|
||||
},
|
||||
)
|
||||
]
|
||||
@@ -593,18 +632,18 @@ def test_remote_create_forwards_refresh_options_without_interval_default(options
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_remote_create_keeps_interval_unspecified():
|
||||
with mock_remote_materialized_view_create() as (host, requests):
|
||||
db = await lancedb.connect_async("db://dev", api_key="fake", host_override=host)
|
||||
job = await db.create_materialized_view_async(
|
||||
"adults",
|
||||
"people",
|
||||
auto_refresh_enabled=True,
|
||||
)
|
||||
assert job.id == "mv-create-123"
|
||||
assert requests[0][1] == {
|
||||
"query": 'SELECT * FROM "people"',
|
||||
"auto_refresh_enabled": True,
|
||||
}
|
||||
native = CapturingCreateConnection()
|
||||
db = AsyncConnection(native)
|
||||
job = await db.create_materialized_view_async(
|
||||
"adults", "people", auto_refresh_enabled=True
|
||||
)
|
||||
assert job.id == QueryBackedCreateJob.id
|
||||
assert job.job_type == "query"
|
||||
assert not hasattr(job, "query_id")
|
||||
assert await job.wait() is None
|
||||
assert await job.status() == "finished"
|
||||
assert native.calls[0][2]["auto_refresh_enabled"] is True
|
||||
assert native.calls[0][2]["auto_refresh_interval_seconds"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
@@ -625,7 +664,7 @@ def test_embedded_create_refuses_automatic_refresh_before_creating_view(
|
||||
|
||||
@pytest.mark.parametrize("seconds", [0, -1, float("nan"), float("inf")])
|
||||
def test_remote_create_rejects_invalid_interval_without_request(seconds):
|
||||
with mock_remote_materialized_view_create() as (host, requests):
|
||||
with mock_remote_materialized_view_drop() as (host, requests):
|
||||
db = lancedb.connect("db://dev", api_key="fake", host_override=host)
|
||||
with pytest.raises(ValueError, match="positive, finite"):
|
||||
db.create_materialized_view_async(
|
||||
|
||||
@@ -137,9 +137,23 @@ async def test_async_namespace_connection_rejects_sql(tmp_path):
|
||||
await connection.describe_query(NIL_QUERY_ID)
|
||||
|
||||
|
||||
def test_describe_query_requires_uuid():
|
||||
with pytest.raises(TypeError, match="UUID"):
|
||||
remote_connection().describe_query(str(NIL_QUERY_ID))
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("identity", [NIL_QUERY_ID, str(NIL_QUERY_ID)])
|
||||
async def test_describe_query_accepts_uuid_and_job_id_string(identity):
|
||||
calls = []
|
||||
|
||||
class NativeConnection:
|
||||
async def describe_query(self, query_id):
|
||||
calls.append(query_id)
|
||||
|
||||
connection = AsyncConnection(NativeConnection())
|
||||
await connection.describe_query(identity)
|
||||
assert calls == [NIL_QUERY_ID]
|
||||
|
||||
|
||||
def test_describe_query_rejects_registry_job_id():
|
||||
with pytest.raises(ValueError, match="UUID"):
|
||||
remote_connection().describe_query("registry-job-id")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
@@ -81,6 +81,11 @@ impl Job {
|
||||
self.inner.id().map(str::to_string)
|
||||
}
|
||||
|
||||
#[getter]
|
||||
pub fn job_type(&self) -> Option<String> {
|
||||
self.inner.job_type()
|
||||
}
|
||||
|
||||
pub fn status(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
|
||||
let inner = self_.inner.clone();
|
||||
future_into_py(
|
||||
|
||||
Reference in New Issue
Block a user