mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-27 14:06:14 +00:00
Jack Ye
9af7757053
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run.
101 lines
4.7 KiB
Plaintext
101 lines
4.7 KiB
Plaintext
---
|
|
title: "Jobs"
|
|
sidebarTitle: "Job"
|
|
description: "Track background work from submission to completion."
|
|
icon: hourglass-half
|
|
keywords: ["job", "background", "progress", "events", "cancel"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Job handles track operations in OSS and Enterprise. Enterprise also
|
|
supports listing server jobs, reopening them by ID and reading their events.
|
|
---
|
|
|
|
## Job handles {#convention}
|
|
|
|
A job handle lets you track an operation, such as an index build or a computed
|
|
column refresh. Use it to check progress, wait for completion and inspect
|
|
failures.
|
|
|
|
Methods such as Python's `create_index_async` and TypeScript's
|
|
`createIndexAsync` return job handles. In OSS, the work runs in your application
|
|
process; in Enterprise, it runs on server workers or through the SQL query
|
|
backend. A returned handle can represent an operation that has already finished,
|
|
so check its status before
|
|
waiting or requesting cancellation. See the operation's API reference for when
|
|
the method returns.
|
|
|
|
## Submit and wait for a job {#waiting}
|
|
|
|
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
|
|
registry job, also keep its ID if another process needs to look it up later.
|
|
2. **Check progress.** Read the current status while the operation runs.
|
|
3. **Wait for completion.** Check that the job succeeded before using its
|
|
result. If it failed or was cancelled, inspect the failure details and any
|
|
results it committed before stopping.
|
|
|
|
In Python, `job.status()` reports the current state and `job.wait()` waits for
|
|
completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting.
|
|
The timeout does not cancel the server job.
|
|
|
|
<span id="inspecting" />
|
|
<span id="events" />
|
|
|
|
## Inspect and cancel jobs {#managing}
|
|
|
|
| Task | What to do |
|
|
| --- | --- |
|
|
| Inspect progress | Check status and refresh the handle to retrieve details. Enterprise job events can help explain ongoing work or failures. |
|
|
| Reattach from another process | Open an existing server job with its ID, such as Python's `db.open_job(job_id)`. The server must still retain the job record. |
|
|
| Diagnose a failure | Inspect failure details and relevant events. Enterprise SQL also provides `SHOW ERRORS`. |
|
|
| Request cancellation | Call the job's cancellation method. Cancellation may take time, and work already committed is not rolled back. |
|
|
|
|
Reopening a job retrieves the existing operation; it does not restart failed
|
|
work. Treat the ID as an opaque identifier and pass it unchanged to the jobs API.
|
|
|
|
Event responses have [row limits](/basics/limits#jobs). An explicit limit can
|
|
request more than the default, but it does not guarantee a complete event
|
|
history.
|
|
|
|
## OSS and Enterprise execution {#in-process}
|
|
|
|
The following table describes local operations and Enterprise registry jobs.
|
|
SQL-backed job handles follow the query lifecycle described below.
|
|
|
|
| | OSS | Enterprise registry job |
|
|
| --- | --- | --- |
|
|
| Execution | In your application process | On server workers |
|
|
| Job identity | No server ID | A server ID for looking up the job |
|
|
| Event history | No server event history | Available through the jobs API |
|
|
| Reattaching after a client restart | No server job to reopen | Reopen by ID while the record is retained |
|
|
|
|
### SQL-backed job handles
|
|
|
|
Remote materialized-view CREATE and REFRESH use the SQL query backend. Python's
|
|
`create_materialized_view_async()` and `mv.refresh_async()` still return job
|
|
handles with `status()`, `wait()` and `cancel()`. Synchronous creation and refresh
|
|
wait on the same SQL execution before returning.
|
|
|
|
The existing `job.job_type` identifies query-backed handles as `"query"` from
|
|
submission, and `job.id` contains the connection-scoped SQL query UUID string.
|
|
Use `describe_query(job.id)` on the submitting connection to inspect the query.
|
|
Registry handles retain their server-reported job types (available after
|
|
`status()` or `refresh()`) and registry IDs. Local handles have no ID or job type.
|
|
|
|
SQL-backed handles do not expose registry records or event history and cannot
|
|
be reopened through `open_job()` or reattached after a client restart. Their IDs
|
|
are not durable resume tokens. Cancellation follows the SQL server's
|
|
capabilities and can be unsupported.
|
|
|
|
The server decides whether SQL executes on its serving fleet or starts SQL job
|
|
workers. That placement does not change the handle's query-backed identity.
|
|
|
|
## Learn more {#next}
|
|
|
|
- [Build and repair indexes](/indexing/reindexing): submit index work.
|
|
- [Column refresh](/feature-engineering/refresh-column): populate derived values.
|
|
- [Background jobs](/maintenance/background-jobs): understand Enterprise maintenance.
|
|
- [SQL job commands](/sql/ddl/job): list jobs and inspect errors.
|
|
- [SDKs](/basics/sdk): find the job API for your client.
|