mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
175 lines
5.9 KiB
Plaintext
175 lines
5.9 KiB
Plaintext
---
|
|
title: "Job"
|
|
sidebarTitle: "Job"
|
|
description: "SHOW JOBS, SHOW JOB EVENTS, DESCRIBE JOB, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL."
|
|
icon: hourglass-half
|
|
keywords: ["show jobs", "show job events", "describe job", "show errors", "pause job", "resume job", "cancel job", "sql"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A [job](/basics/jobs) is not a catalog object (it has no schema and is not
|
|
namespaced), but it has its own SQL statements, grouped here the same way
|
|
the catalog objects are.
|
|
|
|
## SHOW JOBS {#show}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SHOW JOBS
|
|
SHOW JOBS FOR TABLE documents
|
|
SHOW JOBS FOR JOB '<parent_job_id>' [LIMIT n]
|
|
```
|
|
</CodeGroup>
|
|
|
|
The unfiltered command lists jobs in the connected database, newest first:
|
|
id, type, state, creation time -- the SQL counterpart of `list_jobs`. `FOR
|
|
TABLE` filters that listing by the current namespace's table name. `FOR JOB`
|
|
lists the parent's direct child jobs using the REST API's child model.
|
|
Child rows expose their ID, kind, phase, row progress and attempt; unknown
|
|
values remain NULL. Children are read-only diagnostic identities.
|
|
|
|
`FOR JOB` defaults to 1,000 children and accepts `LIMIT` from 1 to 10,000,
|
|
following backend continuation pages up to that limit. Child history can be
|
|
`available`, `not_yet_recorded`, or `unavailable` after archival.
|
|
|
|
<Note>
|
|
A job you completed weeks ago eventually stops appearing here -- it was
|
|
archived, not deleted. Two deployment-scheduled jobs sweep the registry on a
|
|
per-database schedule: `job_registry_archive` moves terminal jobs out of the
|
|
active registry into archival storage, and `job_events_export` separately
|
|
copies terminal jobs' event histories into a curated system table for
|
|
metering, where the deployment enables it. A job's events can survive in
|
|
that exported table after the job itself is archived.
|
|
</Note>
|
|
|
|
## SHOW JOB EVENTS {#events}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SHOW JOB EVENTS '<parent_or_child_job_id>' [LIMIT n]
|
|
```
|
|
</CodeGroup>
|
|
|
|
Events are returned newest first. Parent history does not include child
|
|
history. The default limit is 1,000; valid limits range from 1 to 10,000.
|
|
The result includes event sequence, state, timestamps, origin, claim and
|
|
resource/work fields. Child `kind` and `details` expose the REST API's
|
|
allowlisted diagnostics; raw parent payloads are excluded.
|
|
|
|
## DESCRIBE JOB {#describe}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
DESCRIBE JOB '<job_id>' [FORMAT TEXT | TABLE | JSON]
|
|
DESC JOB '<job_id>' [FORMAT TEXT | TABLE | JSON]
|
|
```
|
|
</CodeGroup>
|
|
|
|
Both spellings show a finished root job's timing profile. `TEXT` is the
|
|
default; `TABLE` returns typed phase/step rows and `JSON` returns the
|
|
profile as one document. A running job has no profile rows.
|
|
|
|
## SHOW ERRORS {#errors}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SHOW ERRORS ON <table> [FOR JOB '<job-id>'] [LIMIT <n>]
|
|
```
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SHOW ERRORS ON documents
|
|
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
|
|
```
|
|
</CodeGroup>
|
|
|
|
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports
|
|
fewer filled rows than you expected.
|
|
|
|
### Errors and limits {#errors-and-limits}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| `LIMIT` | defaults to 10,000; must be between 1 and 100,000 |
|
|
| Failed jobs scanned without `FOR JOB` | 32 most recent |
|
|
| Tables the server does not open directly | not supported |
|
|
|
|
Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the
|
|
listing stops at your `LIMIT`, the message says so. Narrow with `FOR JOB`
|
|
rather than raising the limit blindly.
|
|
|
|
<Note>
|
|
`SHOW JOBS`, `SHOW JOB EVENTS`, `DESCRIBE JOB` and `SHOW ERRORS` require
|
|
system `USAGE`, as do the equivalent REST diagnostics. Explicit table
|
|
listings additionally use the table-list authorization policy.
|
|
</Note>
|
|
|
|
## PAUSE JOB {#pause}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
PAUSE JOB 'j-ci-abc-def'
|
|
```
|
|
</CodeGroup>
|
|
|
|
## RESUME JOB {#resume}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
RESUME JOB 'j-ci-abc-def'
|
|
```
|
|
</CodeGroup>
|
|
|
|
## CANCEL JOB {#cancel}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CANCEL JOB 'j-ci-abc-def'
|
|
```
|
|
</CodeGroup>
|
|
|
|
The job id is always a quoted string literal. All three take exactly one and
|
|
nothing after it: a second token on the line is a syntax error rather than
|
|
being ignored. Pause requires the deployment's pause-write opt-in and a
|
|
resumable job type; it may return `pausing` while workers drain, and resume
|
|
may return `still_pausing` until drain finishes.
|
|
|
|
<Note>
|
|
`PAUSE JOB`, `RESUME JOB` and `CANCEL JOB` require system `OPERATE` -- a
|
|
broader privilege than the `USAGE` that reading job state needs, since
|
|
these change what a running job does.
|
|
</Note>
|
|
|
|
<Note>
|
|
Cancellation is a request, matching the SDK's
|
|
[`cancel()`](/basics/jobs#managing): it does not guarantee the job stops before
|
|
doing more work, and asking twice or cancelling a finished job does nothing.
|
|
`CANCEL JOB` itself can be unavailable on an Enterprise cluster with no job registry
|
|
configured, which is a deployment condition rather than something you can work
|
|
around from SQL.
|
|
</Note>
|
|
|
|
## Compatibility {#compatibility}
|
|
|
|
`SHOW JOBS FOR <table>` (without the `TABLE` keyword), `SHOW JOB SUMMARY`,
|
|
and `SHOW JOB SPLITS` retain their existing behavior as compatibility forms.
|
|
Prefer `SHOW JOBS FOR TABLE`/`FOR JOB` and `DESCRIBE JOB` going forward. See
|
|
[`SHOW JOBS FOR JOB`](#show) for the per-split diagnostics
|
|
form -- each split is a child job, listed with its own phase and progress.
|
|
See [Watching a refresh](/sql/dml/refresh#watching) for `SHOW JOB SPLITS`'s
|
|
full column reference and a `DESCRIBE JOB` example.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Query and job" icon="hourglass-half" href="/basics/jobs">
|
|
The handle these statements are the SQL view of.
|
|
</Card>
|
|
<Card title="Intelligent Auto-Maintenance" icon="brain" href="/enterprise/intelligent-auto-maintenance#cleanup">
|
|
A job type the deployment schedules on its own, without a call.
|
|
</Card>
|
|
</CardGroup>
|