mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 12:08:35 +00:00
feat: skill references to work with jobs (incl server connection) (#3683)
Some additions to our lancedb skill to enable agents to use the jobs methods that we recently added. Eval tests (below, with and without these additions to the skill) suggest that they're helping, mostly to find the right method calls. These are a little unusual because they require REST server connection, they're not yet implemented in the SDKs. ``` ┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐ │ eval │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 8-list-running-jobs │ 2.5/3 │ 3/3 │ +0.5 │ 123s │ 29s │ $0.58 │ $0.18 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 9-describe-job │ 1/5 │ 5/5 │ +4.0 │ 159s │ 52s │ $0.62 │ $0.25 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 10-cancel-job │ 3/3 │ 3/3 │ +0.0 │ 99s │ 35s │ $0.55 │ $0.21 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ TOTAL │ 6.5/11 │ 11/11 │ +4.5 │ 381s │ 116s │ $1.75 │ $0.65 │ └─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘ ``` Failure reasons are because the agent didn't know the right method to call, spent all its turns guessing REST calls, tried to inspect lancedb code, but didn't find the answer in here.
This commit is contained in:
Dan Tasse
GitHub
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: lancedb
|
||||
description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, and apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics.
|
||||
description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics, and resolve connections to the remote server for Enterprise-only operations such as jobs.
|
||||
---
|
||||
|
||||
# Building LanceDB Pipelines
|
||||
@@ -19,7 +19,7 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
|
||||
## Workflow
|
||||
|
||||
1. Identify the SDK: Python, TypeScript, or both.
|
||||
2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.
|
||||
2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path. If the task involves jobs in any way (listing, inspecting, creating, or canceling jobs), it is always the remote path and requires a remote server connection — see "Connecting to the LanceDB remote server" below before doing anything else.
|
||||
3. Read the matching language branch before writing or changing code:
|
||||
- Python patterns: `references/python/patterns.md`
|
||||
- Python API quick reference: `references/python/api_reference.md`
|
||||
@@ -29,7 +29,9 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
|
||||
- TypeScript performance guidance: `references/typescript/performance.md`
|
||||
- Column metadata authoring (both SDKs): `references/column_metadata.md`
|
||||
- Branch operations (both SDKs): `references/branch_ops.md`
|
||||
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main.
|
||||
- Remote server connection resolution (jobs, raw REST): `references/remote_connect.md`
|
||||
- Job operations REST API (list/describe/cancel/query_events): `references/remote_jobs.md`
|
||||
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events).
|
||||
5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
|
||||
6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
|
||||
7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
|
||||
@@ -70,6 +72,10 @@ Rules for portable Enterprise ingestion:
|
||||
|
||||
This is Enterprise/Cloud-specific. Local/OSS tables have no separate data plane, so `mode="overwrite"` and immediate same-name reuse are fine there.
|
||||
|
||||
## Connecting to the LanceDB remote server
|
||||
|
||||
LanceDB Enterprise/Cloud deployments are served by a server implementing the lance-namespace OpenAPI spec (<https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml>). Every remote (`db://...`) connection talks to such a server, and some operations exist only there. In particular, **all operations around jobs (listing, inspecting, creating, or canceling jobs) run server-side** — there is no local/OSS equivalent. Before any job work, or any direct REST call to an Enterprise deployment, read `references/remote_connect.md` to resolve the base URL, credentials, and database header and to validate the connection. Then use the four job REST methods documented in `references/remote_jobs.md` (list, describe, cancel, query_events).
|
||||
|
||||
## Script
|
||||
|
||||
Run the scanner when reviewing or modifying an existing codebase:
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Connecting to a LanceDB remote server
|
||||
|
||||
LanceDB Enterprise/Cloud deployments are served by a server implementing the
|
||||
lance-namespace OpenAPI spec
|
||||
(<https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml>).
|
||||
Every remote (`db://...`) connection talks to such a server, and some operations
|
||||
exist only there. In particular, all operations around jobs (listing, inspecting,
|
||||
creating, or canceling jobs) run server-side — there is no local/OSS equivalent, so
|
||||
resolve a server connection before attempting any job work. The job REST methods
|
||||
themselves are documented in `references/remote_jobs.md`.
|
||||
|
||||
Every request needs two things:
|
||||
|
||||
1. **Base URL** — the server endpoint
|
||||
2. **Credentials** — an API key (`x-api-key` header over REST), and usually a database name (`x-lancedb-database` header)
|
||||
|
||||
## Resolution steps
|
||||
|
||||
1. If the user already gave a URL and API key (or said which environment they're working against), use that.
|
||||
2. Otherwise, look for credentials already available in the environment:
|
||||
- Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
|
||||
- A server endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
|
||||
3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
|
||||
|
||||
## Validating the connection
|
||||
|
||||
Make a cheap authenticated request and check the status before starting real work:
|
||||
|
||||
```bash
|
||||
curl -s -w "\n%{http_code}" "{base_url}/v1/table/?limit=1" \
|
||||
-H "x-api-key: <key>" \
|
||||
-H "x-lancedb-database: <database>"
|
||||
```
|
||||
|
||||
- `200` — connection, key, and database header all good
|
||||
- `401` — API key missing or wrong
|
||||
- `400` mentioning a database header — this deployment expects `x-lancedb-database`
|
||||
|
||||
## Non-REST equivalents
|
||||
|
||||
The same credentials work through the SDKs and CLI:
|
||||
|
||||
- Python SDK: `lancedb.connect("db://<database>", api_key="<key>", host_override="<base_url>")`
|
||||
- TypeScript SDK: `await lancedb.connect("db://<database>", { apiKey: "<key>", hostOverride: "<base_url>" })`
|
||||
- `lancedb` CLI: a `[profiles.<name>]` entry in `~/.lancedb/config.toml` with `http_server_url`, `api_key`, `database`
|
||||
@@ -0,0 +1,151 @@
|
||||
# Job operations over the LanceDB remote server REST API
|
||||
|
||||
Jobs are server-side background operations on LanceDB Enterprise/Cloud — index builds,
|
||||
column backfills, materialized view refreshes, and similar async work. Endpoints that
|
||||
trigger async work (e.g. the column backfill or materialized view refresh endpoints)
|
||||
return a `job_id`; these four methods are how you track and manage those jobs.
|
||||
|
||||
Resolve the connection first — see `references/remote_connect.md`. All four methods
|
||||
are **POST** requests under `{base_url}/v1/jobs/` with JSON bodies, and take the usual
|
||||
`x-api-key` / `x-lancedb-database` headers. If every job call returns `501`, job APIs
|
||||
are disabled on that deployment (the server has no job registry configured) — report
|
||||
that rather than retrying.
|
||||
|
||||
## 1. List jobs — `POST /v1/jobs/list`
|
||||
|
||||
The body is optional; an empty body lists everything. All fields are filters:
|
||||
|
||||
```json
|
||||
{
|
||||
"limit": 100,
|
||||
"table_name": "my_table",
|
||||
"job_type": "...",
|
||||
"job_subtype": "...",
|
||||
"state": "...",
|
||||
"page_token": "..."
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -s -X POST "{base_url}/v1/jobs/list" \
|
||||
-H "x-api-key: <key>" -H "x-lancedb-database: <database>" \
|
||||
-H "content-type: application/json" \
|
||||
-d '{"table_name": "my_table"}'
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"job_id": "...",
|
||||
"table": "my_table",
|
||||
"job_type": "...",
|
||||
"job_subtype": "...",
|
||||
"state": "done",
|
||||
"created_at_millis": 1720000000000
|
||||
}
|
||||
],
|
||||
"page_token": "..."
|
||||
}
|
||||
```
|
||||
|
||||
A `page_token` in the response means there are more results — pass it back in the next
|
||||
request to continue. Note list rows use a lowercase `state` string, while describe uses
|
||||
an uppercase `job_state`.
|
||||
|
||||
## 2. Describe a job — `POST /v1/jobs/describe`
|
||||
|
||||
Body: `{"job_id": "<id>"}`. Returns full detail for one job:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "...",
|
||||
"job_type": "...",
|
||||
"job_subtype": "...",
|
||||
"job_state": "IN_PROGRESS",
|
||||
"creation_ms": 1720000000000,
|
||||
"spec": {},
|
||||
"status": {}
|
||||
}
|
||||
```
|
||||
|
||||
`job_state` is one of `IN_PROGRESS`, `CANCELLED`, `FAILED`, `DONE`. `spec` and `status`
|
||||
are job-type-specific JSON objects (the job's input specification and its current
|
||||
progress/status). Returns `404` for an unknown job id.
|
||||
|
||||
## 3. Cancel a job — `POST /v1/jobs/cancel`
|
||||
|
||||
Body: `{"job_id": "<id>"}`; response echoes `{"job_id": "<id>"}`. Cancellation is a
|
||||
service-level operation requiring the same administrative authorization as the
|
||||
`/admin` routes — a database-scoped API key that can list and describe jobs may still
|
||||
get a permission error here. Other errors: `404` unknown job, `409` state conflict
|
||||
(e.g. already in a terminal state), `429` too much write contention (safe to retry).
|
||||
|
||||
## 4. Query job event history — `POST /v1/jobs/query_events`
|
||||
|
||||
Returns the event history (state transitions, progress updates) for one or more jobs.
|
||||
Body: `{"job_id": "<id>"}` for one job, or `{"job_ids": ["<id>", ...]}` for a batch.
|
||||
Optional fields: `limit` (max event rows), `limit_per_job` (per job in a batch query),
|
||||
and `filter` — a SQL-like expression over the columns `state`, `updated_by`,
|
||||
`owner_component`, and `claim_entity`. (`full_text_search` is reserved and currently
|
||||
rejected as not implemented.)
|
||||
|
||||
The response is **not JSON** — it is an Arrow IPC stream
|
||||
(`content-type: application/vnd.apache.arrow.stream`). Decode it, e.g. in Python:
|
||||
|
||||
```python
|
||||
import pyarrow.ipc
|
||||
import requests
|
||||
|
||||
resp = requests.post(
|
||||
f"{base_url}/v1/jobs/query_events",
|
||||
headers={"x-api-key": key, "x-lancedb-database": database},
|
||||
json={"job_id": job_id},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
events = pyarrow.ipc.open_stream(resp.content).read_all()
|
||||
```
|
||||
|
||||
## Feature engineering (Geneva) jobs
|
||||
|
||||
Feature engineering jobs — UDF column backfills and materialized view refreshes run
|
||||
through Geneva — are tracked **separately** from the `/v1/jobs` registry above. Their
|
||||
records live in a `geneva_jobs` table inside the database itself (in the `__system`
|
||||
namespace), and you access them through a Python `geneva` connection rather than the
|
||||
REST endpoints above:
|
||||
|
||||
```python
|
||||
import geneva
|
||||
from geneva.jobs import JobStateManager
|
||||
|
||||
# Same credentials as lancedb.connect / the REST API
|
||||
conn = geneva.connect("db://<database>", api_key="<key>", host_override="<base_url>")
|
||||
jsm = JobStateManager(conn)
|
||||
|
||||
# List jobs. NOTE: status defaults to "RUNNING"; pass status=None for all jobs.
|
||||
# Statuses: PENDING | RUNNING | DONE | FAILED | CANCELLED
|
||||
jobs = jsm.list_jobs(table_name="my_table", status=None)
|
||||
|
||||
# Fetch one job by id (returns a list of JobRecord)
|
||||
records = jsm.get("<job_id>")
|
||||
```
|
||||
|
||||
Each `JobRecord` has `table_name`, `column_name`, `job_id`, `job_type`, `status`,
|
||||
`launched_at`, `completed_at`, `config`, `launched_by`, `manifest_id`, `cluster_name`,
|
||||
`metrics` (progress counters), `events` (human-readable history), and `updated_at`.
|
||||
For filters `list_jobs` doesn't support (e.g. time ranges), query the underlying table
|
||||
directly: `jsm.get_table(True).search().where("launched_at >= TIMESTAMP '...'")` —
|
||||
pass `True` to check out the latest version, since other processes update job state.
|
||||
|
||||
Stale-status caveat: nothing reaps dead Geneva jobs, so a job can sit in
|
||||
`RUNNING`/`PENDING` forever if its worker died. Treat a job as effectively `FAILED`
|
||||
when it has been running longer than ~36 hours, or its `updated_at` is more than ~2
|
||||
hours old (this matches the heuristic the Geneva console UI applies on read).
|
||||
|
||||
## Workflow tips
|
||||
|
||||
- To wait for async work (a backfill, an index build), poll `describe` until
|
||||
`job_state` leaves `IN_PROGRESS`; on `FAILED`, pull `status` and `query_events` for
|
||||
the failure detail.
|
||||
Reference in New Issue
Block a user