Jack Ye
21f11b4463
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
15 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / Connection
Class: abstract Connection
A LanceDB Connection that allows you to open tables and create new ones.
Connection could be local against filesystem or remote against a server.
A Connection is intended to be a long lived object and may hold open resources such as HTTP connection pools. This is generally fine and a single connection should be shared if it is going to be used many times. However, if you are finished with a connection, you may call close to eagerly free these resources. Any call to a Connection method after it has been closed will result in an error.
Closing a connection is optional. Connections will automatically be closed when they are garbage collected.
Any created tables are independent and will continue to work even if the underlying connection has been closed.
Methods
cancelJob()
abstract cancelJob(jobId): Promise<boolean>
Request cancellation of a server-side job by id.
Resolves to true if the server accepted the cancellation, false if no such job exists. Cancelling an already-terminal job is a no-op success.
Parameters
- jobId:
string
Returns
Promise<boolean>
cloneTable()
abstract cloneTable(
targetTableName,
sourceUri,
options?): Promise<Table>
Clone a table from a source table.
A shallow clone creates a new table that shares the underlying data files with the source table but has its own independent manifest. This allows both the source and cloned tables to evolve independently while initially sharing the same data, deletion, and index files.
Parameters
-
targetTableName:
stringThe name of the target table to create. -
sourceUri:
stringThe URI of the source table to clone from. -
options? Clone options.
-
options.isShallow?:
booleanWhether to perform a shallow clone (defaults to true). -
options.sourceTag?:
stringThe tag of the source table to clone. -
options.sourceVersion?:
numberThe version of the source table to clone. -
options.targetNamespacePath?:
string[] The namespace path for the target table (defaults to root namespace).
Returns
Promise<Table>
close()
abstract close(): void
Close the connection, releasing any underlying resources.
It is safe to call this method multiple times.
Any attempt to use the connection after it is closed will result in an error.
Returns
void
createEmptyTable()
createEmptyTable(name, schema, options)
abstract createEmptyTable(
name,
schema,
options?): Promise<Table>
Creates a new empty Table
Parameters
-
name:
stringThe name of the table. -
schema:
SchemaLikeThe schema of the table -
options?:
Partial<CreateTableOptions> Additional options (backwards compatibility)
Returns
Promise<Table>
createEmptyTable(name, schema, namespacePath, options)
abstract createEmptyTable(
name,
schema,
namespacePath?,
options?): Promise<Table>
Creates a new empty Table
Parameters
-
name:
stringThe name of the table. -
schema:
SchemaLikeThe schema of the table -
namespacePath?:
string[] The namespace path to create the table in (defaults to root namespace) -
options?:
Partial<CreateTableOptions> Additional options
Returns
Promise<Table>
createMaterializedView()
abstract createMaterializedView(
name,
source,
options?): Promise<MaterializedView>
Define a materialized view named name over the table source.
The view is created empty, with the query recorded in its schema
metadata; view.refresh() computes the rows. The view is a normal
table: it can be queried, indexed and searched, and it appears in
tableNames. The source table must have stable row ids (create it with
the newTableEnableStableRowIds storage option); they keep the view's
provenance valid across source compactions and cannot be enabled after
a table exists. Local databases only.
Parameters
-
name:
string -
source:
string -
options?
-
options.limit?:
number -
options.select?:
MaterializedViewSelect -
options.where?:
string
Returns
Promise<MaterializedView>
createNamespace()
abstract createNamespace(namespacePath, options?): Promise<CreateNamespaceResponse>
Create a new namespace at the given path.
Parameters
-
namespacePath:
string[] The namespace path to create. -
options?:
Partial<CreateNamespaceOptions> Creationmode("create" | "exist_ok" | "overwrite") and optionalpropertiesto attach to the namespace.
Returns
Promise<CreateNamespaceResponse>
The properties of the created namespace and an optional transaction id.
createTable()
createTable(options, namespacePath)
abstract createTable(options, namespacePath?): Promise<Table>
Creates a new Table and initialize it with new data.
Parameters
-
options:
object&Partial<CreateTableOptions> The options object. -
namespacePath?:
string[] The namespace path to create the table in (defaults to root namespace)
Returns
Promise<Table>
createTable(name, data, options)
abstract createTable(
name,
data,
options?): Promise<Table>
Creates a new Table and initialize it with new data.
Parameters
-
name:
stringThe name of the table. -
data:
TableLike|Record<string,unknown>[] Non-empty Array of Records to be inserted into the table -
options?:
Partial<CreateTableOptions> Additional options (backwards compatibility)
Returns
Promise<Table>
createTable(name, data, namespacePath, options)
abstract createTable(
name,
data,
namespacePath?,
options?): Promise<Table>
Creates a new Table and initialize it with new data.
Parameters
-
name:
stringThe name of the table. -
data:
TableLike|Record<string,unknown>[] Non-empty Array of Records to be inserted into the table -
namespacePath?:
string[] The namespace path to create the table in (defaults to root namespace) -
options?:
Partial<CreateTableOptions> Additional options
Returns
Promise<Table>
describeNamespace()
abstract describeNamespace(namespacePath): Promise<DescribeNamespaceResponse>
Describe a namespace, returning its properties.
Parameters
- namespacePath:
string[] The namespace path to describe, in parent → child order, e.g.["analytics", "sales"].
Returns
Promise<DescribeNamespaceResponse>
The namespace's properties (may be undefined if the namespace has none).
display()
abstract display(): string
Return a brief description of the connection
Returns
string
dropAllTables()
abstract dropAllTables(namespacePath?): Promise<void>
Drop all tables in the database.
Parameters
- namespacePath?:
string[] The namespace path to drop tables from (defaults to root namespace).
Returns
Promise<void>
dropNamespace()
abstract dropNamespace(namespacePath, options?): Promise<DropNamespaceResponse>
Drop a namespace.
Use behavior: "cascade" to also drop everything contained in the
namespace (sub-namespaces and tables). The default "restrict"
behavior refuses to drop a non-empty namespace.
Parameters
-
namespacePath:
string[] The namespace path to drop. -
options?:
Partial<DropNamespaceOptions>mode("skip" | "fail" for missing-namespace handling) andbehavior("restrict" | "cascade").
Returns
Promise<DropNamespaceResponse>
Any properties returned by the server and an optional transaction id.
dropTable()
abstract dropTable(name, namespacePath?): Promise<void>
Drop an existing table.
Parameters
-
name:
stringThe name of the table to drop. -
namespacePath?:
string[] The namespace path of the table (defaults to root namespace).
Returns
Promise<void>
dropTableAsync()
abstract dropTableAsync(name, namespacePath?): Promise<Job>
Start dropping a table and return its cleanup job.
The table may become unavailable before its data files are removed. Wait on the returned job to know when cleanup has finished.
Parameters
-
name:
string -
namespacePath?:
string[]
Returns
Promise<Job>
isOpen()
abstract isOpen(): boolean
Return true if the connection has not been closed
Returns
boolean
listJobs()
abstract listJobs(): Promise<JobInfo[]>
List server-side jobs across the database's tables.
Returns
Promise<JobInfo[]>
listMaterializedViews()
abstract listMaterializedViews(): Promise<string[]>
The names of the materialized views in this database.
Found by reading every table's schema, so this costs an open per table.
Returns
Promise<string[]>
listNamespaces()
abstract listNamespaces(namespacePath?, options?): Promise<ListNamespacesResponse>
List the immediate child namespaces under the given parent.
Results may be paginated. To retrieve subsequent pages, pass the
pageToken returned by a previous call.
Parameters
-
namespacePath?:
string[] The parent namespace path. Defaults to the root namespace if omitted. -
options?:
Partial<ListNamespacesOptions> Pagination options (pageToken,limit).
Returns
Promise<ListNamespacesResponse>
Child namespace names and an optional token for fetching the next page.
listTables()
listTables(options)
abstract listTables(options?): Promise<ListTablesResponse>
List a page of the tables in this database.
To retrieve the tables after the page, pass the pageToken the response
carries back in. A page can be shorter than limit without being the last
one, so walk until a response carries no page token:
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
Parameters
- options?:
Partial<ListTablesOptions> Pagination options (pageToken,limit).
Returns
Promise<ListTablesResponse>
A page of table names and an optional token for the tables after it.
listTables(namespacePath, options)
abstract listTables(namespacePath?, options?): Promise<ListTablesResponse>
List a page of the tables in this database.
Parameters
-
namespacePath?:
string[] The namespace path to list tables from (defaults to root namespace) -
options?:
Partial<ListTablesOptions> Pagination options (pageToken,limit).
Returns
Promise<ListTablesResponse>
A page of table names and an optional token for the tables after it.
openJob()
abstract openJob(jobId): Promise<Job>
Open a server-side job by id, returning a handle with its record already populated. Rejects when the server has no such job, the way Connection.openTable does for a missing table.
The returned Job answers for its own state, specification, result, failure and event history, so there is no separate connection-level call for any of them.
Parameters
- jobId:
string
Returns
Promise<Job>
openMaterializedView()
abstract openMaterializedView(name): Promise<MaterializedView>
Open the materialized view named name.
Rejects a table that exists but is not a materialized view.
Parameters
- name:
string
Returns
Promise<MaterializedView>
openTable()
abstract openTable(
name,
namespacePath?,
options?): Promise<Table>
Parameters
-
name:
string -
namespacePath?:
string[] -
options?:
Partial<OpenTableOptions>
Returns
Promise<Table>
renameTable()
abstract renameTable(
currentName,
newName,
options?): Promise<void>
Rename a table.
Currently only supported by LanceDB Cloud. Local OSS connections and namespace-backed connections (via connectNamespace) reject with a "not supported" error.
Parameters
-
currentName:
stringThe current name of the table. -
newName:
stringThe new name for the table. -
options?:
RenameTableOptionsOptional namespace paths. WhennewNamespacePathis omitted the table stays innamespacePath.
Returns
Promise<void>
tableNames()
tableNames(options)
abstract tableNames(options?): Promise<string[]>
List all the table names in this database.
Tables will be returned in lexicographical order.
Parameters
- options?:
Partial<TableNamesOptions> options to control the paging / start point (backwards compatibility)
Returns
Promise<string[]>
Deprecated
Use Connection.listTables instead.
tableNames(namespacePath, options)
abstract tableNames(namespacePath?, options?): Promise<string[]>
List all the table names in this database.
Tables will be returned in lexicographical order.
Parameters
-
namespacePath?:
string[] The namespace path to list tables from (defaults to root namespace) -
options?:
Partial<TableNamesOptions> options to control the paging / start point
Returns
Promise<string[]>
Deprecated
Use Connection.listTables instead.