fix(python): retry transient job polling errors

Fixes #4773.

Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately.

Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation.
This commit is contained in:
lancedb-gatefixer[bot]
2026-11-03 05:05:30 +08:00
committed by GitHub
parent 5ca8d836e7
commit ea9194a7f3
11 changed files with 474 additions and 11 deletions
+22 -2
View File
@@ -41,14 +41,21 @@ print(job.status())
results it committed before stopping.
In Python, `job.status()` reports the current state and `job.wait()` waits for
completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting.
The timeout does not cancel the server job.
completion, raising if the job fails or is cancelled. Both `Job.wait()` and
`AsyncJob.wait()` retry transient connection and read errors with capped
exponential backoff within the caller's timeout. Without a timeout, retries
continue indefinitely.
If the timeout expires while disconnected, the wait raises `ConnectionError`
with the job ID and instructions to resume waiting. A timeout while connected
raises `TimeoutError`. The job may still be running; timing out does not cancel it.
```python
from datetime import timedelta
from lancedb.exceptions import JobFailedError, JobCancelledError
job = tbl.create_index_async("vector", replace=True)
job_id = job.id # Save the ID to reopen an Enterprise registry job later.
try:
result = job.wait(timeout=timedelta(minutes=10))
@@ -56,10 +63,23 @@ except JobFailedError:
print(f"job failed: {job.failure}")
except JobCancelledError:
print("job was cancelled")
except ConnectionError as exc:
print(exc) # The job may still be running on the server.
except TimeoutError:
print(f"still running: {job.status()}")
```
After connectivity returns, reopen an Enterprise registry job by its saved ID
and continue waiting on the existing operation:
```python
job = db.open_job(job_id)
result = job.wait(timeout=timedelta(minutes=10))
```
For SQL-backed jobs, call `wait()` again on the original handle. These jobs
cannot be reopened with `db.open_job()`; see [SQL-backed job handles](#sql-backed-job-handles).
<span id="inspecting" />
<span id="events" />