mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
152 lines
6.7 KiB
Plaintext
152 lines
6.7 KiB
Plaintext
---
|
|
title: "Handle Function failures"
|
|
sidebarTitle: "Function Failures"
|
|
description: "Keep usable outputs, inspect row errors, retry by stable identity and isolate inputs that need attention."
|
|
tag: "Early access"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A Function refresh can encounter a bad row, fail to initialize a model, lose
|
|
a worker, or fail while publishing results. These need different remedies.
|
|
This guide uses the existing [refresh error policy](/sql/dml/refresh-column#options)
|
|
and Python error records; it does not require a new retry or quarantine API.
|
|
|
|
## Choose a row-error policy {#policy}
|
|
|
|
The default `fail` stops the job on a row error. `skip` writes null outputs for
|
|
failed rows and records errors, subject to the refresh's failure limits:
|
|
|
|
```python
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
error_policy="skip",
|
|
max_failed_rows=100,
|
|
max_failed_fraction=0.1,
|
|
)
|
|
result = job.wait()
|
|
print(result.rows_assigned, result.rows_failed, result.rows_remaining)
|
|
```
|
|
|
|
Choose limits for your workload; these example limits stop work when more
|
|
than 100 rows or more than 10% fail. `skip` does not make initialization,
|
|
infrastructure or publication failures successful. If `wait()` raises,
|
|
[refresh the job record](/basics/jobs#snapshots), inspect `job.failure`, and
|
|
inspect any recorded row errors. A failed/cancelled job can have committed
|
|
some output, but retention of every successful call is not guaranteed.
|
|
External effects are not rolled back and can be repeated during recovery.
|
|
|
|
## Inspect and group recorded errors {#inspect}
|
|
|
|
Scope diagnostics to the job you are investigating:
|
|
|
|
```python
|
|
from collections import Counter
|
|
|
|
errors = table.function_errors(job_id=job.id, column="embedding", limit=10000)
|
|
groups = Counter((r.error_type, r.error_message) for r in errors.records)
|
|
for cause, count in groups.most_common():
|
|
print(count, cause)
|
|
for fragment in errors.fragments:
|
|
print(fragment.fragment_id, fragment.rows_skipped, fragment.rows_recorded)
|
|
print("listing truncated:", errors.truncated)
|
|
```
|
|
|
|
Each record includes the Function name/version, table version, error type and
|
|
message, creation time, and available row identities. `attempts` and traceback
|
|
details depend on the server/image version. This client-side grouping counts
|
|
only returned detail records. `truncated=True` or fragment summaries with
|
|
more skipped than recorded rows mean it is incomplete; do not label it a full
|
|
error census. See [`SHOW ERRORS`](/sql/ddl/job#errors) for SQL and its limits.
|
|
|
|
Records are historical observations, not a deduplicated list of currently
|
|
failed rows. Reading errors without `job_id` can include older attempts that
|
|
have since succeeded. A null output can mean unfilled, skipped, or a computed
|
|
nullable field; output nullness alone is not a current-error test.
|
|
|
|
## Retry exactly identified rows {#retry}
|
|
|
|
On tables with stable row IDs, supporting server/SDK versions return
|
|
`record.row_id` from the refresh's pinned source version. Keep that table and
|
|
branch identity with the records. Compaction can move physical addresses, so
|
|
do not derive a row ID from fragment numbers or offsets.
|
|
|
|
The following deliberately refuses incomplete or unidentified error sets.
|
|
Review historical records against the current inputs before using them as a
|
|
retry selection. Keep this selection small enough for a SQL predicate.
|
|
|
|
```python
|
|
if errors.truncated or any(f.rows_skipped > f.rows_recorded for f in errors.fragments):
|
|
raise RuntimeError("Inspect a narrower job/error set before selecting retries")
|
|
if any(r.row_id is None for r in errors.records):
|
|
raise RuntimeError("This workflow requires stable row IDs in every error record")
|
|
|
|
row_ids = sorted({int(r.row_id) for r in errors.records})
|
|
if row_ids:
|
|
predicate = "_rowid IN (" + ", ".join(map(str, row_ids)) + ")"
|
|
retry_job = table.refresh_column_async("embedding", where=predicate)
|
|
retry_job.wait()
|
|
```
|
|
|
|
This recomputes every selected live row with the existing binding. It can
|
|
repeat a row already repaired since the error was recorded. Inspect the retry
|
|
job and its errors afterward; an older record need not disappear. On tables
|
|
without stable row IDs, resolve errors to application keys at the recorded
|
|
snapshot before retrying. Do not substitute `_rowaddr` after a rewrite.
|
|
Python `where=` requires a supporting SDK such as `0.42.0b9`, not stable
|
|
`0.41.0`, plus a matching server.
|
|
|
|
## Repair with a newer Function {#new-version}
|
|
|
|
Registering a new version does not change an existing column's immutable
|
|
binding. To evaluate a fix while retaining old outputs, register the corrected
|
|
callable and declare a separate output column. For example, with a corrected
|
|
`embed_fixed` callable already defined and a `text` input column:
|
|
|
|
```python
|
|
from lancedb import col
|
|
|
|
fixed = db.create_function(embed_fixed)
|
|
table.add_columns(
|
|
{"embedding_repaired": fixed(text=col("text"))},
|
|
auto_refresh_enabled=False,
|
|
)
|
|
if row_ids:
|
|
repair_job = table.refresh_column_async("embedding_repaired", where=predicate)
|
|
repair_job.wait()
|
|
```
|
|
|
|
Only the reviewed failed-row selection is populated in the new column; other
|
|
rows remain unfilled. Compare those outputs before deciding how readers
|
|
should use them. This does not promise an in-place upgrade/retry operation or
|
|
per-row producing-version lookup for a column containing mixed executions.
|
|
Error records' Function versions describe failures, not the provenance of
|
|
every successful row.
|
|
|
|
## Quarantine inputs for review {#quarantine}
|
|
|
|
Use an application-owned table of source table/branch, recorded table version,
|
|
stable row ID or business key, Function version, cause and review disposition.
|
|
Keep Blob payloads in the source table; a quarantine entry can refer to their
|
|
source identity. Choose an ordinary table name such as `function_quarantine`,
|
|
and explicitly exclude its reviewed IDs in subsequent input selections.
|
|
|
|
Quarantine is an application workflow here: there is no implied built-in
|
|
quarantine lifecycle, automatic current-error clearing or exactly-once retry.
|
|
Do not delete source rows merely to hide an error. For very large failure sets,
|
|
bounded error listings cannot select every row; use a smaller reviewed scope
|
|
or wait for a deployment with the required complete selection support.
|
|
|
|
Native retry policies, full current-error summaries, in-place version upgrades
|
|
and materialized-view row-error handling have separate server/SDK prerequisites.
|
|
Do not apply `REFRESH COLUMN` options to a materialized view and assume the
|
|
same policy; consult the [MV refresh reference](/sql/dml/refresh-materialized-view)
|
|
for your deployment's accepted options.
|
|
|
|
## Continue {#next}
|
|
|
|
- [Function lifecycle and failures](/management/function#failures)
|
|
- [Conditional and list-element refresh](/feature-engineering/refresh-column#partial)
|
|
- [Job results and events](/basics/jobs#snapshots)
|