mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
661 lines
24 KiB
Plaintext
661 lines
24 KiB
Plaintext
---
|
|
title: "Refresh a computed column"
|
|
sidebarTitle: "Refresh Computed Column"
|
|
description: "Populate a computed column and update it when its inputs change."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Refresh a [computed column](/feature-engineering/add-computed-column) to fill
|
|
new rows and recompute values whose inputs changed. The example below continues
|
|
with the `doubled` column from the computed-column guide.
|
|
|
|
<Note>
|
|
Computed columns require LanceDB Enterprise.
|
|
</Note>
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async("doubled")
|
|
job.wait()
|
|
|
|
table.search().to_arrow().sort_by("x").to_pandas()
|
|
# x doubled
|
|
# 0 1 2
|
|
# 1 2 4
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN doubled ON my_table
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
`table.refresh_column(...)` (sync) is also a local, blocking form, but it
|
|
raises on an Enterprise connection -- use `refresh_column_async`, shown
|
|
above, or the SQL form instead. See
|
|
[Run a refresh on Enterprise](#async) below for what the returned job
|
|
supports.
|
|
</Note>
|
|
|
|
The refresh commits ordinary table versions (the values, then a
|
|
metadata-only record of the inputs they were computed from), so
|
|
[versioning](/management/table#versioning) applies to it like any other write.
|
|
|
|
<Note>
|
|
The SQL form is [`REFRESH COLUMN`](/sql/dml/refresh-column), with the same resource options.
|
|
</Note>
|
|
|
|
## What a refresh recomputes {#idempotent}
|
|
|
|
Without a `WHERE` predicate, a refresh fills unassigned rows and recomputes
|
|
values whose inputs changed since they were computed. Every other row is left
|
|
alone, so a second refresh with nothing changed does nothing. This supports
|
|
two common workflows.
|
|
|
|
**Appending is cheap.** Rows added since the last refresh are filled by the
|
|
next one; filled rows whose inputs are unchanged are skipped. Backfilling a
|
|
column and then ingesting more data does not mean recomputing the backfill.
|
|
|
|
**A filled value catches up with its input at the next refresh.** If `doubled`
|
|
derives from `x` and a row's `x` is later updated, `doubled` keeps its old value
|
|
until the next refresh recomputes it.
|
|
|
|
That is what makes the call idempotent and safe to retry. However, unlike a
|
|
spreadsheet formula, nothing is recomputed until you refresh, so a derived
|
|
column is stale between an update and the next refresh.
|
|
|
|
## Automatic refresh {#automatic}
|
|
|
|
A computed column can refresh itself on a schedule instead of waiting for
|
|
an explicit call here. See [Auto Refresh](/enterprise/auto-refresh) for
|
|
enabling it, the SQL and SDK forms for changing the setting, and how
|
|
catch-up works when it's re-enabled after being off.
|
|
|
|
## Refreshing part of a table {#partial}
|
|
|
|
Python `where=` requires a supporting SDK, such as `0.42.0b9`, and an
|
|
Enterprise server with filtered refresh support. Stable `0.41.0` does not
|
|
accept this argument. The SQL form is available independently of that Python
|
|
argument on supporting servers.
|
|
|
|
<Badge color="red">Enterprise</Badge> A refresh can be scoped to a predicate,
|
|
with `where=` in Python or `WHERE` in SQL, which is useful for filling a slice
|
|
first and checking it before committing to the whole table:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
where="created_at > date '2026-01-01'",
|
|
)
|
|
job.wait()
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > date '2026-01-01'
|
|
```
|
|
</CodeGroup>
|
|
|
|
A filtered refresh recomputes **every matching live row**, including rows
|
|
whose outputs are already current. Repeating the same predicate repeats those
|
|
Function calls and their cost. To select only null outputs within a slice:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
where="created_at > date '2026-01-01' AND embedding IS NULL",
|
|
)
|
|
job.wait()
|
|
```
|
|
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents
|
|
WHERE created_at > date '2026-01-01' AND embedding IS NULL
|
|
```
|
|
</CodeGroup>
|
|
|
|
This skips non-null outputs even when their inputs changed, and can recompute
|
|
outputs that were successfully computed as `NULL`. Use an unfiltered refresh
|
|
to let LanceDB track which outputs need updating.
|
|
|
|
<Note>
|
|
**Enterprise job type:** `computed_refresh` for a SQL-expression column,
|
|
`function_refresh` for one filled by a registered [function](/management/function).
|
|
</Note>
|
|
|
|
### Re-score selected list elements {#list-elements}
|
|
|
|
`WHERE` selects **rows**, not elements within a list. Refreshing a row that
|
|
contains ten answers passes the whole list to the Function. A mask can control
|
|
which elements your code computes, but the Function still returns one complete
|
|
list for that row. There is no element-level refresh or partial-list write API.
|
|
|
|
This example keeps `sample_id` as the link to an existing `samples` table with
|
|
a Blob column named `media`. It creates only answer metadata and scores; it
|
|
neither reads nor copies Blob payloads. Run it on a deployment supporting
|
|
`@function`, list signatures and Python `where=`, with `samples` already
|
|
containing sample IDs `s1` and `s2`.
|
|
|
|
```python
|
|
from lancedb import col, function
|
|
|
|
media_before = db.execute_query(
|
|
"SELECT sample_id, media FROM samples "
|
|
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
|
|
).read_all().to_pylist()
|
|
|
|
answers = db.create_table("answer_rescore_demo", [
|
|
{"group_id": "g1", "sample_id": "s1",
|
|
"answers": ["yes", "maybe", "no"],
|
|
"selected": [False, True, False], "previous": [0.1, 0.2, 0.3]},
|
|
{"group_id": "g1", "sample_id": "s2",
|
|
"answers": ["other"], "selected": [False], "previous": [0.4]},
|
|
])
|
|
|
|
@function
|
|
def rescore_masked(
|
|
answers: list[str], selected: list[bool], previous: list[float]
|
|
) -> list[float]:
|
|
if not (len(answers) == len(selected) == len(previous)):
|
|
raise ValueError("answers, selected and previous must have equal lengths")
|
|
return [
|
|
float(len(answer)) if choose else old
|
|
for answer, choose, old in zip(answers, selected, previous)
|
|
]
|
|
|
|
version = db.create_function(rescore_masked)
|
|
answers.add_columns(
|
|
{"rescored": version(
|
|
answers=col("answers"), selected=col("selected"), previous=col("previous")
|
|
)},
|
|
auto_refresh_enabled=False,
|
|
)
|
|
job = answers.refresh_column_async("rescored", where="sample_id = 's1'")
|
|
job.wait()
|
|
|
|
rows = answers.search().select(
|
|
["group_id", "sample_id", "previous", "rescored"]
|
|
).to_list()
|
|
by_sample = {row["sample_id"]: row for row in rows}
|
|
assert by_sample["s1"]["group_id"] == "g1"
|
|
assert by_sample["s1"]["rescored"] == [0.1, 5.0, 0.3]
|
|
assert by_sample["s1"]["previous"] == [0.1, 0.2, 0.3]
|
|
assert by_sample["s2"]["rescored"] is None
|
|
assert by_sample["s2"]["previous"] == [0.4]
|
|
|
|
media_after = db.execute_query(
|
|
"SELECT sample_id, media FROM samples "
|
|
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
|
|
).read_all().to_pylist()
|
|
assert media_after == media_before
|
|
```
|
|
|
|
Here length is a deterministic stand-in for a scorer. Only `"maybe"` is scored;
|
|
the other scores are copied from `previous`. The final equality checks Blob
|
|
descriptors on an otherwise unchanged source table, without fetching payloads.
|
|
Keep the source table unchanged during this comparison. Use the same model,
|
|
normalization and score type as the original computation in a real re-scoring
|
|
pass. For another pass, explicitly supply the intended prior scores and mask;
|
|
refresh does not merge the previous output into your Function's inputs.
|
|
|
|
If answers need independent retries or versions, prefer one ordinary table row
|
|
per answer, with `group_id`, `sample_id`, a stable application `answer_id`,
|
|
answer text and score. Keep media in `samples` and join by `sample_id` when
|
|
needed. A row predicate can then select exact answer IDs, and Blob storage
|
|
stays in the source table. Do not use `UNNEST` to assume that nested Blob values
|
|
become supported Function inputs; see the
|
|
[nested-Blob limits](/curation/create-function#signature-types).
|
|
|
|
## Run a refresh on Enterprise {#async}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
With an Enterprise cluster that enables computed columns, the refresh runs on the
|
|
cluster. `refresh_column` raises
|
|
there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can
|
|
wait on or poll instead of holding a connection open:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async("doubled")
|
|
result = job.wait()
|
|
result.rows_assigned
|
|
# 2
|
|
job.status()
|
|
# 'finished'
|
|
```
|
|
</CodeGroup>
|
|
|
|
Poll with `status()`, resolve with `wait()`. `status()` is a point snapshot and
|
|
does not raise, which is what makes it safe in a loop; `wait()` raises
|
|
`JobFailedError`, `JobCancelledError` or `TimeoutError`.
|
|
|
|
A long backfill can therefore be started, left, and checked on from a different
|
|
process.
|
|
|
|
{/*
|
|
Draft for the Function freshness release. Keep this section hidden until the
|
|
Enterprise status endpoint and its integration acceptance are deployed.
|
|
|
|
### Check Function column freshness {#freshness}
|
|
|
|
For a Function-backed column, the SDK can report how many live rows are
|
|
`fresh`, `stale`, `never_computed`, or `failed` at the table version it
|
|
reads. `wait_fresh`, `waitFresh`, and `wait_for_function_column` poll until no
|
|
stale or never-computed rows remain. Rows that already failed stay in the
|
|
returned status and do not keep the wait pending, so inspect `failed` after a
|
|
successful wait:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
status = table.column_status("embedding")
|
|
print(status.fresh, status.stale, status.never_computed, status.failed)
|
|
|
|
final_status = table.wait_fresh("embedding")
|
|
print(final_status.failed)
|
|
```
|
|
|
|
```typescript TypeScript icon="typescript"
|
|
const status = await table.functionColumnStatus("embedding");
|
|
console.log(
|
|
status.fresh,
|
|
status.stale,
|
|
status.neverComputed,
|
|
status.failed,
|
|
);
|
|
|
|
const finalStatus = await table.waitFresh("embedding");
|
|
console.log(finalStatus.failed);
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
let status = table.function_column_status("embedding").await?;
|
|
println!(
|
|
"{} {} {} {}",
|
|
status.fresh, status.stale, status.never_computed, status.failed
|
|
);
|
|
|
|
let final_status = table.wait_for_function_column("embedding", None).await?;
|
|
println!("{}", final_status.failed);
|
|
```
|
|
</CodeGroup>
|
|
|
|
`auto_refresh.state` reports `queued` while the scheduler has accepted the
|
|
refresh but a worker has not started it; `running` means execution has started.
|
|
`last_refresh_job` describes the most recent attempt, including a cancellation
|
|
or a failure. If a later attempt was cancelled, `last_failure_job` retains the
|
|
most recent job that actually failed, so its nested `failure` continues to
|
|
explain the failed rows.
|
|
|
|
These fields require a matching Agent, Query Node, and SDK deployment. An older
|
|
client cannot decode the `queued` state, and an older server can omit the new
|
|
optional failure fields.
|
|
|
|
The wait helpers do not submit or cancel a refresh. They retry only the typed
|
|
finalization-pending response and propagate other errors.
|
|
*/}
|
|
|
|
### Admission and waiting {#admission}
|
|
|
|
Submission does not reserve a supervisor or GPU immediately. A refresh can wait
|
|
for source preparation/compaction, an earlier refresh, executor admission or
|
|
worker capacity. A lifecycle `running` value from `status()` alone does not
|
|
prove that Python code has begun executing.
|
|
|
|
Use [`SHOW JOBS`](/sql/ddl/job#show) and the Console's job detail to inspect
|
|
available phase and row progress. On SDK/server versions with `job.progress()`,
|
|
the snapshot can also include `phase`, `queue_reason`, timestamps and row
|
|
counters. These optional fields are version-dependent; missing values are
|
|
unknown, not zero progress or an ETA. `creation_ms` is submission time, not
|
|
worker start time. `DESCRIBE JOB` provides the finished run report, rather than
|
|
a live queue-position API.
|
|
|
|
Limits are deployment settings: the Agent dispatcher concurrency, executor
|
|
capacity, per-table automatic-refresh limit and Function-call
|
|
`max_concurrency` control different things. There is no universal five-refresh
|
|
limit per database. Ask your administrator for the effective settings and use
|
|
the [Enterprise operations guide](/enterprise/function-operations#admission)
|
|
for their configuration names. Repeated cancellation and resubmission does not
|
|
bypass those limits. Cancellation requests a stop, including while queued;
|
|
observe a terminal state before submitting replacement work. Committed work
|
|
is not rolled back.
|
|
|
|
### Configure workers and resources {#resources}
|
|
|
|
SQL exposes the worker budget directly, which is the lever when a backfill is
|
|
too slow or too expensive:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
</CodeGroup>
|
|
|
|
The resource values cap the Function workers together: this example allows
|
|
up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not
|
|
per-worker allocations or a guaranteed worker count. `num_workers` is only
|
|
accepted for SQL-expression columns; registered Function columns reject it.
|
|
|
|
In particular, `gpu_count = 2` does not request two devices inside one Python
|
|
process or enable tensor parallelism. Inspect configured GPU classes and their
|
|
per-worker limits with `db.list_gpu_classes()` or `SHOW GPU CLASSES`. Choose a
|
|
model that fits the worker shape supported by your deployment; see
|
|
[GPU placement and budgets](/enterprise/function-operations#gpu).
|
|
|
|
The SDKs accept the same nested budget. CPU and memory are required; the GPU
|
|
count inside the budget defaults to zero, so a CPU-only Function can omit it:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
resource_budget={
|
|
"cpu_millis": 8000,
|
|
"memory_bytes": 16 * 1024**3,
|
|
},
|
|
)
|
|
job.wait()
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use lancedb::table::{
|
|
FunctionWorkerResourceBudget,
|
|
RefreshColumnOptions,
|
|
};
|
|
|
|
let _job = table
|
|
.refresh_column_async_with_options(
|
|
"embedding",
|
|
RefreshColumnOptions::default().resource_budget(
|
|
FunctionWorkerResourceBudget::new(8000, 17_179_869_184),
|
|
),
|
|
)
|
|
.await?;
|
|
```
|
|
|
|
```typescript TypeScript icon="typescript"
|
|
const job = await table.refreshColumnAsync("embedding", {
|
|
resourceBudget: {
|
|
cpuMillis: 8000,
|
|
memoryBytes: 16 * 1024 ** 3,
|
|
},
|
|
});
|
|
await job.wait();
|
|
```
|
|
</CodeGroup>
|
|
|
|
On current SQL servers, the equivalent `REFRESH COLUMN ... WITH (...)` form
|
|
still takes all three fields when CPU or memory is set; use `gpu_count = 0`
|
|
for a CPU-only Function. The SDK normalizes an omitted nested GPU count to a
|
|
numeric zero before sending the request.
|
|
|
|
The existing top-level `gpu_count` / `gpuCount` option remains the shorthand
|
|
for changing only the aggregate GPU budget. It cannot be combined with the
|
|
complete resource budget. Omitting both preserves the server's existing
|
|
defaults.
|
|
|
|
These examples configure only the explicit refresh shown. A registered
|
|
Function declaration can separately set `auto_refresh_resource_budget` on
|
|
`add_columns`; that declaration controls the budget for its automatic
|
|
refreshes. SQL-expression columns reject the automatic resource-budget option
|
|
because they do not use Function workers.
|
|
|
|
To override the registered Function's call concurrency for this logical
|
|
execution only:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async("embedding", max_concurrency=4)
|
|
job.wait()
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use lancedb::table::RefreshColumnOptions;
|
|
|
|
let _job = table
|
|
.refresh_column_async_with_options(
|
|
"embedding",
|
|
RefreshColumnOptions::default().max_concurrency(4),
|
|
)
|
|
.await?;
|
|
```
|
|
|
|
```typescript TypeScript icon="typescript"
|
|
const job = await table.refreshColumnAsync("embedding", {
|
|
maxConcurrency: 4,
|
|
});
|
|
await job.wait();
|
|
```
|
|
</CodeGroup>
|
|
|
|
`max_concurrency` caps user-code calls in flight for one Function within one
|
|
logical execution. It is not an account-wide quota and does not resize or
|
|
rewrite `resource_budget`. Omitting it uses the registration default; a
|
|
refresh value overrides that default for this execution only.
|
|
|
|
`batch_size` caps rows in one Function call; LanceDB adapts actual call and
|
|
work sizes. For Function columns, the refresh defaults to
|
|
`error_policy = 'fail'`. Set `error_policy = 'skip'` to write null outputs for
|
|
failing rows and record them in the error store, subject to failure limits.
|
|
See the [SQL refresh options](/sql/dml/refresh-column#options).
|
|
|
|
{/*
|
|
Draft for the Function retry policy release.
|
|
|
|
For registered Function columns, pass `retry_policy` to
|
|
`refresh_column_async` to replace the Function's registered retry policy
|
|
for this refresh. For example,
|
|
`retry_policy=FunctionRetryPolicy(transient=True, max_attempts=3)` allows
|
|
three attempts, including the first call. Import `FunctionRetryPolicy`
|
|
from `lancedb.functions`. Use `FunctionRetryPolicy(max_attempts=1)` to
|
|
disable user-error retries. Omitting the override preserves the registered
|
|
policy.
|
|
|
|
This requires a supporting LanceDB Enterprise server as well as an SDK
|
|
that exposes these options. An older Function image may need
|
|
re-registration before it accepts a refresh override. Only enable retries
|
|
when repeating a failed call's partial work is safe, including any external
|
|
requests made before the error.
|
|
|
|
The attempt budget belongs to one failed Function call. With an explicit
|
|
retry policy and `error_policy="skip"`, exhaustion in a batch Function
|
|
writes nulls for the whole failed call and records its errors, subject to
|
|
failure limits. It does not split the batch to restart the retry budget.
|
|
Row Functions isolate failures per row. Infrastructure recovery, framework
|
|
deadlines and cancellation remain separate from this user-error policy.
|
|
See [Function failure handling](/management/function#failures) for the
|
|
retry and replay boundaries.
|
|
*/}
|
|
|
|
For a column filled by a registered Function, the Python and Rust job APIs can
|
|
override the Function's GPU placement for this refresh:
|
|
|
|
<CodeGroup>
|
|
```python
|
|
classes = db.list_gpu_classes()
|
|
gpu_class = next(gpu.name for gpu in classes if gpu.gpu_model == "NVIDIA H100")
|
|
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
gpu_class=gpu_class,
|
|
gpu_count=4,
|
|
batch_size=32,
|
|
)
|
|
```
|
|
|
|
```rust
|
|
use lancedb::{Connection, Table, table::RefreshColumnOptions};
|
|
|
|
async fn refresh(
|
|
conn: &Connection,
|
|
table: &Table,
|
|
) -> Result<(), Box<dyn std::error::Error>> {
|
|
let gpu_class = conn
|
|
.list_gpu_classes()
|
|
.await?
|
|
.into_iter()
|
|
.find(|class| class.gpu_model.as_deref() == Some("NVIDIA H100"))
|
|
.map(|class| class.name)
|
|
.ok_or("H100 GPU class is not configured")?;
|
|
let _job = table
|
|
.refresh_column_async_with_options(
|
|
"embedding",
|
|
RefreshColumnOptions::default()
|
|
.gpu_class(gpu_class)
|
|
.gpu_count(4)
|
|
.batch_size(32),
|
|
)
|
|
.await?;
|
|
Ok(())
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
`gpu_count` is the aggregate GPU budget for the refresh's Function workers,
|
|
not a request for that many GPUs per worker. It must be positive and preserves
|
|
the deployment's CPU and memory defaults.
|
|
`gpu_class` must name a class returned by `list_gpu_classes()`; class names
|
|
are lowercase DNS labels. Omit either option to use the Function's registered
|
|
class or the server's configured budget. `list_gpu_classes()` requires System
|
|
Usage permission, and raises `NotImplementedError` when the server predates
|
|
GPU class discovery.
|
|
|
|
<Note>
|
|
The Rust GPU discovery and refresh controls are not in the latest Rust
|
|
release. TypeScript does not expose GPU class discovery; pass `gpuClass`
|
|
and `gpuCount` in the `refreshColumnAsync` options.
|
|
</Note>
|
|
|
|
Completed Function outputs publish with their fragment by default. To opt into
|
|
a row-count publication threshold, set `publication_interval_rows` in Python,
|
|
`publicationIntervalRows` in TypeScript, or `publication_interval_rows()` in Rust:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
publication_interval_rows=60,
|
|
)
|
|
job.wait()
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const job = await table.refreshColumnAsync("embedding", {
|
|
publicationIntervalRows: 60,
|
|
});
|
|
await job.wait();
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
use lancedb::table::RefreshColumnOptions;
|
|
|
|
let _job = table
|
|
.refresh_column_async_with_options(
|
|
"embedding",
|
|
RefreshColumnOptions::default().publication_interval_rows(60),
|
|
)
|
|
.await?;
|
|
```
|
|
</CodeGroup>
|
|
|
|
The threshold is opt-in and must be greater than zero. The number of rows in an
|
|
individual publication can differ from this value depending on completed work
|
|
and Function batching; this is not an exact-N contract. Omit it to keep
|
|
whole-fragment publication.
|
|
|
|
In TypeScript, the threshold must be a positive safe integer, at most
|
|
`Number.MAX_SAFE_INTEGER`. This option requires a supporting LanceDB Enterprise
|
|
server and Function outputs without Blob types; Blob inputs are allowed.
|
|
|
|
The job APIs also take the row filter and failure handling of the SQL
|
|
`REFRESH COLUMN ... WHERE ... WITH (...)` form, so a refresh can be narrowed
|
|
to new rows (see [what a filter recomputes](#partial)) and told to skip rows
|
|
whose Function call fails:
|
|
|
|
<CodeGroup>
|
|
```python
|
|
job = table.refresh_column_async(
|
|
"embedding",
|
|
where="created_at > date '2026-01-01'",
|
|
error_policy="skip",
|
|
max_failed_fraction=0.01,
|
|
row_timeout_seconds=30,
|
|
)
|
|
```
|
|
|
|
```typescript
|
|
const job = await table.refreshColumnAsync("embedding", {
|
|
where: "created_at > date '2026-01-01'",
|
|
errorPolicy: "skip",
|
|
maxFailedFraction: 0.01,
|
|
rowTimeoutSeconds: 30,
|
|
});
|
|
```
|
|
|
|
```rust
|
|
use lancedb::{function::FunctionErrorPolicy, table::RefreshColumnOptions};
|
|
|
|
let _job = table
|
|
.refresh_column_async_with_options(
|
|
"embedding",
|
|
RefreshColumnOptions::default()
|
|
.where_clause("created_at > date '2026-01-01'")
|
|
.error_policy(FunctionErrorPolicy::Skip)
|
|
.max_failed_fraction(0.01)
|
|
.row_timeout_seconds(30.0),
|
|
)
|
|
.await?;
|
|
```
|
|
</CodeGroup>
|
|
|
|
`max_failed_rows`, `init_timeout_seconds`, `num_workers` and
|
|
`max_executor_processes_per_worker` are accepted the same way; the
|
|
[SQL refresh options](/sql/dml/refresh-column#options) describe each one.
|
|
The SDK rejects a blank `where`, a `max_failed_fraction` outside 0 to 1, a
|
|
timeout that is not a positive number and a zero worker or process count
|
|
before sending the request; the server validates everything else. A LanceDB
|
|
Embedded refresh rejects every option other than `batch_size`, `gpu_class`
|
|
and `gpu_count`; in particular, it rejects `resource_budget` rather than
|
|
silently ignoring it.
|
|
|
|
## When a refresh fails {#failures}
|
|
|
|
What a failure keeps depends on where the refresh ran. The local, blocking
|
|
`refresh_column` writes the column in one commit, so a failure leaves the
|
|
column as it was. A refresh job in LanceDB Enterprise -- always
|
|
`refresh_column_async` or `REFRESH COLUMN` there, never the blocking form --
|
|
commits finished batches as it goes, so some values can remain after a
|
|
failure.
|
|
|
|
Either way, recover by fixing the cause and refreshing again. The new refresh
|
|
fills rows that are still empty or whose inputs changed, and normally skips
|
|
values that were kept. That is not guaranteed: a refresh can recompute values
|
|
that were already written (for example when the record of the inputs they
|
|
were computed from was not saved), so do not count on a retry costing only the
|
|
rows that did not complete.
|
|
|
|
For a column filled by a [function](/management/function), the usual cause is
|
|
the worker environment rather than the data. See
|
|
[environment mismatches](/management/function#mismatch-symptoms) for the
|
|
symptoms, which rarely name their cause.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Job Framework" icon="hourglass-half" href="/basics/jobs">
|
|
Waiting on, inspecting, and cancelling server-side work.
|
|
</Card>
|
|
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/refresh-materialized-view">
|
|
The same idea, for a whole derived table.
|
|
</Card>
|
|
<Card title="Auto Refresh" icon="clock-rotate-left" href="/enterprise/auto-refresh">
|
|
Refresh on a schedule instead of calling this yourself.
|
|
</Card>
|
|
</CardGroup>
|