---
title: "Refresh a computed column"
sidebarTitle: "Refresh Computed Column"
description: "Populate a computed column and update it when its inputs change."
icon: arrows-rotate
keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"]
availability:
embedded: unavailable
enterprise: available
---
Refresh a [computed column](/feature-engineering/add-computed-column) to fill
new rows and recompute values whose inputs changed. The example below continues
with the `doubled` column from the computed-column guide.
Computed columns require LanceDB Enterprise.
```python Python icon="python"
job = table.refresh_column_async("doubled")
job.wait()
table.search().to_arrow().sort_by("x").to_pandas()
# x doubled
# 0 1 2
# 1 2 4
```
```sql SQL icon="database"
REFRESH COLUMN doubled ON my_table
```
`table.refresh_column(...)` (sync) is also a local, blocking form, but it
raises on an Enterprise connection -- use `refresh_column_async`, shown
above, or the SQL form instead. See
[Run a refresh on Enterprise](#async) below for what the returned job
supports.
The refresh commits ordinary table versions (the values, then a
metadata-only record of the inputs they were computed from), so
[versioning](/management/table#versioning) applies to it like any other write.
The SQL form is [`REFRESH COLUMN`](/sql/dml/refresh-column), with the same resource options.
## What a refresh recomputes {#idempotent}
Without a `WHERE` predicate, a refresh fills unassigned rows and recomputes
values whose inputs changed since they were computed. Every other row is left
alone, so a second refresh with nothing changed does nothing. This supports
two common workflows.
**Appending is cheap.** Rows added since the last refresh are filled by the
next one; filled rows whose inputs are unchanged are skipped. Backfilling a
column and then ingesting more data does not mean recomputing the backfill.
**A filled value catches up with its input at the next refresh.** If `doubled`
derives from `x` and a row's `x` is later updated, `doubled` keeps its old value
until the next refresh recomputes it.
That is what makes the call idempotent and safe to retry. However, unlike a
spreadsheet formula, nothing is recomputed until you refresh, so a derived
column is stale between an update and the next refresh.
## Automatic refresh {#automatic}
A computed column can refresh itself on a schedule instead of waiting for
an explicit call here. See [Auto Refresh](/enterprise/auto-refresh) for
enabling it, the SQL and SDK forms for changing the setting, and how
catch-up works when it's re-enabled after being off.
## Refreshing part of a table {#partial}
Python `where=` requires a supporting SDK, such as `0.42.0b9`, and an
Enterprise server with filtered refresh support. Stable `0.41.0` does not
accept this argument. The SQL form is available independently of that Python
argument on supporting servers.
Enterprise A refresh can be scoped to a predicate,
with `where=` in Python or `WHERE` in SQL, which is useful for filling a slice
first and checking it before committing to the whole table:
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
where="created_at > date '2026-01-01'",
)
job.wait()
```
```sql SQL icon="database"
REFRESH COLUMN embedding ON documents WHERE created_at > date '2026-01-01'
```
A filtered refresh recomputes **every matching live row**, including rows
whose outputs are already current. Repeating the same predicate repeats those
Function calls and their cost. To select only null outputs within a slice:
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
where="created_at > date '2026-01-01' AND embedding IS NULL",
)
job.wait()
```
```sql SQL icon="database"
REFRESH COLUMN embedding ON documents
WHERE created_at > date '2026-01-01' AND embedding IS NULL
```
This skips non-null outputs even when their inputs changed, and can recompute
outputs that were successfully computed as `NULL`. Use an unfiltered refresh
to let LanceDB track which outputs need updating.
**Enterprise job type:** `computed_refresh` for a SQL-expression column,
`function_refresh` for one filled by a registered [function](/management/function).
### Re-score selected list elements {#list-elements}
`WHERE` selects **rows**, not elements within a list. Refreshing a row that
contains ten answers passes the whole list to the Function. A mask can control
which elements your code computes, but the Function still returns one complete
list for that row. There is no element-level refresh or partial-list write API.
This example keeps `sample_id` as the link to an existing `samples` table with
a Blob column named `media`. It creates only answer metadata and scores; it
neither reads nor copies Blob payloads. Run it on a deployment supporting
`@function`, list signatures and Python `where=`, with `samples` already
containing sample IDs `s1` and `s2`.
```python
from lancedb import col, function
media_before = db.execute_query(
"SELECT sample_id, media FROM samples "
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
).read_all().to_pylist()
answers = db.create_table("answer_rescore_demo", [
{"group_id": "g1", "sample_id": "s1",
"answers": ["yes", "maybe", "no"],
"selected": [False, True, False], "previous": [0.1, 0.2, 0.3]},
{"group_id": "g1", "sample_id": "s2",
"answers": ["other"], "selected": [False], "previous": [0.4]},
])
@function
def rescore_masked(
answers: list[str], selected: list[bool], previous: list[float]
) -> list[float]:
if not (len(answers) == len(selected) == len(previous)):
raise ValueError("answers, selected and previous must have equal lengths")
return [
float(len(answer)) if choose else old
for answer, choose, old in zip(answers, selected, previous)
]
version = db.create_function(rescore_masked)
answers.add_columns(
{"rescored": version(
answers=col("answers"), selected=col("selected"), previous=col("previous")
)},
auto_refresh_enabled=False,
)
job = answers.refresh_column_async("rescored", where="sample_id = 's1'")
job.wait()
rows = answers.search().select(
["group_id", "sample_id", "previous", "rescored"]
).to_list()
by_sample = {row["sample_id"]: row for row in rows}
assert by_sample["s1"]["group_id"] == "g1"
assert by_sample["s1"]["rescored"] == [0.1, 5.0, 0.3]
assert by_sample["s1"]["previous"] == [0.1, 0.2, 0.3]
assert by_sample["s2"]["rescored"] is None
assert by_sample["s2"]["previous"] == [0.4]
media_after = db.execute_query(
"SELECT sample_id, media FROM samples "
"WHERE sample_id IN ('s1', 's2') ORDER BY sample_id"
).read_all().to_pylist()
assert media_after == media_before
```
Here length is a deterministic stand-in for a scorer. Only `"maybe"` is scored;
the other scores are copied from `previous`. The final equality checks Blob
descriptors on an otherwise unchanged source table, without fetching payloads.
Keep the source table unchanged during this comparison. Use the same model,
normalization and score type as the original computation in a real re-scoring
pass. For another pass, explicitly supply the intended prior scores and mask;
refresh does not merge the previous output into your Function's inputs.
If answers need independent retries or versions, prefer one ordinary table row
per answer, with `group_id`, `sample_id`, a stable application `answer_id`,
answer text and score. Keep media in `samples` and join by `sample_id` when
needed. A row predicate can then select exact answer IDs, and Blob storage
stays in the source table. Do not use `UNNEST` to assume that nested Blob values
become supported Function inputs; see the
[nested-Blob limits](/curation/create-function#signature-types).
## Run a refresh on Enterprise {#async}
Enterprise
With an Enterprise cluster that enables computed columns, the refresh runs on the
cluster. `refresh_column` raises
there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can
wait on or poll instead of holding a connection open:
```python Python icon="python"
job = table.refresh_column_async("doubled")
result = job.wait()
result.rows_assigned
# 2
job.status()
# 'finished'
```
Poll with `status()`, resolve with `wait()`. `status()` is a point snapshot and
does not raise, which is what makes it safe in a loop; `wait()` raises
`JobFailedError`, `JobCancelledError` or `TimeoutError`.
A long backfill can therefore be started, left, and checked on from a different
process.
{/*
Draft for the Function freshness release. Keep this section hidden until the
Enterprise status endpoint and its integration acceptance are deployed.
### Check Function column freshness {#freshness}
For a Function-backed column, the SDK can report how many live rows are
`fresh`, `stale`, `never_computed`, or `failed` at the table version it
reads. `wait_fresh`, `waitFresh`, and `wait_for_function_column` poll until no
stale or never-computed rows remain. Rows that already failed stay in the
returned status and do not keep the wait pending, so inspect `failed` after a
successful wait:
```python Python icon="python"
status = table.column_status("embedding")
print(status.fresh, status.stale, status.never_computed, status.failed)
final_status = table.wait_fresh("embedding")
print(final_status.failed)
```
```typescript TypeScript icon="typescript"
const status = await table.functionColumnStatus("embedding");
console.log(
status.fresh,
status.stale,
status.neverComputed,
status.failed,
);
const finalStatus = await table.waitFresh("embedding");
console.log(finalStatus.failed);
```
```rust Rust icon="rust"
let status = table.function_column_status("embedding").await?;
println!(
"{} {} {} {}",
status.fresh, status.stale, status.never_computed, status.failed
);
let final_status = table.wait_for_function_column("embedding", None).await?;
println!("{}", final_status.failed);
```
`auto_refresh.state` reports `queued` while the scheduler has accepted the
refresh but a worker has not started it; `running` means execution has started.
`last_refresh_job` describes the most recent attempt, including a cancellation
or a failure. If a later attempt was cancelled, `last_failure_job` retains the
most recent job that actually failed, so its nested `failure` continues to
explain the failed rows.
These fields require a matching Agent, Query Node, and SDK deployment. An older
client cannot decode the `queued` state, and an older server can omit the new
optional failure fields.
The wait helpers do not submit or cancel a refresh. They retry only the typed
finalization-pending response and propagate other errors.
*/}
### Admission and waiting {#admission}
Submission does not reserve a supervisor or GPU immediately. A refresh can wait
for source preparation/compaction, an earlier refresh, executor admission or
worker capacity. A lifecycle `running` value from `status()` alone does not
prove that Python code has begun executing.
Use [`SHOW JOBS`](/sql/ddl/job#show) and the Console's job detail to inspect
available phase and row progress. On SDK/server versions with `job.progress()`,
the snapshot can also include `phase`, `queue_reason`, timestamps and row
counters. These optional fields are version-dependent; missing values are
unknown, not zero progress or an ETA. `creation_ms` is submission time, not
worker start time. `DESCRIBE JOB` provides the finished run report, rather than
a live queue-position API.
Limits are deployment settings: the Agent dispatcher concurrency, executor
capacity, per-table automatic-refresh limit and Function-call
`max_concurrency` control different things. There is no universal five-refresh
limit per database. Ask your administrator for the effective settings and use
the [Enterprise operations guide](/enterprise/function-operations#admission)
for their configuration names. Repeated cancellation and resubmission does not
bypass those limits. Cancellation requests a stop, including while queued;
observe a terminal state before submitting replacement work. Committed work
is not rolled back.
### Configure workers and resources {#resources}
SQL exposes the worker budget directly, which is the lever when a backfill is
too slow or too expensive:
```sql SQL icon="database"
REFRESH COLUMN embedding ON documents
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
```
The resource values cap the Function workers together: this example allows
up to 8 CPUs, 16 GiB of memory, and 2 GPUs across the fleet. They are not
per-worker allocations or a guaranteed worker count. `num_workers` is only
accepted for SQL-expression columns; registered Function columns reject it.
In particular, `gpu_count = 2` does not request two devices inside one Python
process or enable tensor parallelism. Inspect configured GPU classes and their
per-worker limits with `db.list_gpu_classes()` or `SHOW GPU CLASSES`. Choose a
model that fits the worker shape supported by your deployment; see
[GPU placement and budgets](/enterprise/function-operations#gpu).
The SDKs accept the same nested budget. CPU and memory are required; the GPU
count inside the budget defaults to zero, so a CPU-only Function can omit it:
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
resource_budget={
"cpu_millis": 8000,
"memory_bytes": 16 * 1024**3,
},
)
job.wait()
```
```rust Rust icon="rust"
use lancedb::table::{
FunctionWorkerResourceBudget,
RefreshColumnOptions,
};
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default().resource_budget(
FunctionWorkerResourceBudget::new(8000, 17_179_869_184),
),
)
.await?;
```
```typescript TypeScript icon="typescript"
const job = await table.refreshColumnAsync("embedding", {
resourceBudget: {
cpuMillis: 8000,
memoryBytes: 16 * 1024 ** 3,
},
});
await job.wait();
```
On current SQL servers, the equivalent `REFRESH COLUMN ... WITH (...)` form
still takes all three fields when CPU or memory is set; use `gpu_count = 0`
for a CPU-only Function. The SDK normalizes an omitted nested GPU count to a
numeric zero before sending the request.
The existing top-level `gpu_count` / `gpuCount` option remains the shorthand
for changing only the aggregate GPU budget. It cannot be combined with the
complete resource budget. Omitting both preserves the server's existing
defaults.
{/*
Draft for the Function refresh control-plane resource release.
`resource_budget` above sizes the Function workers. Two independent options
size the refresh control plane for a registered Function.
`supervisor_resources` sets the CPU, memory and ephemeral-storage limits of
the supervisor pod. `planning_workers` sets the resources of one planning
worker and the sizing of the CPU-only planning fleet: `max_workers`,
`rows_per_worker` and `fragments_per_worker`. Every field is required and
must be positive. Omitting either option leaves the server's configured
defaults unchanged, and either can be combined with `resource_budget`.
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
supervisor_resources={
"cpu_millis": 2_000,
"memory_bytes": 4 * 1024**3,
"ephemeral_storage_bytes": 10 * 1024**3,
},
planning_workers={
"resources": {
"cpu_millis": 1_000,
"memory_bytes": 2 * 1024**3,
"ephemeral_storage_bytes": 5 * 1024**3,
},
"max_workers": 4,
"rows_per_worker": 100_000,
"fragments_per_worker": 8,
},
)
job.wait()
```
```rust Rust icon="rust"
use lancedb::table::{
FunctionRefreshPlanningConfig,
FunctionRefreshPodResources,
RefreshColumnOptions,
};
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default()
.supervisor_resources(
FunctionRefreshPodResources::new(
2_000,
4 * 1024 * 1024 * 1024,
10 * 1024 * 1024 * 1024,
),
)
.planning_workers(
FunctionRefreshPlanningConfig::new(
FunctionRefreshPodResources::new(
1_000,
2 * 1024 * 1024 * 1024,
5 * 1024 * 1024 * 1024,
),
4,
100_000,
8,
),
),
)
.await?;
```
```typescript TypeScript icon="typescript"
const job = await table.refreshColumnAsync("embedding", {
supervisorResources: {
cpuMillis: 2_000,
memoryBytes: 4 * 1024 ** 3,
ephemeralStorageBytes: 10 * 1024 ** 3,
},
planningWorkers: {
resources: {
cpuMillis: 1_000,
memoryBytes: 2 * 1024 ** 3,
ephemeralStorageBytes: 5 * 1024 ** 3,
},
maxWorkers: 4,
rowsPerWorker: 100_000,
fragmentsPerWorker: 8,
},
});
await job.wait();
```
The SDK validates the shape and sends these values only with this explicit
refresh request. A supporting LanceDB Enterprise server is required.
*/}
These examples configure only the explicit refresh shown. A registered
Function declaration can separately set `auto_refresh_resource_budget` on
`add_columns`; that declaration controls the budget for its automatic
refreshes. SQL-expression columns reject the automatic resource-budget option
because they do not use Function workers.
To override the registered Function's call concurrency for this logical
execution only:
```python Python icon="python"
job = table.refresh_column_async("embedding", max_concurrency=4)
job.wait()
```
```rust Rust icon="rust"
use lancedb::table::RefreshColumnOptions;
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default().max_concurrency(4),
)
.await?;
```
```typescript TypeScript icon="typescript"
const job = await table.refreshColumnAsync("embedding", {
maxConcurrency: 4,
});
await job.wait();
```
`max_concurrency` caps user-code calls in flight for one Function within one
logical execution. It is not an account-wide quota and does not resize or
rewrite `resource_budget`. Omitting it uses the registration default; a
refresh value overrides that default for this execution only.
`batch_size` caps rows in one Function call; LanceDB adapts actual call and
work sizes. For Function columns, the refresh defaults to
`error_policy = 'fail'`. Set `error_policy = 'skip'` to write null outputs for
failing rows and record them in the error store, subject to failure limits.
See the [SQL refresh options](/sql/dml/refresh-column#options).
{/*
Draft for the Function retry policy release.
For registered Function columns, pass `retry_policy` to
`refresh_column_async` to replace the Function's registered retry policy
for this refresh. For example,
`retry_policy=FunctionRetryPolicy(transient=True, max_attempts=3)` allows
three attempts, including the first call. Import `FunctionRetryPolicy`
from `lancedb.functions`. Use `FunctionRetryPolicy(max_attempts=1)` to
disable user-error retries. Omitting the override preserves the registered
policy.
```python Python icon="python"
from lancedb.functions import FunctionRetryPolicy
job = table.refresh_column_async(
"embedding",
retry_policy=FunctionRetryPolicy(transient=True, max_attempts=3),
)
job.wait()
```
```rust Rust icon="rust"
use lancedb::{function::FunctionRetryPolicy, table::RefreshColumnOptions};
let mut retry_policy = FunctionRetryPolicy::transient();
retry_policy.max_attempts = 3;
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default().retry_policy(retry_policy),
)
.await?;
```
```typescript TypeScript icon="typescript"
const job = await table.refreshColumnAsync("embedding", {
retryPolicy: { transient: true, maxAttempts: 3 },
});
await job.wait();
```
This requires a supporting LanceDB Enterprise server as well as an SDK
that exposes these options. An older Function image may need
re-registration before it accepts a refresh override. Only enable retries
when repeating a failed call's partial work is safe, including any external
requests made before the error.
The attempt budget belongs to one failed Function call. With an explicit
retry policy and `error_policy="skip"`, exhaustion in a batch Function
writes nulls for the whole failed call and records its errors, subject to
failure limits. It does not split the batch to restart the retry budget.
Row Functions isolate failures per row. Infrastructure recovery, framework
deadlines and cancellation remain separate from this user-error policy.
See [Function failure handling](/management/function#failures) for the
retry and replay boundaries.
*/}
For a column filled by a registered Function, the Python and Rust job APIs can
override the Function's GPU placement for this refresh:
```python
classes = db.list_gpu_classes()
gpu_class = next(gpu.name for gpu in classes if gpu.gpu_model == "NVIDIA H100")
job = table.refresh_column_async(
"embedding",
gpu_class=gpu_class,
gpu_count=4,
batch_size=32,
)
```
```rust
use lancedb::{Connection, Table, table::RefreshColumnOptions};
async fn refresh(
conn: &Connection,
table: &Table,
) -> Result<(), Box> {
let gpu_class = conn
.list_gpu_classes()
.await?
.into_iter()
.find(|class| class.gpu_model.as_deref() == Some("NVIDIA H100"))
.map(|class| class.name)
.ok_or("H100 GPU class is not configured")?;
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default()
.gpu_class(gpu_class)
.gpu_count(4)
.batch_size(32),
)
.await?;
Ok(())
}
```
`gpu_count` is the aggregate GPU budget for the refresh's Function workers,
not a request for that many GPUs per worker. It must be positive and preserves
the deployment's CPU and memory defaults.
`gpu_class` must name a class returned by `list_gpu_classes()`; class names
are lowercase DNS labels. Omit either option to use the Function's registered
class or the server's configured budget. `list_gpu_classes()` requires System
Usage permission, and raises `NotImplementedError` when the server predates
GPU class discovery.
The Rust GPU discovery and refresh controls are not in the latest Rust
release. TypeScript does not expose GPU class discovery; pass `gpuClass`
and `gpuCount` in the `refreshColumnAsync` options.
Completed Function outputs publish with their fragment by default. To opt into
a row-count publication threshold, set `publication_interval_rows` in Python,
`publicationIntervalRows` in TypeScript, or `publication_interval_rows()` in Rust:
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
publication_interval_rows=60,
)
job.wait()
```
```typescript TypeScript icon="square-js"
const job = await table.refreshColumnAsync("embedding", {
publicationIntervalRows: 60,
});
await job.wait();
```
```rust Rust icon="rust"
use lancedb::table::RefreshColumnOptions;
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default().publication_interval_rows(60),
)
.await?;
```
The threshold is opt-in and must be greater than zero. The number of rows in an
individual publication can differ from this value depending on completed work
and Function batching; this is not an exact-N contract. Omit it to keep
whole-fragment publication.
In TypeScript, the threshold must be a positive safe integer, at most
`Number.MAX_SAFE_INTEGER`. This option requires a supporting LanceDB Enterprise
server and Function outputs without Blob types; Blob inputs are allowed.
The job APIs also take the row filter and failure handling of the SQL
`REFRESH COLUMN ... WHERE ... WITH (...)` form, so a refresh can be narrowed
to new rows (see [what a filter recomputes](#partial)) and told to skip rows
whose Function call fails:
```python
job = table.refresh_column_async(
"embedding",
where="created_at > date '2026-01-01'",
error_policy="skip",
max_failed_fraction=0.01,
row_timeout_seconds=30,
)
```
```typescript
const job = await table.refreshColumnAsync("embedding", {
where: "created_at > date '2026-01-01'",
errorPolicy: "skip",
maxFailedFraction: 0.01,
rowTimeoutSeconds: 30,
});
```
```rust
use lancedb::{function::FunctionErrorPolicy, table::RefreshColumnOptions};
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default()
.where_clause("created_at > date '2026-01-01'")
.error_policy(FunctionErrorPolicy::Skip)
.max_failed_fraction(0.01)
.row_timeout_seconds(30.0),
)
.await?;
```
`max_failed_rows`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` are accepted the same way; the
[SQL refresh options](/sql/dml/refresh-column#options) describe each one.
The SDK rejects a blank `where`, a `max_failed_fraction` outside 0 to 1, a
timeout that is not a positive number and a zero worker or process count
before sending the request; the server validates everything else. A LanceDB
Embedded refresh rejects every option other than `batch_size`, `gpu_class`
and `gpu_count`; in particular, it rejects `resource_budget` rather than
silently ignoring it.
## When a refresh fails {#failures}
What a failure keeps depends on where the refresh ran. The local, blocking
`refresh_column` writes the column in one commit, so a failure leaves the
column as it was. A refresh job in LanceDB Enterprise -- always
`refresh_column_async` or `REFRESH COLUMN` there, never the blocking form --
commits finished batches as it goes, so some values can remain after a
failure.
Either way, recover by fixing the cause and refreshing again. The new refresh
fills rows that are still empty or whose inputs changed, and normally skips
values that were kept. That is not guaranteed: a refresh can recompute values
that were already written (for example when the record of the inputs they
were computed from was not saved), so do not count on a retry costing only the
rows that did not complete.
For a column filled by a [function](/management/function), the usual cause is
the worker environment rather than the data. See
[environment mismatches](/management/function#mismatch-symptoms) for the
symptoms, which rarely name their cause.
## Related guides {#next}
Waiting on, inspecting, and cancelling server-side work.
The same idea, for a whole derived table.
Refresh on a schedule instead of calling this yourself.