mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
95 lines
3.8 KiB
Plaintext
95 lines
3.8 KiB
Plaintext
---
|
|
title: "REFRESH COLUMN"
|
|
sidebarTitle: "REFRESH COLUMN"
|
|
description: "Filling a computed column: idempotent, scoped, and resumable."
|
|
icon: arrows-rotate
|
|
keywords: ["refresh column", "backfill", "computed column"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A LanceDB extension with no DataFusion equivalent. It is what makes
|
|
[feature engineering](/feature-engineering) reachable from SQL: a column
|
|
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN <column> ON [TABLE] <table>
|
|
[WHERE <predicate>]
|
|
[WITH (<option> = <value> [, ...])]
|
|
```
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN doubled ON my_table
|
|
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
|
```
|
|
</CodeGroup>
|
|
|
|
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
|
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
|
to repeat as data arrives, and safe to re-run after a failure, though a re-run
|
|
can recompute values the failed run had already written.
|
|
|
|
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
|
committing to the whole table.
|
|
|
|
## Options {#options}
|
|
|
|
| Option | Meaning |
|
|
| --- | --- |
|
|
| `num_workers` | how many workers to use |
|
|
| `cpu_millis` | aggregate CPU ceiling across the refresh's workers, not a per-worker size |
|
|
| `memory_bytes` | aggregate memory ceiling across the refresh's workers |
|
|
| `gpu_count` | aggregate GPU budget across the refresh's workers. Function columns only |
|
|
| `gpu_class` | GPU class name for the refresh's workers. Function columns only |
|
|
| `batch_size` | rows passed to each Function call. Function columns only |
|
|
| `max_executor_processes_per_worker` | cap on executor processes per worker. Function columns only |
|
|
| `row_timeout_seconds` | deadline for one Function call. Function columns only |
|
|
| `error_policy` | `'skip'` continues past failing rows instead of failing the job. Function columns only |
|
|
| `max_failed_fraction` | fail the job once more than this share of rows has failed. Function columns only |
|
|
| `max_failed_rows` | fail the job once more than this many rows has failed. Function columns only |
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
REFRESH COLUMN embedding ON documents
|
|
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
|
```
|
|
</CodeGroup>
|
|
|
|
`num_workers` sets a count; the budget options size the refresh's workers in
|
|
aggregate, not each one individually. They are alternatives rather than a
|
|
pair.
|
|
|
|
<Note>
|
|
`num_workers` is refused for a column filled by a registered
|
|
[function](/management/function), which sizes its own workers from the CPU,
|
|
memory and GPU options instead. `error_policy` (and the Function-only options
|
|
above it in the table) is the reverse: it's refused for a plain SQL-expression
|
|
column, which has no per-row failures to skip -- it's accepted only for a
|
|
Function column.
|
|
</Note>
|
|
|
|
## Watching a refresh {#watching}
|
|
|
|
A refresh submits work that outlives the call. Track it with
|
|
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
|
[`SHOW ERRORS`](/sql/ddl/job#errors). For per-split progress, worker fleet
|
|
detail, and a `DESCRIBE JOB` example, see [Watching a refresh](/sql/dml/refresh#watching).
|
|
|
|
## See also {#see-also}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
The same operation from the SDKs, and what a refresh guarantees.
|
|
</Card>
|
|
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
|
Declaring the columns this fills.
|
|
</Card>
|
|
<Card title="REFRESH MATERIALIZED VIEW" icon="arrows-rotate" href="/sql/dml/refresh-materialized-view">
|
|
Bringing a view's stored result up to date, instead of a column.
|
|
</Card>
|
|
</CardGroup>
|