mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-14 22:56:54 +00:00
Jack YeandClaude Sonnet 5
de95ed3f6d
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
97 lines
4.4 KiB
Plaintext
97 lines
4.4 KiB
Plaintext
---
|
|
title: "Auto Refresh"
|
|
sidebarTitle: "Auto Refresh"
|
|
description: "Let a computed column or materialized view refresh itself on a schedule, instead of calling refresh explicitly."
|
|
icon: "clock-rotate-left"
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
Both a [computed column](/feature-engineering/add-computed-column) and a
|
|
[materialized view](/feature-engineering/materialized-views) can refresh
|
|
themselves on a schedule instead of waiting for an explicit
|
|
`refresh_column()`, `view.refresh()`, or `REFRESH` statement. Either way,
|
|
`auto_refresh_enabled` turns it on and `auto_refresh_interval_seconds` sets
|
|
a minimum delay before the next automatic refresh is eligible -- omit the
|
|
interval to let the server decide. What that delay is measured from differs
|
|
between the two; see [Pacing follows different state for
|
|
each](#history) below. See [Architecture](/index#architecture) for the
|
|
scheduler's place among the jobs fleet.
|
|
|
|
## Where the two differ {#granularity}
|
|
|
|
A computed column's auto-refresh setting is per output: each column
|
|
declared in the same `add_columns` call gets its own independent
|
|
interval, set when you declare it or changed later. Automatic execution
|
|
additionally needs the deployment's `column_refresh.enabled` switch,
|
|
which is off in the base configuration.
|
|
|
|
```sql
|
|
ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_enabled = false;
|
|
ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_enabled = true;
|
|
ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_interval_seconds = 300;
|
|
```
|
|
|
|
A materialized view's setting is whole-view -- there is no per-column
|
|
granularity the way a computed column has one setting per output. The
|
|
view's *initial* population always happens regardless of this setting;
|
|
only the refreshes after that are gated. Unlike a computed column, a view's
|
|
setting is declared at `CREATE` time, but it can still be changed
|
|
afterward with `ALTER`:
|
|
|
|
```sql
|
|
CREATE MATERIALIZED VIEW my_view
|
|
WITH (auto_refresh_enabled = true, auto_refresh_interval_seconds = 300)
|
|
AS SELECT * FROM source_table;
|
|
|
|
ALTER MATERIALIZED VIEW my_view SET auto_refresh_enabled = false;
|
|
ALTER MATERIALIZED VIEW my_view SET auto_refresh_enabled = true;
|
|
ALTER MATERIALIZED VIEW my_view SET auto_refresh_interval_seconds = 300;
|
|
```
|
|
|
|
Each `SET` above changes the scheduling setting immediately and
|
|
synchronously; it does not itself run a refresh. Whether and when a
|
|
refresh actually runs next is a separate, asynchronous decision the
|
|
scheduler makes against the pacing rules below. If values stop appearing
|
|
after a `SET`, check the column's Job Registry history rather than assuming
|
|
the setting itself failed to apply.
|
|
|
|
## Pacing follows different state for each {#history}
|
|
|
|
A computed column's pacing is measured from the last successful refresh
|
|
recorded in the Job Registry's live history, not from a separate
|
|
timestamp LanceDB keeps elsewhere. If completed jobs are archived before
|
|
the column's configured interval has elapsed, the scheduler can lose that
|
|
last-success record and refresh earlier than the interval implies. Leave
|
|
archival disabled, or retain live history for at least the longest
|
|
configured interval you rely on. A deployment archives completed jobs by
|
|
default, after a default seven-day retention threshold; check with your
|
|
deployment operator if you rely on a different setting.
|
|
|
|
A materialized view's pacing instead reads a timestamp the view itself
|
|
persists for when it was last checked, not Job Registry history -- so
|
|
archival doesn't affect it. Once the interval has elapsed, the scheduler
|
|
also skips the refresh outright if the source table's version is
|
|
unchanged since the view's last check, rather than re-running a refresh
|
|
that would produce the same rows.
|
|
|
|
## Catching up {#catch-up}
|
|
|
|
Re-enabling a column's auto-refresh after it was off catches up the
|
|
entire accumulated backlog in one pass, not just rows written from that
|
|
point on -- "catch up," not "resume cadence." Disabling never skips or
|
|
drops writes either: rows written while disabled simply commit with no
|
|
computed value yet, the same as a row written before the column existed.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Manual refresh, partial refreshes, and what happens on failure.
|
|
</Card>
|
|
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/refresh-materialized-view">
|
|
Manual refresh and what happens on failure, for a view.
|
|
</Card>
|
|
</CardGroup>
|