mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-17 16:14:00 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
84 lines
4.0 KiB
Plaintext
84 lines
4.0 KiB
Plaintext
---
|
|
title: "Function Refresh Checkpointing"
|
|
sidebarTitle: "Checkpointing"
|
|
description: "How a Function refresh's progress survives a worker failure or restart, part by part."
|
|
icon: "floppy-disk"
|
|
keywords: ["checkpoint", "recovery", "function refresh", "blob v2", "parts", "resume"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
This page is about checkpointing a [Function refresh](/enterprise/adaptive-execution)
|
|
specifically -- not [PyTorch training checkpointing](/training/checkpointing),
|
|
a different, unrelated mechanism covered elsewhere.
|
|
|
|
A Function refresh fills a column in [parts](/enterprise/adaptive-execution#part-size),
|
|
each one a bounded range of rows. Every part's lifecycle is recorded as a
|
|
durable, append-only event -- started, then completed -- in a log scoped
|
|
to the data file that part belongs to. A worker failure or restart does
|
|
not require redoing the refresh from scratch: the next attempt reads that
|
|
log, finds which parts already completed, and only works on what's left.
|
|
Retrying is at-least-once at the part level, not exactly-once: a part
|
|
that started but never recorded completion is redone in full on the next
|
|
attempt, so a Function with external side effects (a remote API call, for
|
|
example) can have those side effects repeated for that part's rows.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
S["Part started<br/>event recorded"] --> E["Function executes"]
|
|
E --> C["Part completed<br/>event recorded"]
|
|
C --> M{"More parts?"}
|
|
M -->|yes| S
|
|
M -->|no| F["Finalized:<br/>digest-verified assembly<br/>into the real data file"]
|
|
|
|
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
|
|
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
|
|
```
|
|
|
|
## What gets recorded, and when {#events}
|
|
|
|
Each part contributes two events to its data file's log: one when a
|
|
worker starts it, one when it completes with the rows it filled, the rows
|
|
it skipped, and how long it took. A completed part additionally carries
|
|
enough identity that a *different* job -- one that took over after the
|
|
part's original job died -- can verify the part is really done and adopt
|
|
it rather than redo it. Once every part for a data file has completed,
|
|
one last event finalizes it: the assembled file's checksum is verified
|
|
against what the parts actually wrote before it's accepted as the real
|
|
data file, not just assumed correct because every part reported success.
|
|
|
|
Writes to this log are batched, not one durable write per event, so
|
|
checkpointing a fast-finishing part costs a fraction of a write rather
|
|
than a full one.
|
|
|
|
## The Blob v2 optimization {#blob-v2}
|
|
|
|
A Function whose output includes a [Blob v2](/management/table#managed-blobs)
|
|
column needs an identifier for every blob it writes. Allocating those
|
|
IDs piggybacks on the same durable event that already records a part
|
|
starting: the event log itself hands out the next unclaimed range,
|
|
computed from the highest range any earlier part in this log has already
|
|
claimed. This means:
|
|
|
|
- **No separate allocation round trip.** The ID range arrives for free
|
|
inside a write the part was already going to make durably.
|
|
- **A retried part can't collide with itself or with other parts.** Each
|
|
attempt at a part writes a new part-started event and gets a fresh range
|
|
past the highest range already claimed in this log, so no two attempts --
|
|
of the same part or of different parts -- ever hold overlapping ranges.
|
|
Re-sending the *same* event, for example after a lost acknowledgement, is
|
|
idempotent and returns the range it was already given. A range left
|
|
behind by a failed attempt is not reclaimed.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
|
|
Part sizing, and why checkpoint frequency follows it.
|
|
</Card>
|
|
<Card title="Create a Function" icon="code" href="/curation/create-function">
|
|
Shaping a Function, including ones that return Blob v2 output.
|
|
</Card>
|
|
</CardGroup>
|