Files
lancedb/docs/web/enterprise/checkpointing.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

84 lines
4.0 KiB
Plaintext

---
title: "Function Refresh Checkpointing"
sidebarTitle: "Checkpointing"
description: "How a Function refresh's progress survives a worker failure or restart, part by part."
icon: "floppy-disk"
keywords: ["checkpoint", "recovery", "function refresh", "blob v2", "parts", "resume"]
availability:
embedded: unavailable
enterprise: available
---
This page is about checkpointing a [Function refresh](/enterprise/adaptive-execution)
specifically -- not [PyTorch training checkpointing](/training/checkpointing),
a different, unrelated mechanism covered elsewhere.
A Function refresh fills a column in [parts](/enterprise/adaptive-execution#part-size),
each one a bounded range of rows. Every part's lifecycle is recorded as a
durable, append-only event -- started, then completed -- in a log scoped
to the data file that part belongs to. A worker failure or restart does
not require redoing the refresh from scratch: the next attempt reads that
log, finds which parts already completed, and only works on what's left.
Retrying is at-least-once at the part level, not exactly-once: a part
that started but never recorded completion is redone in full on the next
attempt, so a Function with external side effects (a remote API call, for
example) can have those side effects repeated for that part's rows.
```mermaid
flowchart LR
S["Part started<br/>event recorded"] --> E["Function executes"]
E --> C["Part completed<br/>event recorded"]
C --> M{"More parts?"}
M -->|yes| S
M -->|no| F["Finalized:<br/>digest-verified assembly<br/>into the real data file"]
S -. "worker dies here" .-> R["Next attempt reads the log,<br/>sees no completion,<br/>redoes only this part"]
C -. "worker dies here" .-> R2["Next attempt reads the log,<br/>sees this part done,<br/>inherits it, moves on"]
```
## What gets recorded, and when {#events}
Each part contributes two events to its data file's log: one when a
worker starts it, one when it completes with the rows it filled, the rows
it skipped, and how long it took. A completed part additionally carries
enough identity that a *different* job -- one that took over after the
part's original job died -- can verify the part is really done and adopt
it rather than redo it. Once every part for a data file has completed,
one last event finalizes it: the assembled file's checksum is verified
against what the parts actually wrote before it's accepted as the real
data file, not just assumed correct because every part reported success.
Writes to this log are batched, not one durable write per event, so
checkpointing a fast-finishing part costs a fraction of a write rather
than a full one.
## The Blob v2 optimization {#blob-v2}
A Function whose output includes a [Blob v2](/management/table#managed-blobs)
column needs an identifier for every blob it writes. Allocating those
IDs piggybacks on the same durable event that already records a part
starting: the event log itself hands out the next unclaimed range,
computed from the highest range any earlier part in this log has already
claimed. This means:
- **No separate allocation round trip.** The ID range arrives for free
inside a write the part was already going to make durably.
- **A retried part can't collide with itself or with other parts.** Each
attempt at a part writes a new part-started event and gets a fresh range
past the highest range already claimed in this log, so no two attempts --
of the same part or of different parts -- ever hold overlapping ranges.
Re-sending the *same* event, for example after a lost acknowledgement, is
idempotent and returns the range it was already given. A range left
behind by a failed attempt is not reclaimed.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Adaptive Execution" icon="gauge-high" href="/enterprise/adaptive-execution">
Part sizing, and why checkpoint frequency follows it.
</Card>
<Card title="Create a Function" icon="code" href="/curation/create-function">
Shaping a Function, including ones that return Blob v2 output.
</Card>
</CardGroup>