mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-18 09:42:27 +00:00
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
442 lines
17 KiB
Plaintext
442 lines
17 KiB
Plaintext
---
|
|
title: "Branches"
|
|
sidebarTitle: "Branch"
|
|
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
|
|
icon: "code-branch"
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Branches work in the embedded library and in LanceDB Enterprise. Comparing a
|
|
branch with `main` and promoting its new columns onto `main` is available
|
|
only in LanceDB Enterprise.
|
|
---
|
|
import {
|
|
PyConnect,
|
|
TsConnect,
|
|
RsConnect,
|
|
PyConnectEnterprisePlaceholders,
|
|
TsConnectEnterprisePlaceholders,
|
|
RsConnectEnterprisePlaceholders,
|
|
} from '/snippets/connection.mdx';
|
|
import {
|
|
PyBranchCreate as BranchCreate,
|
|
TsBranchCreate,
|
|
RsBranchCreate,
|
|
PyBranchWrite as BranchWrite,
|
|
TsBranchWrite,
|
|
RsBranchWrite,
|
|
PyBranchReopen as BranchReopen,
|
|
TsBranchReopen,
|
|
RsBranchReopen,
|
|
PyBranchDelete as BranchDelete,
|
|
TsBranchDelete,
|
|
RsBranchDelete,
|
|
PyBranchUpsertToMain as BranchUpsertToMain,
|
|
TsBranchUpsertToMain,
|
|
RsBranchUpsertToMain,
|
|
PyBranchIndex as BranchIndex,
|
|
TsBranchIndex,
|
|
RsBranchIndex,
|
|
} from '/snippets/tables.mdx';
|
|
|
|
A branch is an isolated, writable line of history forked from `main` (or from any
|
|
other branch). Anything you do on a branch (e.g., adding rows, changing the schema,
|
|
building an index) stays on that branch, so `main` keeps serving production
|
|
reads exactly as before. Branches are a natural fit when you want to:
|
|
|
|
- Experiment with a new index, schema change, or reprocessing step without
|
|
affecting live queries on `main`.
|
|
- Run a backfill or migration you'd like to validate before applying it to
|
|
`main`.
|
|
- Hand a collaborator a frozen point-in-time fork while you keep writing to `main`.
|
|
|
|
## How branches relate to versions and tags {#how-branches-relate-to-versions-and-tags}
|
|
|
|
Every LanceDB table already tracks a linear history of [versions](/management/table#versioning),
|
|
and you can [tag](/management/table#versioning-tag-based-versioning) a version or `checkout`
|
|
one to read it. Branches add the missing piece: a *separate, writable* line of
|
|
history. Where a tag is a read-only label and `checkout` is a read-only view, a
|
|
branch forks from a point in history and then evolves on its own. Creating or
|
|
checking out a branch hands you a new table handle whose reads and writes are
|
|
scoped to that branch. The [comparison table](#branches-vs-tags-vs-versions) at
|
|
the end of this page lays out when to reach for each.
|
|
|
|
<Note>
|
|
Branches are supported on local and namespace-backed tables in LanceDB OSS, as
|
|
well as on LanceDB Enterprise (remote) tables.
|
|
</Note>
|
|
|
|
## Connect to a table {#connect-to-a-table}
|
|
|
|
The branch API is identical no matter how you connect — only the connection
|
|
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
|
|
a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
|
|
in every example that follows.
|
|
|
|
### LanceDB OSS {#lancedb-oss}
|
|
|
|
Point LanceDB at a local directory (or an object-storage URI) to use it as an
|
|
embedded library.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnect}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
|
|
{TsConnect}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{ "use lancedb::connect;\n\n" }
|
|
{RsConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### LanceDB Enterprise {#lancedb-enterprise}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
|
|
noted below, but you connect to your
|
|
Enterprise deployment with a `db://` URI, an API key, and your
|
|
region and endpoint. Replace the placeholders below, including `us-east-1` if your
|
|
deployment uses another region. In application code, load credentials from environment
|
|
variables or a secrets manager. Once connected, open a table and use the same branch
|
|
calls in every example that follows.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{ "use lancedb::connect;\n\n" }
|
|
{RsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Work with branches {#work-with-branches}
|
|
|
|
The lifecycle of a branch is short and predictable: fork it, write to it, reopen
|
|
it whenever you need it, and delete it once you're done. The examples below use a
|
|
small `quotes` table with three rows on `main`.
|
|
|
|
### Create a branch {#create-a-branch}
|
|
|
|
Forking from `main` returns a table handle scoped to the new branch. `main` is
|
|
the reserved default source, so `create` needs only a name; to fork from
|
|
somewhere else, pass a branch name, a specific version, or both.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchCreate}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchCreate}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchCreate}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Write to a branch {#write-to-a-branch}
|
|
|
|
Writes go through the branch handle and stay there — the `main` handle keeps
|
|
reporting its original row count. Listing branches returns a mapping of each
|
|
branch name to its metadata, including the version it was forked from.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchWrite}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchWrite}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchWrite}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Reopen a branch {#reopen-a-branch}
|
|
|
|
A branch outlives the handle that created it. Reopen it later by name — either
|
|
from an existing table handle or straight from the connection when you open the
|
|
table. Both routes give you a writable handle tracking the branch's latest state.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchReopen}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchReopen}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchReopen}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Delete a branch {#delete-a-branch}
|
|
|
|
Deleting a branch removes it and its branch-local history; `main` is untouched.
|
|
A deployment refuses to delete a branch whose columns have been promoted onto
|
|
`main`, because `main` still depends on it.
|
|
Before deleting a branch, make sure you've retained any results you need — see
|
|
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
|
|
below.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchDelete}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchDelete}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchDelete}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Apply branch-tested changes to `main` {#apply-branch-tested-changes-to-main}
|
|
|
|
<Warning>
|
|
A branch has its own writable history. Outside of the [diff and cherry-pick
|
|
APIs](#compare-and-merge-a-branch-into-main) — which are available on
|
|
LanceDB Enterprise and promote added columns only — LanceDB does not reconcile
|
|
one branch's history with another or detect conflicts between them. To carry
|
|
other accepted work forward, rerun the validated operation against `main` or
|
|
explicitly write selected results to it.
|
|
</Warning>
|
|
|
|
How you apply a validated change depends on the type of work:
|
|
|
|
- **Added columns (Enterprise):** review the branch with `diff`, then promote
|
|
the new columns onto `main` with `cherry_pick` (`cherryPick` in TypeScript). See
|
|
[Compare and promote a branch into `main`](#compare-and-merge-a-branch-into-main).
|
|
- **Backfill or transformation:** rerun the validated job against `main`.
|
|
- **Schema change:** apply the same reviewed schema operation to `main`.
|
|
- **Index change:** build the index on `main` using the configuration validated
|
|
on the branch.
|
|
- **Selected row results:** upsert those rows into `main` using a stable unique
|
|
key.
|
|
|
|
### Upsert selected branch rows into `main` {#upsert-selected-branch-rows-into-main}
|
|
|
|
If the result you want to retain is a set of inserted or updated rows, use
|
|
[`merge_insert`](/ingestion/merge-insert#merge-incoming-rows-by-key) to write them to
|
|
`main`. Despite its name, `merge_insert` is a row-ingestion operation: it
|
|
matches incoming rows by key and does not merge branch histories.
|
|
|
|
Read the rows you want from the reviewed branch, then upsert them into `main` on
|
|
your key column — `id` in this example. Here, `candidate` is the branch handle:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchUpsertToMain}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchUpsertToMain}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchUpsertToMain}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
With LanceDB Enterprise, the synchronous Python table raises for `to_arrow()`; read
|
|
the branch's rows with a query instead -- see
|
|
[Avoid materializing the whole table](/search#avoid-materializing-the-whole-table).
|
|
|
|
This operation transfers only the inserted or updated rows in its input. It does
|
|
not transfer branch history, schema changes, indexes, or branch-local deletions,
|
|
and it does not determine which rows changed after the branch was created.
|
|
|
|
Reapplying an identical payload is idempotent on the key, but the operation is
|
|
not conflict-aware. If `main` has diverged, incoming branch rows can overwrite
|
|
newer values with the same key. Read and upsert the whole branch only when that
|
|
overwrite is intentional; otherwise, filter the branch read to the rows you
|
|
intend to apply.
|
|
|
|
## Compare and promote a branch into `main` {#compare-and-merge-a-branch-into-main}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
On LanceDB Enterprise, branches expose two review-and-land calls: `diff`
|
|
inspects what a branch has changed relative to `main`, and `cherry_pick`
|
|
(`cherryPick` in TypeScript) promotes the branch's new columns onto `main` in
|
|
place, without reissuing the branch's writes.
|
|
|
|
<Info>
|
|
`diff` and `cherry_pick` work on Enterprise (remote) tables only. On a local
|
|
table they fail as not supported: Python raises `NotImplementedError`, and
|
|
TypeScript and Rust return an error. A cherry-pick promotes added columns; use
|
|
the [upsert](#upsert-selected-branch-rows-into-main) or rerun patterns above for
|
|
row and index changes.
|
|
</Info>
|
|
|
|
### Diff a branch {#diff-a-branch}
|
|
|
|
`diff` reads the branch and `main` without modifying either, and returns what
|
|
has changed: columns added, removed or changed, indexes added or removed, row
|
|
counts, and a list of `errors` -- the reasons a cherry-pick of the branch could
|
|
not land now. An empty list means it could.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
diff = table.branches.diff("exp")
|
|
|
|
print([c["name"] for c in diff["addedColumns"]]) # columns the branch introduced
|
|
print(diff["baseMoved"]) # True if main has moved on since the branch's fork
|
|
for error in diff["errors"]: # empty when a cherry-pick could land
|
|
print(error["code"], error["message"])
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const branches = await table.branches();
|
|
const diff = await branches.diff("exp");
|
|
|
|
console.log(diff.addedColumns.map((c) => c.name)); // columns the branch introduced
|
|
console.log(diff.baseMoved); // true if main has moved on since the branch's fork
|
|
for (const error of diff.errors) { // empty when a cherry-pick could land
|
|
console.log(error.code, error.message);
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
The result also carries the versions it compared (`parentVersion`,
|
|
`mainVersion`, `branchVersion`), `rowCountMain` and `rowCountBranch`, a
|
|
`rowSummary`, and `removedColumns`, `changedColumns`, `addedIndexes` and
|
|
`removedIndexes`. In Python the result is a dictionary with the same camelCase
|
|
keys.
|
|
|
|
The error codes are `baseMoved` (`main` has moved on since the version the
|
|
branch was forked from), `rowCountMismatch`, `rowsChanged`, `columnRemoved`,
|
|
`columnChanged`, `nothingToApply`, `noColumnChanges`, `inputColumnDependency`
|
|
and `parentNotMain`. A code the client does not know surfaces as `unknown`, so
|
|
older clients keep working.
|
|
|
|
### Promote a branch's columns {#merge-a-branch}
|
|
|
|
`cherry_pick` promotes the columns a branch added onto `main`. The server
|
|
re-evaluates the diff when the request arrives and either lands the change or
|
|
refuses it for the reasons `diff` would report. A refusal is not an exception:
|
|
the call returns a result whose `status` is `"failed"`, with the reasons in
|
|
`diff.errors`, so you can decide what to do next. Cherry-picks are not retried.
|
|
|
|
Pass `dry_run=True` in Python, or `true` as the second argument in TypeScript,
|
|
to check without landing anything. `preview.promotedColumns` lists the columns
|
|
the cherry-pick would promote, or did.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Check first; this does not modify main.
|
|
preview = table.branches.cherry_pick("exp", dry_run=True)
|
|
print(preview["status"]) # "ready" when it would land
|
|
print(preview["preview"]["promotedColumns"])
|
|
|
|
# Land it.
|
|
result = table.branches.cherry_pick("exp")
|
|
if result["status"] == "cherryPicked":
|
|
print("landed at main version", result["mainVersionAfter"])
|
|
elif result["status"] == "failed":
|
|
for error in result["diff"]["errors"]:
|
|
print(error["code"], error["message"])
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const branches = await table.branches();
|
|
|
|
// Check first; this does not modify main.
|
|
const preview = await branches.cherryPick("exp", true);
|
|
console.log(preview.status); // "ready" when it would land
|
|
console.log(preview.preview.promotedColumns);
|
|
|
|
// Land it.
|
|
const result = await branches.cherryPick("exp");
|
|
if (result.status === "cherryPicked") {
|
|
console.log("landed at main version", result.mainVersionAfter);
|
|
} else if (result.status === "failed") {
|
|
for (const error of result.diff.errors) {
|
|
console.log(error.code, error.message);
|
|
}
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
`status` is `ready` (a dry run that would land), `cherryPicked` (it landed, and
|
|
`mainVersionAfter` is the new version of `main`), `failed` (refused, or for a
|
|
dry run, would be refused; see `diff.errors`), `notImplemented`, or `unknown`
|
|
for a status the client does not know. A branch that does not exist raises a
|
|
not-found error instead of returning a status.
|
|
|
|
In async Python, await the same calls on an `AsyncTable`:
|
|
`await table.branches.diff("exp")` and
|
|
`await table.branches.cherry_pick("exp", dry_run=True)`. In Rust they are
|
|
`Table::diff_branch` and `Table::cherry_pick`.
|
|
|
|
## Build indexes on a branch {#build-indexes-on-a-branch}
|
|
|
|
One of the most useful things a branch buys you is a safe place to build and
|
|
validate an index without affecting what's in production on the `main` branch.
|
|
Fork a branch, create your vector (ANN) and full-text search (FTS)
|
|
indexes on it, and check recall and latency. Once you have selected a
|
|
configuration, build the corresponding index on `main` through your normal
|
|
deployment workflow. Because the indexes live on the branch, queries against
|
|
`main` never see a half-built index and are never slowed down by the branch's
|
|
build.
|
|
|
|
This pattern is especially valuable on Enterprise deployments, where `main` is
|
|
typically serving production traffic while you tune an index configuration on
|
|
the side. A deployment builds the branch's indexes in the background, so wait
|
|
for them before measuring.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchIndex}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchIndex}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchIndex}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Schema changes such as adding, altering, or dropping columns are branch-scoped in
|
|
the same way, so you can stage and review a larger reshaping of a table before
|
|
applying the same schema operations to `main`.
|
|
|
|
## Branches vs. tags vs. versions {#branches-vs-tags-vs-versions}
|
|
|
|
Now that you're familiar with branches, you can see how they complement the other ways LanceDB
|
|
give you to work with table history. Choose these approaches based on whether you need to
|
|
*label*, *read*, or *write* data at a point in history:
|
|
|
|
| Feature | Writable? | Purpose |
|
|
| --- | --- | --- |
|
|
| **[Branch](#work-with-branches)** | ✅ Yes | Write on top of a point in history without touching `main`. |
|
|
| **[Tag](/management/table#versioning-tag-based-versioning)** | ❌ No | Attach a human-readable label to an existing version; protects it from cleanup. |
|
|
| **[`checkout(version)`](/management/table#versioning-rollback-to-previous-versions)** | ❌ No | Read a historical version of `main` without forking. Read-only until you `restore`. |
|
|
|
|
<Note>
|
|
For linear version history — creating versions, listing them, rolling back, and
|
|
tagging — see the [Versioning and Reproducibility](/management/table#versioning) guide.
|
|
</Note>
|