mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 00:02:26 +00:00
The TypeScript connection example only defined a helper that nothing called, so every later step failed with `db is not defined`. It now connects, as the Python one does. The Rust one kept its connection inside a helper, the Enterprise one read its settings without connecting, and the later steps relied on imports and a record-batch helper the page never showed. The quickstart now says how the steps make one program: an ES module run with Node in TypeScript, and the body of an async `main` in Rust, whose dependencies the install step lists. Each Rust step brings its own imports, the data step shows the helper that turns the records into a reader, and Arrow comes from lancedb's re-exports, so no Arrow crate is needed. The row structs no longer derive serde traits nothing used. The connection examples are shared, so the connect, create-table and branching pages gain the import line their connection code now needs.
440 lines
17 KiB
Plaintext
440 lines
17 KiB
Plaintext
---
|
|
title: "Branches"
|
|
sidebarTitle: "Branch"
|
|
description: "Fork isolated, writable lines of table history in LanceDB. Run experiments, backfills, and index rebuilds without disturbing production reads on main."
|
|
icon: "code-branch"
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Branches work in the embedded library and on a deployment. Comparing a
|
|
branch with `main` and promoting its new columns onto `main` is available
|
|
only on a deployment.
|
|
---
|
|
import {
|
|
PyConnect,
|
|
TsConnect,
|
|
RsConnect,
|
|
PyConnectEnterpriseQuickstart,
|
|
TsConnectEnterpriseQuickstart,
|
|
RsConnectEnterpriseQuickstart,
|
|
} from '/snippets/connection.mdx';
|
|
import {
|
|
PyBranchCreate as BranchCreate,
|
|
TsBranchCreate,
|
|
RsBranchCreate,
|
|
PyBranchWrite as BranchWrite,
|
|
TsBranchWrite,
|
|
RsBranchWrite,
|
|
PyBranchReopen as BranchReopen,
|
|
TsBranchReopen,
|
|
RsBranchReopen,
|
|
PyBranchDelete as BranchDelete,
|
|
TsBranchDelete,
|
|
RsBranchDelete,
|
|
PyBranchUpsertToMain as BranchUpsertToMain,
|
|
TsBranchUpsertToMain,
|
|
RsBranchUpsertToMain,
|
|
PyBranchIndex as BranchIndex,
|
|
TsBranchIndex,
|
|
RsBranchIndex,
|
|
} from '/snippets/tables.mdx';
|
|
|
|
A branch is an isolated, writable line of history forked from `main` (or from any
|
|
other branch). Anything you do on a branch (e.g., adding rows, changing the schema,
|
|
building an index) stays on that branch, so `main` keeps serving production
|
|
reads exactly as before. Branches are a natural fit when you want to:
|
|
|
|
- Experiment with a new index, schema change, or reprocessing step without
|
|
affecting live queries on `main`.
|
|
- Run a backfill or migration you'd like to validate before applying it to
|
|
`main`.
|
|
- Hand a collaborator a frozen point-in-time fork while you keep writing to `main`.
|
|
|
|
## How branches relate to versions and tags {#how-branches-relate-to-versions-and-tags}
|
|
|
|
Every LanceDB table already tracks a linear history of [versions](/management/table#versioning),
|
|
and you can [tag](/management/table#versioning-tag-based-versioning) a version or `checkout`
|
|
one to read it. Branches add the missing piece: a *separate, writable* line of
|
|
history. Where a tag is a read-only label and `checkout` is a read-only view, a
|
|
branch forks from a point in history and then evolves on its own. Creating or
|
|
checking out a branch hands you a new table handle whose reads and writes are
|
|
scoped to that branch. The [comparison table](#branches-vs-tags-vs-versions) at
|
|
the end of this page lays out when to reach for each.
|
|
|
|
<Note>
|
|
Branches are supported on local and namespace-backed tables in LanceDB OSS, as
|
|
well as on LanceDB Enterprise (remote) tables.
|
|
</Note>
|
|
|
|
## Connect to a table {#connect-to-a-table}
|
|
|
|
The branch API is identical no matter how you connect — only the connection
|
|
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
|
|
a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
|
|
in every example that follows.
|
|
|
|
### LanceDB OSS {#lancedb-oss}
|
|
|
|
Point LanceDB at a local directory (or an object-storage URI) to use it as an
|
|
embedded library.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnect}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
|
|
{TsConnect}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{ "use lancedb::connect;\n\n" }
|
|
{RsConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### LanceDB Enterprise {#lancedb-enterprise}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Branching on LanceDB Enterprise works the same way as on OSS, apart from the differences
|
|
noted below, but you connect to your
|
|
Enterprise deployment with a `db://` URI, an API key, and your
|
|
region. Once you have a connection, open a table and use the same branch calls in
|
|
every example that follows.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectEnterpriseQuickstart}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsConnectEnterpriseQuickstart}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{ "use lancedb::connect;\n\n" }
|
|
{RsConnectEnterpriseQuickstart}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Work with branches {#work-with-branches}
|
|
|
|
The lifecycle of a branch is short and predictable: fork it, write to it, reopen
|
|
it whenever you need it, and delete it once you're done. The examples below use a
|
|
small `quotes` table with three rows on `main`.
|
|
|
|
### Create a branch {#create-a-branch}
|
|
|
|
Forking from `main` returns a table handle scoped to the new branch. `main` is
|
|
the reserved default source, so `create` needs only a name; to fork from
|
|
somewhere else, pass a branch name, a specific version, or both.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchCreate}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchCreate}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchCreate}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Write to a branch {#write-to-a-branch}
|
|
|
|
Writes go through the branch handle and stay there — the `main` handle keeps
|
|
reporting its original row count. Listing branches returns a mapping of each
|
|
branch name to its metadata, including the version it was forked from.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchWrite}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchWrite}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchWrite}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Reopen a branch {#reopen-a-branch}
|
|
|
|
A branch outlives the handle that created it. Reopen it later by name — either
|
|
from an existing table handle or straight from the connection when you open the
|
|
table. Both routes give you a writable handle tracking the branch's latest state.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchReopen}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchReopen}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchReopen}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Delete a branch {#delete-a-branch}
|
|
|
|
Deleting a branch removes it and its branch-local history; `main` is untouched.
|
|
A deployment refuses to delete a branch whose columns have been promoted onto
|
|
`main`, because `main` still depends on it.
|
|
Before deleting a branch, make sure you've retained any results you need — see
|
|
[Apply branch-tested changes to `main`](#apply-branch-tested-changes-to-main)
|
|
below.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchDelete}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchDelete}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchDelete}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Apply branch-tested changes to `main` {#apply-branch-tested-changes-to-main}
|
|
|
|
<Warning>
|
|
A branch has its own writable history. Outside of the [diff and cherry-pick
|
|
APIs](#compare-and-merge-a-branch-into-main) — which are available on
|
|
LanceDB Enterprise and promote added columns only — LanceDB does not reconcile
|
|
one branch's history with another or detect conflicts between them. To carry
|
|
other accepted work forward, rerun the validated operation against `main` or
|
|
explicitly write selected results to it.
|
|
</Warning>
|
|
|
|
How you apply a validated change depends on the type of work:
|
|
|
|
- **Added columns (Enterprise):** review the branch with `diff`, then promote
|
|
the new columns onto `main` with `cherry_pick` (`cherryPick` in TypeScript). See
|
|
[Compare and promote a branch into `main`](#compare-and-merge-a-branch-into-main).
|
|
- **Backfill or transformation:** rerun the validated job against `main`.
|
|
- **Schema change:** apply the same reviewed schema operation to `main`.
|
|
- **Index change:** build the index on `main` using the configuration validated
|
|
on the branch.
|
|
- **Selected row results:** upsert those rows into `main` using a stable unique
|
|
key.
|
|
|
|
### Upsert selected branch rows into `main` {#upsert-selected-branch-rows-into-main}
|
|
|
|
If the result you want to retain is a set of inserted or updated rows, use
|
|
[`merge_insert`](/ingestion/merge-insert#merge-incoming-rows-by-key) to write them to
|
|
`main`. Despite its name, `merge_insert` is a row-ingestion operation: it
|
|
matches incoming rows by key and does not merge branch histories.
|
|
|
|
Read the rows you want from the reviewed branch, then upsert them into `main` on
|
|
your key column — `id` in this example. Here, `candidate` is the branch handle:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchUpsertToMain}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchUpsertToMain}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchUpsertToMain}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Against a deployment, the synchronous Python table raises for `to_arrow()`; read
|
|
the branch's rows with a query instead -- see
|
|
[Avoid materializing the whole table](/search#avoid-materializing-the-whole-table).
|
|
|
|
This operation transfers only the inserted or updated rows in its input. It does
|
|
not transfer branch history, schema changes, indexes, or branch-local deletions,
|
|
and it does not determine which rows changed after the branch was created.
|
|
|
|
Reapplying an identical payload is idempotent on the key, but the operation is
|
|
not conflict-aware. If `main` has diverged, incoming branch rows can overwrite
|
|
newer values with the same key. Read and upsert the whole branch only when that
|
|
overwrite is intentional; otherwise, filter the branch read to the rows you
|
|
intend to apply.
|
|
|
|
## Compare and promote a branch into `main` {#compare-and-merge-a-branch-into-main}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
On LanceDB Enterprise, branches expose two review-and-land calls: `diff`
|
|
inspects what a branch has changed relative to `main`, and `cherry_pick`
|
|
(`cherryPick` in TypeScript) promotes the branch's new columns onto `main` in
|
|
place, without reissuing the branch's writes.
|
|
|
|
<Info>
|
|
`diff` and `cherry_pick` work on Enterprise (remote) tables only. On a local
|
|
table they fail as not supported: Python raises `NotImplementedError`, and
|
|
TypeScript and Rust return an error. A cherry-pick promotes added columns; use
|
|
the [upsert](#upsert-selected-branch-rows-into-main) or rerun patterns above for
|
|
row and index changes.
|
|
</Info>
|
|
|
|
### Diff a branch {#diff-a-branch}
|
|
|
|
`diff` reads the branch and `main` without modifying either, and returns what
|
|
has changed: columns added, removed or changed, indexes added or removed, row
|
|
counts, and a list of `errors` -- the reasons a cherry-pick of the branch could
|
|
not land now. An empty list means it could.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
diff = table.branches.diff("exp")
|
|
|
|
print([c["name"] for c in diff["addedColumns"]]) # columns the branch introduced
|
|
print(diff["baseMoved"]) # True if main has moved on since the branch's fork
|
|
for error in diff["errors"]: # empty when a cherry-pick could land
|
|
print(error["code"], error["message"])
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const branches = await table.branches();
|
|
const diff = await branches.diff("exp");
|
|
|
|
console.log(diff.addedColumns.map((c) => c.name)); // columns the branch introduced
|
|
console.log(diff.baseMoved); // true if main has moved on since the branch's fork
|
|
for (const error of diff.errors) { // empty when a cherry-pick could land
|
|
console.log(error.code, error.message);
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
The result also carries the versions it compared (`parentVersion`,
|
|
`mainVersion`, `branchVersion`), `rowCountMain` and `rowCountBranch`, a
|
|
`rowSummary`, and `removedColumns`, `changedColumns`, `addedIndexes` and
|
|
`removedIndexes`. In Python the result is a dictionary with the same camelCase
|
|
keys.
|
|
|
|
The error codes are `baseMoved` (`main` has moved on since the version the
|
|
branch was forked from), `rowCountMismatch`, `rowsChanged`, `columnRemoved`,
|
|
`columnChanged`, `nothingToApply`, `noColumnChanges`, `inputColumnDependency`
|
|
and `parentNotMain`. A code the client does not know surfaces as `unknown`, so
|
|
older clients keep working.
|
|
|
|
### Promote a branch's columns {#merge-a-branch}
|
|
|
|
`cherry_pick` promotes the columns a branch added onto `main`. The server
|
|
re-evaluates the diff when the request arrives and either lands the change or
|
|
refuses it for the reasons `diff` would report. A refusal is not an exception:
|
|
the call returns a result whose `status` is `"failed"`, with the reasons in
|
|
`diff.errors`, so you can decide what to do next. Cherry-picks are not retried.
|
|
|
|
Pass `dry_run=True` in Python, or `true` as the second argument in TypeScript,
|
|
to check without landing anything. `preview.promotedColumns` lists the columns
|
|
the cherry-pick would promote, or did.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Check first; this does not modify main.
|
|
preview = table.branches.cherry_pick("exp", dry_run=True)
|
|
print(preview["status"]) # "ready" when it would land
|
|
print(preview["preview"]["promotedColumns"])
|
|
|
|
# Land it.
|
|
result = table.branches.cherry_pick("exp")
|
|
if result["status"] == "cherryPicked":
|
|
print("landed at main version", result["mainVersionAfter"])
|
|
elif result["status"] == "failed":
|
|
for error in result["diff"]["errors"]:
|
|
print(error["code"], error["message"])
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const branches = await table.branches();
|
|
|
|
// Check first; this does not modify main.
|
|
const preview = await branches.cherryPick("exp", true);
|
|
console.log(preview.status); // "ready" when it would land
|
|
console.log(preview.preview.promotedColumns);
|
|
|
|
// Land it.
|
|
const result = await branches.cherryPick("exp");
|
|
if (result.status === "cherryPicked") {
|
|
console.log("landed at main version", result.mainVersionAfter);
|
|
} else if (result.status === "failed") {
|
|
for (const error of result.diff.errors) {
|
|
console.log(error.code, error.message);
|
|
}
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
`status` is `ready` (a dry run that would land), `cherryPicked` (it landed, and
|
|
`mainVersionAfter` is the new version of `main`), `failed` (refused, or for a
|
|
dry run, would be refused; see `diff.errors`), `notImplemented`, or `unknown`
|
|
for a status the client does not know. A branch that does not exist raises a
|
|
not-found error instead of returning a status.
|
|
|
|
In async Python, await the same calls on an `AsyncTable`:
|
|
`await table.branches.diff("exp")` and
|
|
`await table.branches.cherry_pick("exp", dry_run=True)`. In Rust they are
|
|
`Table::diff_branch` and `Table::cherry_pick`.
|
|
|
|
## Build indexes on a branch {#build-indexes-on-a-branch}
|
|
|
|
One of the most useful things a branch buys you is a safe place to build and
|
|
validate an index without affecting what's in production on the `main` branch.
|
|
Fork a branch, create your vector (ANN) and full-text search (FTS)
|
|
indexes on it, and check recall and latency. Once you have selected a
|
|
configuration, build the corresponding index on `main` through your normal
|
|
deployment workflow. Because the indexes live on the branch, queries against
|
|
`main` never see a half-built index and are never slowed down by the branch's
|
|
build.
|
|
|
|
This pattern is especially valuable on Enterprise deployments, where `main` is
|
|
typically serving production traffic while you tune an index configuration on
|
|
the side. A deployment builds the branch's indexes in the background, so wait
|
|
for them before measuring.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{BranchIndex}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsBranchIndex}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsBranchIndex}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Schema changes such as adding, altering, or dropping columns are branch-scoped in
|
|
the same way, so you can stage and review a larger reshaping of a table before
|
|
applying the same schema operations to `main`.
|
|
|
|
## Branches vs. tags vs. versions {#branches-vs-tags-vs-versions}
|
|
|
|
Now that you're familiar with branches, you can see how they complement the other ways LanceDB
|
|
give you to work with table history. Choose these approaches based on whether you need to
|
|
*label*, *read*, or *write* data at a point in history:
|
|
|
|
| Feature | Writable? | Purpose |
|
|
| --- | --- | --- |
|
|
| **[Branch](#work-with-branches)** | ✅ Yes | Write on top of a point in history without touching `main`. |
|
|
| **[Tag](/management/table#versioning-tag-based-versioning)** | ❌ No | Attach a human-readable label to an existing version; protects it from cleanup. |
|
|
| **[`checkout(version)`](/management/table#versioning-rollback-to-previous-versions)** | ❌ No | Read a historical version of `main` without forking. Read-only until you `restore`. |
|
|
|
|
<Note>
|
|
For linear version history — creating versions, listing them, rolling back, and
|
|
tagging — see the [Versioning and Reproducibility](/management/table#versioning) guide.
|
|
</Note>
|