mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 03:58:26 +00:00
chore: consolidate skills into one (#3672)
Consolidating skills so we have only one `lancedb` skill, making it
easier to install and work with, vs. installing and using different
skills for "lancedb-column-metadata", "lancedb-branch-ops", etc.
Also deleted lancedb-connect, because the new monoskill uses the
python/TS APIs so it doesn't need extra handholding to connect to the
REST API.
## does it work?
Test 1: do some column metadata operations with 1. no skills, 2. our
previous baseline lancedb skill, 3. the baseline lancedb skill with the
lancedb-column-metadata skill folded in:
```
┌───────────────────────────┬──────────────────────┬────────────────────┬───────────────────────┐
│ eval │ no-skill │ lancedb (original) │ lancedb2-incl-columns │
├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤
│ 2-add-all-metadata-types │ 2.5/5 · 116s · $0.44 │ 0/5 · 154s · $0.60 │ 5/5 · 58s · $0.26 │
├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤
│ 3-delete-one-metadata-key │ 4/4 · 67s · $0.23 │ 4/4 · 100s · $0.45 │ 4/4 · 45s · $0.20 │
├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤
│ TOTAL (per rep avg) │ 6.5/9 · 183s · $0.66 │ 4/9 · 254s · $1.05 │ 9/9 · 103s · $0.46 │
└───────────────────────────┴──────────────────────┴────────────────────┴───────────────────────┘
```
without column-metadata-specific content, it failed because it wrote
keys like `description` instead of `lancedb:description`. That's pretty
undiscoverable without the skill.
Test 2: do some simple branch operations with 1. no skills, 2. our
previous baseline lancedb skill, 3. the combined skill (in this PR):
```
┌───────────────────────────┬────────────────────┬────────────────────┬────────────────────────────────┐
│ eval │ no-skill │ lancedb (original) │ lancedb3-incl-columns-branches │
├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤
│ 5-create-branch │ 2/2 · 64s · $0.30 │ 2/2 · 57s · $0.34 │ 2/2 · 38s · $0.22 │
├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤
│ 6-delete-branch │ 2/2 · 38s · $0.21 │ 2/2 · 44s · $0.27 │ 2/2 · 40s · $0.22 │
├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤
│ 7-switch-branch-and-write │ 1/2 · 96s · $0.49 │ 1/2 · 106s · $0.58 │ 2/2 · 66s · $0.40 │
├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤
│ TOTAL (per rep avg) │ 5/6 · 199s · $1.00 │ 5/6 · 207s · $1.20 │ 6/6 · 143s · $0.83 │
└───────────────────────────┴────────────────────┴────────────────────┴────────────────────────────────┘
```
Test 3: run everything, with the lancedb (original) skill,
lancedb(original) + all the separate skills, and
lancedb3-incl-columns-branches
```
┌────────────────────────────────┬──────────────────────┬────────────────────────────────┬─────────────────────────────────────────────────────────────────┐
│ eval │ lancedb (original) │ lancedb3-incl-columns-branches │ all-separate (lancedb + connect + column-metadata + branch-ops) │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 1-pick-column-for-image-search │ 2/2 · 141s · $0.67 │ 2/2 · 128s · $0.43 │ 1/2 · 257s · $0.81 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 2-add-all-metadata-types │ 5/5 · 93s · $0.52 │ 5/5 · 54s · $0.30 │ 5/5 · 40s · $0.24 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 3-delete-one-metadata-key │ 4/4 · 84s · $0.44 │ 4/4 · 36s · $0.24 │ 4/4 · 28s · $0.20 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 4-build-index │ 3/3 · 275s · $0.52 │ 3/3 · 84s · $0.39 │ 3/3 · 68s · $0.50 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 5-create-branch │ 2/2 · 39s · $0.31 │ 2/2 · 41s · $0.20 │ 2/2 · 15s · $0.17 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 6-delete-branch │ 2/2 · 59s · $0.25 │ 2/2 · 69s · $0.29 │ 2/2 · 26s · $0.17 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ 7-switch-branch-and-write │ 1/2 · 84s · $0.46 │ 2/2 · 62s · $0.40 │ 2/2 · 27s · $0.23 │
├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤
│ TOTAL │ 19/20 · 774s · $3.17 │ 20/20 · 474s · $2.24 │ 19/20 · 462s · $2.32 │
└────────────────────────────────┴──────────────────────┴────────────────────────────────┴─────────────────────────────────────────────────────────────────┘
```
("lancedb3-incl-columns-branches" is the combined skill in this PR,
all-separate is using the four separate skills.)
For overall performance, it helps to have the specialized skills for
metadata and branching; doesn't really matter whether they're separate
skills or all together. Also doesn't matter much whether it's REST or
Python. So let's merge these skills to make it easier for users.
This commit is contained in:
Dan Tasse
GitHub
@@ -1,145 +0,0 @@
|
||||
---
|
||||
name: lancedb-branch-ops
|
||||
description: >-
|
||||
Manage LanceDB table branches through the REST API: list, create, and delete
|
||||
branches; target schema reads, field-metadata updates, and index creation to a
|
||||
named branch; and verify that branch changes remain isolated from main. Use
|
||||
when a task involves branch lifecycle, an experimental or isolated table
|
||||
version, directing an operation to a non-main branch, or confirming that a
|
||||
mutation did not affect main. This skill also explains that LanceDB has no
|
||||
checkout operation; each request selects its target branch in the request
|
||||
body.
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main.
|
||||
|
||||
## Step 0: Establish the connection
|
||||
|
||||
Use the `lancedb-connect` skill to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`). Skip this only if the connection is already known from the current conversation.
|
||||
|
||||
All examples below use `{base_url}` — substitute the resolved endpoint and include the auth headers on every request.
|
||||
|
||||
## The branch model (important)
|
||||
|
||||
LanceDB branches are named snapshots that diverge from the table's current state at creation time. There is **no checkout command** — you never switch the whole table to a branch. Instead, you **pass `"branch": "<name>"` in the request body** of any operation to target that branch. Omitting the key (or sending an empty body) always targets main.
|
||||
|
||||
`branches/list` returns only non-main branches. Main always exists and is not listed.
|
||||
|
||||
## List branches
|
||||
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/branches/list
|
||||
Content-Type: application/json
|
||||
|
||||
{}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"branches": {
|
||||
"experiment-reindex": {"parentVersion": 1, "createAt": 1782506085, "manifestSize": 1029}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If `branches` is `{}`, the table has no branches besides main.
|
||||
|
||||
## Create a branch
|
||||
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/branches/create
|
||||
Content-Type: application/json
|
||||
|
||||
{"name": "experiment-reindex"}
|
||||
```
|
||||
|
||||
HTTP 200 with `{}` body = success. The branch is created off the table's current state on main.
|
||||
|
||||
Verify by calling `branches/list` and confirming the new name appears.
|
||||
|
||||
## Delete a branch
|
||||
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/branches/delete
|
||||
Content-Type: application/json
|
||||
|
||||
{"name": "stale-2024"}
|
||||
```
|
||||
|
||||
HTTP 200 with `{}` body = success. Only the branch pointer is removed — main and all row data remain intact.
|
||||
|
||||
Verify by calling `branches/list` (name gone) and `describe` with no branch param (main still responds).
|
||||
|
||||
## Operate on a specific branch
|
||||
|
||||
Pass `"branch": "<name>"` in the body of any operation to scope it to that branch:
|
||||
|
||||
**Read schema on a branch:**
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/describe
|
||||
Content-Type: application/json
|
||||
|
||||
{"branch": "wip-branch"}
|
||||
```
|
||||
|
||||
**Write metadata to a branch (not main):**
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/update_field_metadata
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"branch": "wip-branch",
|
||||
"updates": [
|
||||
{
|
||||
"path": "category",
|
||||
"metadata": {"lancedb:description": "Product category label."},
|
||||
"replace": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Build an index on a branch:**
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/create_index
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"branch": "wip-branch",
|
||||
"column": "category",
|
||||
"index_type": "BTREE"
|
||||
}
|
||||
```
|
||||
|
||||
## Verifying isolation
|
||||
|
||||
After writing to a branch, always confirm the change did NOT land on main:
|
||||
|
||||
```bash
|
||||
# Should show the new metadata
|
||||
curl -s -X POST {base_url}/v1/table/{table_id}/describe \
|
||||
-H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
|
||||
-H "content-type: application/json" \
|
||||
-d '{"branch": "wip-branch"}'
|
||||
|
||||
# Should NOT show the new metadata
|
||||
curl -s -X POST {base_url}/v1/table/{table_id}/describe \
|
||||
-H "x-api-key: <key>" -H "x-lancedb-database: <db>" \
|
||||
-H "content-type: application/json" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Goal | Endpoint | Body |
|
||||
|------|----------|------|
|
||||
| List all branches | `branches/list` | `{}` |
|
||||
| Create a branch | `branches/create` | `{"name": "..."}` |
|
||||
| Delete a branch | `branches/delete` | `{"name": "..."}` |
|
||||
| Read schema on branch | `describe` | `{"branch": "..."}` |
|
||||
| Write metadata on branch | `update_field_metadata` | `{"branch": "...", "updates": [...]}` |
|
||||
| Build index on branch | `create_index` | `{"branch": "...", "column": ..., "index_type": ...}` |
|
||||
| Target main (default) | any endpoint | omit `"branch"` key |
|
||||
@@ -1,178 +0,0 @@
|
||||
---
|
||||
name: lancedb-column-metadata
|
||||
description: Column metadata authoring for LanceDB tables via the REST API. This skill is required for tasks like writing field descriptions, setting tags on columns (field_type, model, project_id, version), classifying columns as embeddings vs labels vs eval metrics, or grouping versioned columns into logical families — because it has the API integration needed to read the schema and persist metadata back. Invoke whenever someone wants to document, annotate, tag, or classify what their table columns ARE. Trigger even without an explicit "LanceDB" mention, as long as the context is column-level documentation or tagging for an ML or vector database table.
|
||||
metadata:
|
||||
short-description: Write column descriptions, tags, and logical groupings to a LanceDB table
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This skill authors column-level metadata for a LanceDB table. It connects to a LanceDB deployment over its REST API, inspects the table schema, generates appropriate metadata, and writes it back.
|
||||
|
||||
## Step 0: Establish the connection
|
||||
|
||||
Use the `lancedb-connect` skill (invoke it via the Skill tool) to resolve the base URL and auth headers (`x-api-key`, `x-lancedb-database`) for whichever deployment the user is working against — enterprise/self-hosted or a local dev server. Skip it only if the connection details are already established in the conversation.
|
||||
|
||||
All examples below use `{base_url}` — substitute the resolved endpoint and include the resolved headers on every request.
|
||||
|
||||
## Metadata keys
|
||||
|
||||
All metadata uses namespaced keys:
|
||||
|
||||
| Key | Purpose | Example value |
|
||||
|-----|---------|---------------|
|
||||
| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
|
||||
| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
|
||||
| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
|
||||
|
||||
Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*.
|
||||
|
||||
## Step 1: Resolve the table identifier
|
||||
|
||||
You need:
|
||||
- **Table name** (required) — e.g., `my_table` or `my_namespace.my_table`
|
||||
- **Database name** — ask if not provided and not inferable from context; it goes in the `x-lancedb-database` header, never in the URL path
|
||||
|
||||
The table identifier in the URL path is typically `table_name` for a top-level table, or `namespace$table_name` if the table lives in a namespace. The API accepts a `delimiter` query parameter to parse compound identifiers (default `$`).
|
||||
|
||||
## Step 2: Describe the table
|
||||
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/describe
|
||||
Content-Type: application/json
|
||||
|
||||
{}
|
||||
```
|
||||
|
||||
The response contains `schema.fields` — an array of field objects:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": {
|
||||
"fields": [
|
||||
{
|
||||
"name": "clip_embedding_v3",
|
||||
"type": { "type": "FixedSizeList", "fields": [...], "listSize": 768 },
|
||||
"nullable": true,
|
||||
"metadata": { "lancedb:description": "..." }
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each field has:
|
||||
- `name` — field name
|
||||
- `type` — Arrow data type (check `type.type` for the type string)
|
||||
- `nullable` — boolean
|
||||
- `metadata` — existing key-value metadata (read this before writing to avoid redundant updates)
|
||||
|
||||
For struct/nested fields, recurse into `type.fields` and represent them as dot-notation paths (e.g., `parent.child`).
|
||||
|
||||
If the user hasn't specified which columns to update, work with all columns.
|
||||
|
||||
## Step 3: Generate metadata
|
||||
|
||||
Decide what to generate based on the user's request.
|
||||
|
||||
### Writing descriptions (`lancedb:description`)
|
||||
|
||||
Base descriptions on:
|
||||
- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
|
||||
- User-supplied context (upstream pipeline, sample values, domain knowledge)
|
||||
- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
|
||||
|
||||
Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
|
||||
|
||||
### Tagging columns (`lancedb:tag:<name>`)
|
||||
|
||||
Choose tag key names that match what the user asked to annotate. Common patterns:
|
||||
|
||||
- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
|
||||
- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
|
||||
- Project affiliation → `lancedb:tag:project_id: "<name>"`
|
||||
- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
|
||||
|
||||
Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
|
||||
|
||||
Multiple tags on the same column are fine — each is a separate key.
|
||||
|
||||
### Grouping into logical columns (`lancedb:logical-column`)
|
||||
|
||||
Look for naming patterns across columns:
|
||||
- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
|
||||
- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
|
||||
|
||||
Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
|
||||
|
||||
## Step 4: Write the metadata
|
||||
|
||||
```http
|
||||
POST {base_url}/v1/table/{table_id}/update_field_metadata
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"updates": [
|
||||
{
|
||||
"path": "clip_v3",
|
||||
"metadata": {
|
||||
"lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v3",
|
||||
"lancedb:tag:latest": "true",
|
||||
"lancedb:logical-column": "clip"
|
||||
},
|
||||
"replace": false
|
||||
},
|
||||
{
|
||||
"path": "clip_v2",
|
||||
"metadata": {
|
||||
"lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v2",
|
||||
"lancedb:logical-column": "clip"
|
||||
},
|
||||
"replace": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- **Use `"replace": false`** (merge) by default — this preserves existing metadata the user didn't ask to change
|
||||
- Use `"replace": true` only if the user explicitly asks to overwrite all existing metadata on a column
|
||||
- Set a value to `null` to delete a specific key
|
||||
- Batch all updates in a single request when possible
|
||||
|
||||
The response includes `version` (new table version) and `fields` (the updated metadata per field).
|
||||
|
||||
## Step 5: Confirm
|
||||
|
||||
Report back:
|
||||
- Which columns were updated and what was written
|
||||
- The new table version number
|
||||
- Any columns skipped (e.g., already had up-to-date metadata)
|
||||
|
||||
---
|
||||
|
||||
## Quick examples
|
||||
|
||||
**"Write descriptions for all columns in the `product_embeddings` table"**
|
||||
1. POST `/v1/table/product_embeddings/describe` → get all fields
|
||||
2. Generate a `lancedb:description` for each column based on name + type
|
||||
3. POST `update_field_metadata` with descriptions
|
||||
4. Report
|
||||
|
||||
**"Tag the columns in `model_outputs` with their field type and model"**
|
||||
1. Describe `model_outputs`
|
||||
2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
|
||||
3. POST `update_field_metadata`
|
||||
4. Report
|
||||
|
||||
**"Group the feature columns in `training_features` into logical families and mark the latest version"**
|
||||
1. Describe the table
|
||||
2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
|
||||
3. POST `update_field_metadata`
|
||||
4. Show the grouping
|
||||
@@ -1,42 +0,0 @@
|
||||
---
|
||||
name: lancedb-connect
|
||||
description: Resolve how to connect to a LanceDB deployment over the REST API — figure out the base URL, API key, and database header. Use this before making any REST requests to a LanceDB table, whenever the endpoint or auth setup is not already known. Also useful on its own when someone asks how to connect, authenticate, or curl their LanceDB instance.
|
||||
metadata:
|
||||
short-description: Resolve the base URL and auth headers for a LanceDB deployment
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Produce two things every REST request needs:
|
||||
|
||||
1. **Base URL** — the endpoint
|
||||
2. **Headers** — `x-api-key`, and usually `x-lancedb-database`
|
||||
|
||||
## Resolution steps
|
||||
|
||||
1. If the user already gave a URL and API key (or said which environment they're working against), use that.
|
||||
2. Otherwise, look for credentials already available in the environment:
|
||||
- Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
|
||||
- A LanceDB endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
|
||||
3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
|
||||
|
||||
## Validating the connection
|
||||
|
||||
Make a cheap authenticated request and check the status:
|
||||
|
||||
```bash
|
||||
curl -s -w "\n%{http_code}" "{base_url}/v1/table/?limit=1" \
|
||||
-H "x-api-key: <key>" \
|
||||
-H "x-lancedb-database: <database>"
|
||||
```
|
||||
|
||||
- `200` — connection, key, and database header all good
|
||||
- `401` — API key missing or wrong
|
||||
- `400` mentioning a database header — this deployment expects `x-lancedb-database`
|
||||
|
||||
## Non-REST equivalents
|
||||
|
||||
If the caller would rather use the SDK or CLI than raw REST, the same credentials work:
|
||||
|
||||
- Python SDK: `lancedb.connect("db://<database>", api_key="<key>", host_override="<base_url>")`
|
||||
- `lancedb` CLI: a `[profiles.<name>]` entry in `~/.lancedb/config.toml` with `http_server_url`, `api_key`, `database`
|
||||
@@ -27,7 +27,9 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
|
||||
- TypeScript patterns: `references/typescript/patterns.md`
|
||||
- TypeScript API quick reference: `references/typescript/api_reference.md`
|
||||
- TypeScript performance guidance: `references/typescript/performance.md`
|
||||
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.
|
||||
- Column metadata authoring (both SDKs): `references/column_metadata.md`
|
||||
- Branch operations (both SDKs): `references/branch_ops.md`
|
||||
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main.
|
||||
5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
|
||||
6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
|
||||
7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Branch Operations
|
||||
|
||||
Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main. Use for branch lifecycle tasks, experimental/isolated table versions, targeting an operation at a non-main branch, or confirming a mutation did not affect main.
|
||||
|
||||
Works on local/OSS and remote Enterprise/Cloud tables.
|
||||
|
||||
## The branch model (important)
|
||||
|
||||
Branches are isolated, writable lines of history forked from another branch (or a specific version). Writes on a branch never affect `main`.
|
||||
|
||||
There is **no global "switch branch" state** — you never repoint the whole table at a branch. Instead, **operations are scoped by which table handle you use**:
|
||||
|
||||
- The handle you got from `open_table(name)` / `openTable(name)` targets `main`.
|
||||
- `branches.create(...)` and `branches.checkout(...)` return a **new table handle scoped to that branch**. Every read/write on that handle (add, update, `update_field_metadata`, `create_index`, search, …) lands on the branch.
|
||||
- The original main handle is unaffected — keep it around to verify isolation.
|
||||
|
||||
`branches.list()` returns only non-main branches. Main always exists and is not listed.
|
||||
|
||||
## Python
|
||||
|
||||
`table.branches` is a property returning the branch manager; `table.current_branch()` tells you what a handle is scoped to (`None` = main).
|
||||
|
||||
```python
|
||||
table = db.open_table("products") # scoped to main
|
||||
|
||||
# list — dict of name -> metadata (parent_branch, parent_version, ...); {} = only main
|
||||
table.branches.list()
|
||||
|
||||
# create: forks from main by default and returns a handle scoped to the new branch
|
||||
exp = table.branches.create("experiment-reindex")
|
||||
exp = table.branches.create("exp2", from_ref="main", from_version=None) # optional fork point
|
||||
|
||||
# checkout an existing branch -> branch-scoped handle
|
||||
wip = table.branches.checkout("wip-branch")
|
||||
# with version= it pins to that version (read-only detached view); omit to track latest, writable
|
||||
|
||||
# operate on the branch simply by using its handle
|
||||
wip.update_field_metadata(
|
||||
{"path": "category", "metadata": {"lancedb:description": "Product category label."}}
|
||||
)
|
||||
wip.create_scalar_index("category")
|
||||
|
||||
# delete: removes only the branch pointer; main and row data remain intact
|
||||
table.branches.delete("stale-2024")
|
||||
|
||||
# alternatively, open a branch handle directly from the connection
|
||||
wip = db.open_table("products", branch="wip-branch")
|
||||
|
||||
exp.current_branch() # "experiment-reindex"
|
||||
table.current_branch() # None (main)
|
||||
```
|
||||
|
||||
Async: same shape — `table.branches` returns `AsyncBranches`; `await table.branches.create(...)` etc.
|
||||
|
||||
## TypeScript
|
||||
|
||||
`table.branches()` is an **async method** returning the `Branches` manager; `table.currentBranch()` returns the scoped branch or `null` for main.
|
||||
|
||||
```typescript
|
||||
const table = await db.openTable("products"); // scoped to main
|
||||
const branches = await table.branches();
|
||||
|
||||
// list — Record<string, BranchContents>; {} = only main
|
||||
await branches.list();
|
||||
|
||||
// create: forks from main by default, returns a Table scoped to the new branch
|
||||
const exp = await branches.create("experiment-reindex");
|
||||
const exp2 = await branches.create("exp2", "main" /* fromRef */, undefined /* fromVersion */);
|
||||
|
||||
// checkout an existing branch -> branch-scoped Table
|
||||
const wip = await branches.checkout("wip-branch");
|
||||
// with a version arg it pins (read-only detached view); omit to track latest, writable
|
||||
|
||||
// operate on the branch simply by using its handle
|
||||
await wip.updateFieldMetadata([
|
||||
{ path: "category", metadata: { "lancedb:description": "Product category label." } },
|
||||
]);
|
||||
await wip.createIndex("category");
|
||||
|
||||
// delete: removes only the branch pointer; main and row data remain intact
|
||||
await branches.delete("stale-2024");
|
||||
|
||||
// alternatively, open a branch handle directly from the connection
|
||||
const wip2 = await db.openTable("products", { branch: "wip-branch" });
|
||||
|
||||
exp.currentBranch(); // "experiment-reindex"
|
||||
table.currentBranch(); // null (main)
|
||||
```
|
||||
|
||||
## Verifying isolation
|
||||
|
||||
After writing to a branch, confirm the change did NOT land on main by reading through both handles:
|
||||
|
||||
```python
|
||||
wip = table.branches.checkout("wip-branch")
|
||||
wip.update_field_metadata({"path": "category", "metadata": {"lancedb:description": "..."}})
|
||||
|
||||
assert b"lancedb:description" in (wip.schema.field("category").metadata or {})
|
||||
assert b"lancedb:description" not in (table.schema.field("category").metadata or {}) # main untouched
|
||||
```
|
||||
|
||||
Two handles on the same branch see each other's writes (e.g. `table.branches.create("exp")` and `db.open_table(name, branch="exp")`); main stays isolated.
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Goal | Python | TypeScript |
|
||||
|------|--------|------------|
|
||||
| List branches (non-main) | `table.branches.list()` | `await (await table.branches()).list()` |
|
||||
| Create branch (off main) | `table.branches.create(name)` → branch handle | `await branches.create(name)` → branch `Table` |
|
||||
| Create from a fork point | `table.branches.create(name, from_ref=..., from_version=...)` | `await branches.create(name, fromRef, fromVersion)` |
|
||||
| Get a branch handle | `table.branches.checkout(name)` or `db.open_table(t, branch=name)` | `await branches.checkout(name)` or `await db.openTable(t, { branch: name })` |
|
||||
| Pin to a branch version (read-only) | `table.branches.checkout(name, version=v)` | `await branches.checkout(name, v)` |
|
||||
| Delete branch | `table.branches.delete(name)` | `await branches.delete(name)` |
|
||||
| Which branch is this handle on? | `table.current_branch()` (`None` = main) | `table.currentBranch()` (`null` = main) |
|
||||
| Target main | use the original (non-branch) handle | use the original (non-branch) handle |
|
||||
|
||||
Branch names must be non-empty; empty names raise a validation error.
|
||||
@@ -0,0 +1,183 @@
|
||||
# Column Metadata Authoring
|
||||
|
||||
Write column-level descriptions, tags, and logical groupings onto a LanceDB table's schema. Use this when the user wants to document, annotate, tag, or classify what their table columns ARE (embeddings vs labels vs eval metrics, model provenance, version families, etc.).
|
||||
|
||||
Works on local/OSS and remote Enterprise/Cloud tables alike — read the schema through the table handle, write through `update_field_metadata` (Python) / `updateFieldMetadata` (TypeScript).
|
||||
|
||||
## Metadata key conventions
|
||||
|
||||
All metadata uses namespaced keys:
|
||||
|
||||
| Key | Purpose | Example value |
|
||||
|-----|---------|---------------|
|
||||
| `lancedb:description` | Human-readable explanation of what the column contains | `"CLIP ViT-L/14 image embedding, L2-normalized (768-dim)"` |
|
||||
| `lancedb:tag:<name>` | Flexible key-value tag; the suffix names the tag category | `lancedb:tag:field_type: "embedding"`, `lancedb:tag:model: "clip"`, `lancedb:tag:project_id: "foo"` |
|
||||
| `lancedb:logical-column` | Logical group/family this column belongs to | `"clip_features"` |
|
||||
|
||||
Tags are open-ended — use whatever key suffix and value make sense given the user's intent. The tag suffix should describe *what is being classified* (e.g., `field_type`, `model`, `project_id`) and the value describes *how*. Multiple tags on the same column are fine — each is a separate key. All values are strings.
|
||||
|
||||
## Step 1: Read the schema and existing metadata
|
||||
|
||||
Read existing metadata before writing, to avoid redundant updates.
|
||||
|
||||
Python — `table.schema` (sync property; async: `await table.schema()`) returns a `pyarrow.Schema`. **Arrow field metadata is bytes-keyed in Python**:
|
||||
|
||||
```python
|
||||
schema = table.schema
|
||||
for field in schema:
|
||||
meta = field.metadata or {} # dict[bytes, bytes], e.g. {b"lancedb:description": b"..."}
|
||||
print(field.name, field.type, field.nullable, meta)
|
||||
```
|
||||
|
||||
TypeScript — `await table.schema()` returns an Arrow `Schema`; field metadata is a `Map<string, string>`:
|
||||
|
||||
```typescript
|
||||
const schema = await table.schema();
|
||||
for (const field of schema.fields) {
|
||||
console.log(field.name, field.type, field.nullable, field.metadata); // Map
|
||||
// field.metadata.get("lancedb:description")
|
||||
}
|
||||
```
|
||||
|
||||
For struct/nested fields, recurse into the field's children and address them as dot-paths (e.g., `parent.child`).
|
||||
|
||||
If the user hasn't specified which columns to update, work with all columns.
|
||||
|
||||
## Step 2: Generate metadata
|
||||
|
||||
Decide what to generate based on the user's request.
|
||||
|
||||
### Descriptions (`lancedb:description`)
|
||||
|
||||
Base descriptions on:
|
||||
- The column name and Arrow type (e.g., `FixedSizeList` of floats → likely an embedding)
|
||||
- User-supplied context (upstream pipeline, sample values, domain knowledge)
|
||||
- Name patterns: `_embedding`/`_vec`/`_embed` → vector; `_label`/`_class` → label; `_score`/`_eval`/`_metric` → evaluation metric
|
||||
|
||||
Be specific and concise. Good: `"Sentence-BERT embedding of the query text (768-dim)."` Not: `"An embedding column."`
|
||||
|
||||
### Tags (`lancedb:tag:<name>`)
|
||||
|
||||
Choose tag key names that match what the user asked to annotate. Common patterns:
|
||||
|
||||
- Semantic field type → `lancedb:tag:field_type: "embedding"` / `"text"` / `"image"` / `"label"` / `"eval"` / `"id"` / `"metadata"`
|
||||
- Model or source → `lancedb:tag:model: "clip"` / `"bert"` / `"vit"`
|
||||
- Project affiliation → `lancedb:tag:project_id: "<name>"`
|
||||
- Version → `lancedb:tag:version: "v3"` (and `lancedb:tag:latest: "true"` for the newest)
|
||||
|
||||
Use Arrow type as a hint: `FixedSizeList` + float → embedding; `Utf8`/`LargeUtf8` → text; `Binary` → image or blob.
|
||||
|
||||
### Logical groupings (`lancedb:logical-column`)
|
||||
|
||||
Look for naming patterns across columns:
|
||||
- `clip_v1`, `clip_v2`, `clip_v3` → logical column `"clip"`, latest is `v3`
|
||||
- `text_embed_20240101`, `text_embed_20240601` → logical column `"text_embed"`, latest is the most recent date suffix
|
||||
|
||||
Write `lancedb:logical-column` on all members of a group. Mark the newest with `lancedb:tag:latest: "true"` (in addition to its version tag).
|
||||
|
||||
## Step 3: Write the metadata
|
||||
|
||||
Each update names a field by dot-path and carries a metadata map. Semantics (identical in both SDKs):
|
||||
|
||||
- **Merge by default** (`replace` omitted/false) — preserves existing metadata the user didn't ask to change
|
||||
- `replace: true` swaps the field's entire metadata map — only if the user explicitly asks to overwrite
|
||||
- A value of `None`/`null` deletes that specific key
|
||||
- Batch all field updates into a single call when possible
|
||||
- Returns the new table version
|
||||
|
||||
Python (sync and async take one dict per field, as varargs):
|
||||
|
||||
```python
|
||||
res = table.update_field_metadata(
|
||||
{
|
||||
"path": "clip_v3",
|
||||
"metadata": {
|
||||
"lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v3",
|
||||
"lancedb:tag:latest": "true",
|
||||
"lancedb:logical-column": "clip",
|
||||
},
|
||||
},
|
||||
{
|
||||
"path": "clip_v2",
|
||||
"metadata": {
|
||||
"lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v2",
|
||||
"lancedb:logical-column": "clip",
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res.version) # new table version
|
||||
|
||||
# merge semantics: add a key, delete one via None, keep the rest
|
||||
table.update_field_metadata(
|
||||
{"path": "clip_v2", "metadata": {"lancedb:tag:archived": "true", "lancedb:tag:latest": None}}
|
||||
)
|
||||
```
|
||||
|
||||
(`replace_field_metadata` is deprecated — use `update_field_metadata`.)
|
||||
|
||||
TypeScript (takes an array of `FieldMetadataUpdate`):
|
||||
|
||||
```typescript
|
||||
const res = await table.updateFieldMetadata([
|
||||
{
|
||||
path: "clip_v3",
|
||||
metadata: {
|
||||
"lancedb:description": "CLIP ViT-L/14 image embedding, L2-normalized (1024-dim).",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v3",
|
||||
"lancedb:tag:latest": "true",
|
||||
"lancedb:logical-column": "clip",
|
||||
},
|
||||
},
|
||||
{
|
||||
path: "clip_v2",
|
||||
metadata: {
|
||||
"lancedb:description": "CLIP ViT-B/32 image embedding (768-dim), superseded by v3.",
|
||||
"lancedb:tag:field_type": "embedding",
|
||||
"lancedb:tag:model": "clip",
|
||||
"lancedb:tag:version": "v2",
|
||||
"lancedb:logical-column": "clip",
|
||||
},
|
||||
},
|
||||
]);
|
||||
console.log(res.version); // new table version
|
||||
|
||||
// merge semantics: add a key, delete one via null, keep the rest
|
||||
await table.updateFieldMetadata([
|
||||
{ path: "clip_v2", metadata: { "lancedb:tag:archived": "true", "lancedb:tag:latest": null } },
|
||||
]);
|
||||
```
|
||||
|
||||
## Step 4: Confirm
|
||||
|
||||
Report back:
|
||||
- Which columns were updated and what was written
|
||||
- The new table version number (from the result)
|
||||
- Any columns skipped (e.g., already had up-to-date metadata)
|
||||
|
||||
## Quick examples
|
||||
|
||||
**"Write descriptions for all columns in the `product_embeddings` table"**
|
||||
1. Read `table.schema` → all fields + existing metadata
|
||||
2. Generate a `lancedb:description` for each column based on name + type
|
||||
3. One `update_field_metadata` call with all descriptions
|
||||
4. Report
|
||||
|
||||
**"Tag the columns in `model_outputs` with their field type and model"**
|
||||
1. Read the schema
|
||||
2. For each field, classify by name + Arrow type → set `lancedb:tag:field_type` and `lancedb:tag:model` where applicable
|
||||
3. Write in one batched call
|
||||
4. Report
|
||||
|
||||
**"Group the feature columns in `training_features` into logical families and mark the latest version"**
|
||||
1. Read the schema
|
||||
2. Find version patterns → assign `lancedb:logical-column` and `lancedb:tag:version`; mark newest with `lancedb:tag:latest: "true"`
|
||||
3. Write in one batched call
|
||||
4. Show the grouping
|
||||
@@ -96,6 +96,32 @@ print(table.index_stats("vector_idx"))
|
||||
|
||||
Use these before changing indexes or search tuning.
|
||||
|
||||
## Column (Field) Metadata
|
||||
|
||||
```python
|
||||
schema = table.schema # sync property; async: await table.schema()
|
||||
meta = schema.field("category").metadata # dict[bytes, bytes] — Arrow metadata is bytes-keyed
|
||||
res = table.update_field_metadata( # varargs: one dict per field; works local + remote
|
||||
{"path": "category", "metadata": {"lancedb:description": "...", "lancedb:tag:field_type": "label"}}
|
||||
)
|
||||
res.version # new table version
|
||||
```
|
||||
|
||||
Merges by default; a `None` value deletes that key; `"replace": True` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). `replace_field_metadata` is deprecated. See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
|
||||
|
||||
## Branches
|
||||
|
||||
```python
|
||||
table.branches.list() # non-main branches; {} = only main
|
||||
exp = table.branches.create("exp") # fork off main -> handle scoped to the branch
|
||||
wip = table.branches.checkout("wip") # existing branch -> scoped handle (version= pins read-only)
|
||||
wip = db.open_table("t", branch="wip") # or open scoped directly
|
||||
table.branches.delete("stale") # removes only the branch pointer
|
||||
table.current_branch() # None = main
|
||||
```
|
||||
|
||||
There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
|
||||
|
||||
## Maintenance
|
||||
|
||||
```python
|
||||
|
||||
@@ -69,6 +69,33 @@ console.log(await table.indexStats("vector_idx"));
|
||||
|
||||
Use these before changing indexes or search tuning.
|
||||
|
||||
## Column (Field) Metadata
|
||||
|
||||
```typescript
|
||||
const schema = await table.schema();
|
||||
const meta = schema.fields.find((f) => f.name === "category")?.metadata; // Map<string, string>
|
||||
const res = await table.updateFieldMetadata([
|
||||
{ path: "category", metadata: { "lancedb:description": "...", "lancedb:tag:field_type": "label" } },
|
||||
]);
|
||||
res.version; // new table version
|
||||
```
|
||||
|
||||
Merges by default; a `null` value deletes that key; `replace: true` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
|
||||
|
||||
## Branches
|
||||
|
||||
```typescript
|
||||
const branches = await table.branches(); // async manager
|
||||
await branches.list(); // non-main branches; {} = only main
|
||||
const exp = await branches.create("exp"); // fork off main -> Table scoped to the branch
|
||||
const wip = await branches.checkout("wip"); // existing branch -> scoped Table (version arg pins read-only)
|
||||
const wip2 = await db.openTable("t", { branch: "wip" }); // or open scoped directly
|
||||
await branches.delete("stale"); // removes only the branch pointer
|
||||
table.currentBranch(); // null = main
|
||||
```
|
||||
|
||||
There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
|
||||
|
||||
## Maintenance
|
||||
|
||||
```typescript
|
||||
|
||||
Reference in New Issue
Block a user