From b4eae0d5b9ff83926577ed00df36618d1fac940c Mon Sep 17 00:00:00 2001 From: Dan Tasse Date: Wed, 5 Aug 2026 15:09:35 -0400 Subject: [PATCH] feat: update skill to navigate table history --- plugins/lancedb/skills/lancedb/SKILL.md | 3 +- .../lancedb/references/table_history.md | 182 ++++++++++++++++++ 2 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 plugins/lancedb/skills/lancedb/references/table_history.md diff --git a/plugins/lancedb/skills/lancedb/SKILL.md b/plugins/lancedb/skills/lancedb/SKILL.md index 8b8f761c2..5a7641598 100644 --- a/plugins/lancedb/skills/lancedb/SKILL.md +++ b/plugins/lancedb/skills/lancedb/SKILL.md @@ -29,9 +29,10 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks - TypeScript performance guidance: `references/typescript/performance.md` - Column metadata authoring (both SDKs): `references/column_metadata.md` - Branch operations (both SDKs): `references/branch_ops.md` + - Table version history and auditing (both SDKs + REST): `references/table_history.md` - Remote server connection resolution (jobs, raw REST): `references/remote_connect.md` - Job operations REST API (list/describe/cancel/query_events): `references/remote_jobs.md` -4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events). +4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `table_history.md` when the task involves a table's past rather than its present — comparing or diffing versions, auditing what changed since some point in time, finding the version that introduced or dropped a column, reading the table as it was, or tracing a change back to the job behind it. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events). 5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited. 6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads. 7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall. diff --git a/plugins/lancedb/skills/lancedb/references/table_history.md b/plugins/lancedb/skills/lancedb/references/table_history.md new file mode 100644 index 000000000..e38b004ca --- /dev/null +++ b/plugins/lancedb/skills/lancedb/references/table_history.md @@ -0,0 +1,182 @@ +# Table version history + +Every operation that modifies a LanceDB table commits a new **version**, and the whole chain +stays readable. Use this for auditing and forensics on a table: diffing two versions, +answering "what changed since \", finding which version introduced or dropped a column, +reading the table as it was at some point, or tracing a change back to the background job +behind it. + +Version numbers and timestamps are available on local/OSS and remote Enterprise/Cloud tables +through both SDKs. **What changed at a version is only available over REST** — see the +limitation below before you plan an approach. + +## The version model + +- Versions start at 1 and increase by one per commit. Every mutation commits: creating a + table, appending or deleting rows, adding or dropping columns, building an index, editing + column or table metadata. +- A version is a **manifest snapshot, not a diff**. "What changed at version N" is always + derived by comparing N against N-1 — which is what `include_operations` does server-side. +- Branches have their own version chains. Pass the branch explicitly to read one; see + `references/branch_ops.md`. + +## Listing history + +### Through the SDKs — version and timestamp only + +```python +t = db.open_table("products") +t.version # current version number, e.g. 7 +t.list_versions() # [{'version': 1, 'timestamp': datetime(...), 'metadata': {}}, ...] +``` + +```typescript +const t = await db.openTable("products"); +await t.listVersions(); // [{ version, timestamp: Date, metadata }] +``` + +**Limitation worth planning around:** both SDKs return only `version`, `timestamp`, and +`metadata`. Neither tells you *what happened* at a version — no operation name, no row count, +no schema diff. If the task is "what changed", the SDK list alone cannot answer it. Either +use the REST endpoint below, or read the schema at each version and diff them yourself +(N round trips, and still no operation names or row counts). + +### Through REST — the full picture + +`POST {base_url}/v1/table/{table}/version/list`, with the usual `x-api-key` / +`x-lancedb-database` headers. Resolve the connection first — see `references/remote_connect.md`. + +The options are **query parameters, not body fields**. The body is `{}`. + +| query param | effect | +|---|---| +| `include_operations=true` | fill in `operation`, `num_rows`, `added_columns`, `removed_columns` | +| `descending=true` | newest first (ordering is otherwise implementation-defined) | +| `limit=N`, `page_token=...` | paginate | +| `branch=` | that branch's chain instead of main's | + +```bash +curl -s -X POST "{base_url}/v1/table/products/version/list?include_operations=true&descending=true" \ + -H "x-api-key: " -H "x-lancedb-database: " \ + -H "content-type: application/json" -d '{}' +``` + +```json +{ + "versions": [ + { + "version": 14, + "timestamp": "2025-03-11T09:22:07Z", + "timestamp_millis": 1741684927000, + "manifest_path": "...", "manifest_size": 1313, "e_tag": "...", + "metadata": {}, + "operation": "Merge", + "num_rows": 1204, + "added_columns": [{"name": "price_tier", "type": {"type": "int32"}, "nullable": false}] + } + ] +} +``` + +`added_columns` / `removed_columns` are **omitted entirely** when that version changed no +columns — treat a missing key as "no schema change", not as an error. + +## Reading the operation names + +`operation` is the Lance transaction name, not the API you called. The mapping you will +actually see: + +| operation | what someone did | +|---|---| +| `Overwrite` | table created, or fully rewritten (`mode="overwrite"`) | +| `Append` | rows added | +| `Delete` | rows deleted by predicate | +| `Update` | rows updated in place | +| `Merge` | columns added (`add_columns`) or a merge-insert landed | +| `Project` | columns dropped | +| `CreateIndex` | an index was built | +| `UpdateConfig` | table or column metadata changed (e.g. `update_field_metadata`) | +| `Rewrite` | compaction — data rearranged, nothing logically changed | +| `Restore` | the table was rolled back to an earlier version | + +Other names exist for lower-level maintenance. The useful split is +that **data-only operations leave the schema untouched**, so their `added_columns` / +`removed_columns` are empty — that is how you separate a schema change from a data change, +rather than guessing from the operation name alone. + +## Common tasks + +### Diff two versions + +List the range with `include_operations=true` and read the versions **after** the older one, +up to and including the newer one: "what changed between vN and vM" means the commits +vN+1 … vM. Version N's own operation is the baseline — it already happened before the window, +and reporting it as part of the change set is the usual off-by-one here. + +Row-count movement comes from `num_rows`; schema movement from the column diffs. For the full +schema at either end, describe that version (below) rather than reconstructing it. + +### What changed since a point in time + +Compare each version's `timestamp` (ISO-8601 `...Z`) or `timestamp_millis` against the cutoff. + +- Timestamps land at **second resolution**. Commits inside the same second are + indistinguishable in time — order by version number, never by timestamp, when they tie. +- A **schema** change is a version with a non-empty `added_columns` / `removed_columns`. + An **index** change is `operation == "CreateIndex"`. +- Data-only versions (`Append`, `Delete`, `Update`) are neither. They may well fall inside the + window; report them as what they are instead of folding them into a schema-change answer. + +### Find the version that introduced or dropped a column + +Scan the history for the version whose `added_columns` (or `removed_columns`) names it. Note +that **version 1 lists every original column as added**, since it is diffed against an empty +schema — so a base column's "added" version is 1, and only later additions are interesting. + +### Read the table as it was + +```bash +# schema + stats as of version 9 +curl -s -X POST "{base_url}/v1/table/products/describe" \ + -H "x-api-key: " -H "x-lancedb-database: " \ + -H "content-type: application/json" -d '{"version": 9}' +``` + +```python +t.checkout(9) # handle becomes a read-only view pinned at v9 +t.schema # ... the schema as of v9 +t.checkout_latest() # back to tracking the newest version +``` + +`checkout` mutates the handle it is called on, so restore it with `checkout_latest()` (TS: +`checkoutLatest()`) before writing through it again. + +Do **not** confuse `version/describe` with `describe`: `POST /v1/table/{table}/version/describe` +returns manifest facts only (path, size, etag, `timestamp_millis`) and no schema. For the +schema at a version, use `describe` with `{"version": N}`. + +### Index history + +`POST /v1/table/{table}/index/list` returns each index with a `created_at` timestamp in +milliseconds, which is how you date an index without walking the version chain. Cross-check +against the `CreateIndex` versions when you need the version number too. + +### Trace a version back to the job behind it + +Jobs that were started against a specific table version record it in their spec — for example +a `prewarm_page_cache` job's spec carries `table_version`. So the link from history to the job +registry is: find the version, list the jobs on that table, describe them, and match on +`spec.table_version`. Job listing and describing are in `references/remote_jobs.md`. + +## Gotchas + +- `include_operations`, `descending`, `limit`, and `branch` are **query params**. Putting them + in the JSON body does nothing and you silently get the bare response with no operations. +- `include_operations=true` reads a manifest per version, which is why it is opt-in. Pair it + with `limit` on tables with long histories. +- Without `descending=true` the order is implementation-defined — sort by `version` yourself + rather than trusting the response order. +- The SDKs' `list_versions()` / `listVersions()` will not gain the operation fields by passing + extra arguments; the REST endpoint is the only route to them. +- `restore()` (and `POST /v1/table/{table}/restore`) rolls the table back by committing a new + version. It is a **write** — never reach for it during a read-only investigation.