rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-08-22 05:58:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: remove unnecessary skill references (#3977)

Browse Source 
Background: if we keep adding stuff to the lancedb skill that repeats
other knowledge, we're basically creating a whole new docs site, which
means one more thing that can get out of date. Worse, if it gets out of
date, it will tell agents to do the wrong thing.

These files were added without a ton of analysis of whether they'd be
improving agent performance at all. It looks like they don't really:
<img width="644" height="90" alt="Screenshot 2026-08-20 at 5 21 03 PM"
src="https://github.com/user-attachments/assets/44e60436-b7ad-498b-8e73-0181385c7c60"
/>
(top run is without these docs, bottom run is with them - arguably these
docs might even make the agent a little slower! that's probably noise
though; I'd just say at least they're unnecessary.)

So this PR just removes them. We'll more judiciously add bits we need
and/or point to preexisting docs, to avoid duplication.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dan Tasse
2026-08-20 18:03:24 -04:00
committed by GitHub
parent 5c1b44020a
commit e517ba5205
7 changed files with 23 additions and 733 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/lancedb/skills/lancedb/SKILL.md
+23 -8
View File
@@ -20,18 +20,16 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
1. Identify the SDK: Python, TypeScript, or both.
2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path. If the task involves jobs in any way (listing, inspecting, creating, or canceling jobs), it is always the remote path and requires a remote server connection — see "Connecting to the LanceDB remote server" below before doing anything else.
3. Read the matching language branch before writing or changing code:
- Python patterns: `references/python/patterns.md`
- Python API quick reference: `references/python/api_reference.md`
- Python performance guidance: `references/python/performance.md`
- TypeScript patterns: `references/typescript/patterns.md`
- TypeScript API quick reference: `references/typescript/api_reference.md`
- TypeScript performance guidance: `references/typescript/performance.md`
3. Read the matching topic reference before writing or changing code:
- Column metadata authoring (both SDKs): `references/column_metadata.md`
- Branch operations (both SDKs): `references/branch_ops.md`
- Remote server connection resolution (jobs, raw REST): `references/remote_connect.md`
- Job operations REST API (list/describe/cancel/query_events): `references/remote_jobs.md`
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events).
There is no bundled per-language guide. For exact method names, signatures, and options, look them up in the canonical sources instead of relying on memory:
- Python: `docs/src/python/python.md` (the hand-maintained API reference) and the source under `python/python/lancedb/` when working inside the LanceDB repo; otherwise <https://lancedb.github.io/lancedb/python/python/>.
- TypeScript: the generated typedoc under `docs/src/js/` and the source under `nodejs/lancedb/` when working inside the LanceDB repo; otherwise <https://lancedb.github.io/lancedb/js/globals/>.
4. Apply the SDK invariants in "Per-SDK Invariants" below. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events).
5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
@@ -54,6 +52,23 @@ The unsafe pattern is table-level or unbounded collection, plus local-only datas
- Python: `table.to_pandas()`, `table.to_arrow()`, `table.to_polars()`; `table.to_lance()` is local/OSS-only dataset access, not materialization
- TypeScript: `await table.toArrow()`, `await table.query().toArray()` without `limit()`
## Per-SDK Invariants
Python:
- Result collectors: default to `.to_list()` (plain dicts, no extra dependency) or `.to_arrow()` (PyArrow ships with LanceDB). Use `.to_pandas()` / `.to_polars()` only when the project already declares that dependency — do not assume pandas or polars is installed.
- Plain scans differ by client: the sync client has no `.query()` method — use `table.search()` with no argument; the async client uses `await async_table.query()`.
TypeScript:
- Collect bounded results with `.toArray()` (objects) or `.toArrow()` (Arrow) after `select()` and `limit()`.
- For large reads, stream batches instead of collecting: `for await (const batch of table.query().where(...).select(...).limit(...)) { ... }`.
Both SDKs:
- Ingest in bulk or in batches of thousands of rows; never write per-row in a loop — each write creates a version and fragment, slowing ingestion and later queries.
- Build a vector index once brute-force search is too slow (rule of thumb: beyond roughly 100K vectors locally), and scalar indexes for filtered columns and merge/upsert keys. Use index defaults unless the task states recall/latency requirements.
## Enterprise: never drop-then-reuse the same table name
LanceDB Enterprise/Cloud splits a **control plane** (DDL: create/drop/rename) from a **data plane** (query nodes that serve reads). Query nodes cache the resolved dataset for a table name for up to `table_cache_ttl` — **default 300 seconds (5 minutes)**. After you drop or overwrite a table, the control plane updates immediately but the data plane keeps serving the *old* dataset until that cache entry expires. During the window the two planes disagree.
plugins/lancedb/skills/lancedb/references/python/api_reference.md
-138
View File
@@ -1,138 +0,0 @@
# Python API Reference
Quick method reference for Python LanceDB code. Cross-check source for non-trivial claims.
## Connect
If you're connecting to a remote database, use this:
```python
import lancedb
db = lancedb.connect("db://my-db", api_key=api_key, host_override=host_override) # remote
```
(values may be found in LANCEDB_API_KEY and LANCEDB_HOST_OVERRIDE, either in env vars or a .env file)
If you're connecting to a local table using OSS LanceDB, use this:
```python
db = lancedb.connect("./camelot-db") # local/OSS
```
If you're not sure which, or if you can't find the api_key or host_override params, ask the user.
**Place the local database directory next to the script/entrypoint that opens it** (i.e. resolve the path relative to the script, `Path(__file__).parent / "camelot-db"`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.
**Do not name the directory `lancedb`** (e.g. `./lancedb`, `./data/lancedb`). It collides with the imported `lancedb` package name, which is confusing to read and easy to shadow in scripts. Give it a name derived from the repo or dataset with a clear prefix/suffix — for example `./<dataset>-db`, `./<repo>_lancedb`, or `./vectordb`.
Async:
```python
db = await lancedb.connect_async("./camelot-db")
```
## Table Reads
| Task | Preferred API |
| --- | --- |
| Vector search | `table.search(query_vector).limit(k)` |
| Full scan with filters/projection (sync) | `table.search().where(...).select(...).limit(...)` |
| Full scan with filters/projection (async) | `table.query().where(...).select(...).limit(...)` |
| Filter | `.where("col > 10")` |
| Projection | `.select(["id", "text"])` |
| Bound result count | `.limit(20)` |
| Collect bounded result as Python objects (default, no extra deps) | `.to_list()` on query/search result |
| Collect bounded result as Arrow (default, `pyarrow` always available) | `.to_arrow()` on query/search result |
| Collect bounded result as pandas (only if project uses pandas) | `.to_pandas()` on query/search result |
| Collect bounded result as Polars (only if project uses polars) | `.to_polars()` on query/search result |
## Sync vs Async Scan API
The plain-scan entry point differs between the sync and async clients. **Verified against `lancedb` 0.34.0** — re-check if the pinned version changes:
- **Sync** (`lancedb.connect(...)`): the table has **no `.query()` method**. Use `.search()` with no argument for a plain scan; it returns a query builder that supports `.where()`, `.select()`, `.limit()`, and the `.to_list()` / `.to_arrow()` / `.to_pandas()` / `.to_polars()` collectors.
```python
rows = table.search().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
```
- **Async** (`lancedb.connect_async(...)`): the table has **both** `.query()` and `.search()`. Use `.query()` for a plain scan.
```python
rows = await async_table.query().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
```
Do not call `table.query()` on a sync table — it raises `AttributeError`.
## Local vs Remote Table Methods
| API | Local table | Remote table | Agent guidance |
| --- | --- | --- | --- |
| `table.search(...)` | Yes | Yes | Preferred read path (sync + async) |
| `table.query()` | Async only | Async only | Sync scan path is `table.search()`; `.query()` is the async scan builder |
| `table.to_pandas()` | Yes | No / unsafe for portability | Avoid in portable code |
| `table.to_arrow()` | Yes | No / unsafe for portability | Avoid in portable code |
| `table.to_polars()` | Yes | No / unsafe for portability | Avoid in portable code |
| `table.to_lance()` | Yes | No | Local/OSS escape hatch only |
## Indexes
Use `create_index(...)` for vector indexes and modern index configs. Use scalar indexes for filtered or merge keys.
Common calls:
```python
table.create_index("vector")
table.create_scalar_index("status")
table.create_fts_index("text")
```
Check source docs before specifying advanced index config names or parameters.
## Filtering And Recall Knobs
```python
table.search(query_vector).where("status = 'ready'") # pre-filter by default
table.search(query_vector).where("status = 'ready'", prefilter=False)
table.search(query_vector).limit(10).refine_factor(20)
table.search(query_vector).limit(10).nprobes(50)
```
Use post-filtering only when fewer than `limit` results are acceptable.
## Diagnostics
```python
print(table.search(query_vector).where("year > 2000").limit(10).analyze_plan())
print(table.index_stats("vector_idx"))
```
Use these before changing indexes or search tuning.
## Column (Field) Metadata
```python
schema = table.schema # sync property; async: await table.schema()
meta = schema.field("category").metadata # dict[bytes, bytes] — Arrow metadata is bytes-keyed
res = table.update_field_metadata( # varargs: one dict per field; works local + remote
{"path": "category", "metadata": {"lancedb:description": "...", "lancedb:tag:field_type": "label"}}
)
res.version # new table version
```
Merges by default; a `None` value deletes that key; `"replace": True` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). `replace_field_metadata` is deprecated. See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
## Branches
```python
table.branches.list() # non-main branches; {} = only main
exp = table.branches.create("exp") # fork off main -> handle scoped to the branch
wip = table.branches.checkout("wip") # existing branch -> scoped handle (version= pins read-only)
wip = db.open_table("t", branch="wip") # or open scoped directly
table.branches.delete("stale") # removes only the branch pointer
table.current_branch() # None = main
```
There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
## Maintenance
```python
table.optimize()
```
Call this after every successful local/OSS ingestion. It handles compaction, cleanup of old versions according to retention, and index optimization. Do not add this for LanceDB Enterprise/Cloud remote tables; Enterprise handles compaction and cleanup automatically from cluster configuration.
plugins/lancedb/skills/lancedb/references/python/patterns.md
-173
View File
@@ -1,173 +0,0 @@
# Python Patterns
Use these patterns when writing Python code with `lancedb`.
## Before Writing Code
Choose the output type from what the project actually depends on. **Do not assume `pandas` or `polars` is installed** — they are heavy dependencies that many LanceDB projects do not use. `pyarrow`, by contrast, ships as a LanceDB dependency and is always available, so it is a safe default to lean on.
Default output (after applying `select()` and `limit()`):
- **Python objects**: `.to_list()` — a list of dicts, no extra dependencies. Prefer this for scripts, examples, and agent-generated code unless there is a reason to do otherwise.
- **PyArrow**: `.to_arrow()` — a `pyarrow.Table`, when the surrounding code is Arrow-native or you need columnar/zero-copy handoff.
Only reach for a DataFrame when the project *already* declares that dependency:
- Pandas projects (pandas in `pyproject.toml`/requirements): `.to_pandas()`.
- Polars projects (polars declared): `.to_polars()`.
If unsure, check the dependency manifest or the imports in surrounding files. When in doubt, use `.to_list()` or `.to_arrow()`.
## Schema Design and Validation
Favor `LanceModel` and Pydantic validation for Python schemas. They keep field
types readable, validate source records before a write, and map directly to a
LanceDB schema. Use `Vector(dimension)` for fixed-size vectors:
```python
from lancedb.pydantic import LanceModel, Vector
class Document(LanceModel):
id: int
text: str
vector: Vector(384, nullable=False)
rows = [Document.model_validate(row) for row in source_rows]
table = db.create_table("documents", schema=Document)
table.add(rows)
```
Use PyArrow schemas instead when the pipeline is already Arrow-native, needs
record-batch streaming, or has runtime schema requirements that would make a
Pydantic model harder to understand. Declare Pydantic as a direct project
dependency when application code imports it, even if LanceDB also depends on it.
## Recommended Patterns
### Bounded search or query
Use this for application reads, examples, notebooks, and agent-generated scripts:
```python
results = (
table.search(query_vector)
.where("status = 'ready'")
.select(["id", "text"])
.limit(20)
.to_list() # or .to_arrow(); .to_pandas()/.to_polars() only if the project uses them
)
```
Why: `search()` works across local and remote tables and on both the sync and async clients. `select()` avoids fetching unused columns. `limit()` prevents accidental full-table reads. `.to_list()` and `.to_arrow()` avoid assuming pandas/polars is installed (see "Before Writing Code").
For a **plain scan** (no query vector), the entry point differs by client:
```python
# Sync client: no .query() method — use .search() with no argument.
rows = table.search().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
# Async client: use .query().
rows = await async_table.query().where("status = 'ready'").select(["id", "text"]).limit(20).to_list()
```
`table.query()` on a sync table raises `AttributeError` (verified on `lancedb` 0.34.0). See the "Sync vs Async Scan API" section in `api_reference.md`.
### Bounded query result conversion
It is fine to collect bounded query/search results:
```python
arrow_table = table.search().select(["id"]).limit(100).to_arrow() # sync plain scan
rows = table.search(query_vector).limit(10).to_list()
df = table.search(query_vector).limit(10).to_pandas() # only if pandas is a project dep
```
### Local-only Lance dataset API
`table.to_lance()` does not itself materialize the full dataset. It returns the underlying `lance.LanceDataset`, making the table accessible through the PyLance dataset API. Use it when the task is explicitly local/OSS and needs Lance dataset methods not exposed by LanceDB:
```python
# Local/OSS only: RemoteTable does not expose table.to_lance().
ds = table.to_lance()
for batch in ds.to_batches(columns=["id", "text"], batch_size=10_000):
process(batch)
```
### Async Python
Keep the same shape and bound the result before collecting:
```python
results = await (
async_table.query()
.where("status = 'ready'")
.select(["id", "text"])
.limit(20)
.to_list() # or .to_arrow()
)
```
## Anti-Patterns
**Avoid the following anti-patterns in your code.**
### Table-level full materialization
Avoid whole-table collectors in portable or large-table code:
```python
df = table.to_pandas()
arrow_table = table.to_arrow()
polars_df = table.to_polars()
```
Why: local tables expose these whole-table collectors, but remote tables intentionally do not — a remote production table can be far larger than a local development table, so it is easy to accidentally pull the entire table into memory.
`table.to_lance()` is different: it is not a full materialization call, but it is still local/OSS-only and should not appear in code meant to run against remote Enterprise tables.
### Unbounded result collection
Avoid query/search collection without a meaningful limit:
```python
rows = table.search().to_list() # unbounded plain scan
rows = table.search(query_vector).to_list() # unbounded vector search
```
Prefer `select(...).limit(...)` before collecting; for large reads, stream in batches instead.
### Per-row writes
Avoid loops that write one row per call:
```python
for row in rows:
table.add([row]) # one commit + fragment per row
```
Each `add()` creates a new version and fragment. Pass the whole batch in a single call, or chunk very large inputs:
```python
table.add(rows) # single commit
# for very large inputs, add batches of several thousand rows
```
After the final successful write to an embedded OSS table, call
`table.optimize()`. Skip this for Enterprise/Cloud tables because their
maintenance is automatic.
### Drop-then-reuse the same table name (Enterprise/Cloud)
Avoid dropping or overwriting a remote table and then reusing that name right away:
```python
db.drop_table("my_table")
table = db.create_table("my_table", data=rows) # reads 500 for ~5 min
table = db.create_table("my_table", data=rows, mode="overwrite") # same problem
```
Why: Enterprise/Cloud splits DDL (control plane) from query serving (data plane). The data plane caches the dataset behind a table name for up to `table_cache_ttl` (default 300s / 5 min), so after a drop/overwrite the DDL succeeds but queries against the reused name return `500 Internal Server Error` until the cache expires — and a fresh `describe` may still show the old schema. Instead, write to a **fresh name**, use `list_tables()` and fail if it already exists, then `rename_table(fresh, final)` onto the final name only after the old table's drop has propagated (~5 min). See the "Enterprise: never drop-then-reuse the same table name" section in `SKILL.md`. Local/OSS tables have no separate data plane — overwrite freely there.
### Guessing performance fixes
Avoid changing `nprobes`, `refine_factor`, or index types before checking the query plan and index stats. Diagnose first, then tune one knob at a time.
plugins/lancedb/skills/lancedb/references/python/performance.md
-131
View File
@@ -1,131 +0,0 @@
# Python Performance Guidance
Use this when writing Python code that ingests data, queries large tables, builds indexes, or investigates latency.
## Ingestion
### Recommended: validate schemas and records with Pydantic
Favor `LanceModel` for readable Python schema definitions and validate source
records before writing. Use PyArrow directly for Arrow-native or streaming
pipelines where it is the clearer representation.
```python
from lancedb.pydantic import LanceModel, Vector
class Document(LanceModel):
id: int
text: str
vector: Vector(384, nullable=False)
rows = [Document.model_validate(row) for row in source_rows]
table = db.create_table("documents", schema=Document)
table.add(rows)
```
### Recommended: bulk ingestion for materialized data
```python
table.add(arrow_table)
table.add(df)
table.add(pa.dataset("data/", format="parquet"))
```
For very large initial loads, create the table empty first, then call `add(...)`. Passing data directly to `create_table(name, data)` can skip the auto-parallel write path.
### Recommended: iterator ingestion for generated or streamed data
```python
def batches():
for raw in source:
vectors = model.encode(raw["text"])
yield pa.RecordBatch.from_pydict({**raw, "vector": vectors})
table.add(batches())
```
Use chunks of several thousand rows or more when practical. Tiny batches and per-row writes create many small fragments.
### Anti-pattern: per-row `add()`
```python
for row in rows:
table.add([row])
```
Each call creates a version and fragment. This slows ingestion and later queries.
## Indexing
- Build a vector index once brute-force vector search becomes too slow. As a rule of thumb, local brute force is fine below roughly 100K vectors; beyond that, build an index.
- Use `IVF_PQ` as the general-purpose default. Enterprise builds this automatically.
- Use scalar indexes for filtered columns and merge/upsert keys.
- Use `BTREE` for mostly distinct numeric/string/temporal columns, `BITMAP` for booleans and low-cardinality columns, and `LABEL_LIST` for list membership queries.
- Keep full-text defaults unless phrase queries require position data.
## Querying
Always be explicit:
```python
table.search(query_vector).select(["id", "title"]).limit(20)
```
- `select()` reduces bytes read and transferred.
- `limit()` prevents accidental full-table materialization.
- Pre-filtering is the default and guarantees returned rows satisfy the predicate.
- Use post-filtering only when fewer than `limit` results are acceptable.
## Recall Tuning
Tune one knob at a time:
- Quantized indexes: raise `refine_factor` to rescore more candidates on full vectors.
- HNSW-backed indexes: raise `ef`; start around `1.5 * k`, increase toward `10 * k` if recall is short.
- IVF candidate breadth: `nprobes` is auto-tuned; override only when a selective pre-filter leaves too few neighbors.
## Maintenance
After every successful embedded OSS/local ingestion, call `table.optimize()`.
Do not add this to LanceDB Enterprise/Cloud remote table code; remote compaction
and cleanup are handled automatically based on the Enterprise cluster
configuration.
Why local maintenance is needed:
- Frequent writes can create many small fragments. Queries then need to scan across more files, which can increase latency.
- Updates, deletes, and appends create new table versions. Old versions are retained for time travel and rollback, which can grow disk usage.
- Indexes may have newly added rows that are not yet fully optimized into the index structure.
For local/OSS tables, run `optimize()` after the final successful ingestion
write. Also run it after later batches of update/delete operations or on a
regular maintenance schedule:
```python
table.optimize()
```
If the user wants more aggressive local disk cleanup, pass a shorter cleanup retention window:
```python
from datetime import timedelta
table.optimize(cleanup_older_than=timedelta(days=1))
```
Do not use very short cleanup windows when the application depends on time travel, rollback, or old versions.
## Diagnostics
Before changing code or indexes, inspect:
```python
print(table.search(query_vector).where("year > 2000").limit(10).analyze_plan())
print(table.index_stats("vector_idx"))
```
Look for high scan bytes, missing indexes, fragmented data, and unindexed rows.
## Python Multiprocessing
When using multiprocessing, use `spawn` rather than `fork`. LanceDB is multi-threaded internally, and `fork` plus a multi-threaded process is unsafe.
plugins/lancedb/skills/lancedb/references/typescript/api_reference.md
-105
View File
@@ -1,105 +0,0 @@
# TypeScript API Reference
Quick method reference for TypeScript LanceDB code. Cross-check source for non-trivial claims.
## Connect
```typescript
import * as lancedb from "@lancedb/lancedb";
const db = await lancedb.connect("./camelot-db");
```
**Place the local database directory next to the script/entrypoint that opens it** (resolve the path relative to the module, e.g. via `import.meta.dirname` / `__dirname`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.
**Do not name the directory `lancedb`** (e.g. `./lancedb`, `./data/lancedb`). It collides with the imported `lancedb` package/namespace, which is confusing to read. Give it a name derived from the repo or dataset with a clear prefix/suffix — for example `./<dataset>-db`, `./<repo>_lancedb`, or `./vectordb`.
Remote connections use `db://...` plus Enterprise/Cloud credentials and deployment settings. Check current source/docs for exact connection options.
## Table Reads
| Task | Preferred API |
| --- | --- |
| Vector search | `table.search(queryVector).limit(k)` |
| Full scan with filters/projection | `table.query().where(...).select(...).limit(...)` |
| Filter | `.where("col > 10")` |
| Projection | `.select(["id", "text"])` |
| Bound result count | `.limit(20)` |
| Collect bounded result as objects | `.toArray()` on query/search result |
| Collect bounded result as Arrow | `.toArrow()` on query/search result |
| Stream result batches | `for await (const batch of table.query()...)` |
## Local vs Remote Safety
| API | Agent guidance |
| --- | --- |
| `table.search(...)` | Preferred read path |
| `table.query()` | Preferred scan/filter path |
| `await table.toArrow()` | Avoid in portable or large-table code |
| `await table.query().toArray()` with no `limit()` | Avoid; unbounded collection |
| `await table.query().toArrow()` with no `limit()` | Avoid; unbounded collection |
## Indexes
```typescript
await table.createIndex("vector");
await table.createIndex("status");
```
Use vector indexes for large vector search workloads and scalar indexes for filtered columns or merge/upsert keys. Check source/docs before specifying advanced index options.
## Filtering And Recall Knobs
```typescript
await table.search(queryVector).where("status = 'ready'").limit(10).toArray();
await table.search(queryVector).limit(10).refineFactor(20).toArray();
await table.search(queryVector).limit(10).nprobes(50).toArray();
await table.search(queryVector).limit(10).ef(100).toArray();
await table.search(queryVector).where("status = 'ready'").postfilter().limit(10).toArray();
```
Use `postfilter()` only when fewer than `limit` results are acceptable.
## Diagnostics
```typescript
console.log(await table.search(queryVector).where("year > 2000").limit(10).analyzePlan());
console.log(await table.indexStats("vector_idx"));
```
Use these before changing indexes or search tuning.
## Column (Field) Metadata
```typescript
const schema = await table.schema();
const meta = schema.fields.find((f) => f.name === "category")?.metadata; // Map<string, string>
const res = await table.updateFieldMetadata([
{ path: "category", metadata: { "lancedb:description": "...", "lancedb:tag:field_type": "label" } },
]);
res.version; // new table version
```
Merges by default; a `null` value deletes that key; `replace: true` swaps the whole map. Nested fields use dot-paths (`"a.b.c"`). See `references/column_metadata.md` for key conventions (`lancedb:description`, `lancedb:tag:<name>`, `lancedb:logical-column`) and the authoring workflow.
## Branches
```typescript
const branches = await table.branches(); // async manager
await branches.list(); // non-main branches; {} = only main
const exp = await branches.create("exp"); // fork off main -> Table scoped to the branch
const wip = await branches.checkout("wip"); // existing branch -> scoped Table (version arg pins read-only)
const wip2 = await db.openTable("t", { branch: "wip" }); // or open scoped directly
await branches.delete("stale"); // removes only the branch pointer
table.currentBranch(); // null = main
```
There is no global switch — scoping is per table handle: any read/write on a branch handle lands on that branch; the original handle keeps targeting main. See `references/branch_ops.md` for the model and isolation checks.
## Maintenance
```typescript
await table.optimize();
```
Call this after every successful local/OSS ingestion. It handles compaction, cleanup of old versions according to retention, and index optimization. Do not add this for LanceDB Enterprise/Cloud remote tables; Enterprise handles compaction and cleanup automatically from cluster configuration.
plugins/lancedb/skills/lancedb/references/typescript/patterns.md
-100
View File
@@ -1,100 +0,0 @@
# TypeScript Patterns
Use these patterns when writing TypeScript code with `@lancedb/lancedb`.
## Recommended Patterns
### Bounded query
Use this for application reads, scripts, and examples:
```typescript
const rows = await table
.query()
.where("status = 'ready'")
.select(["id", "text"])
.limit(20)
.toArray();
```
### Bounded vector search
```typescript
const rows = await table
.search(queryVector)
.select(["id", "text"])
.limit(20)
.toArray();
```
### Batch streaming for larger reads
When the task needs many rows, avoid collecting everything at once:
```typescript
for await (const batch of table
.query()
.where("status = 'ready'")
.select(["id", "text"])
.limit(10_000)) {
process(batch);
}
```
## Anti-Patterns
**Avoid the following anti-patterns in your code.**
### Table-level full materialization
Avoid whole-table collectors in portable or large-table code:
```typescript
const tableArrow = await table.toArrow();
```
Why: local tables expose these whole-table collectors, but remote tables intentionally do not — a remote production table can be far larger than a local development table, so it is easy to accidentally pull the entire table into memory.
### Unbounded result collection
Avoid query/search collection without a meaningful limit:
```typescript
const rows = await table.query().toArray(); // unbounded plain scan
const rows = await table.search(queryVector).toArray(); // unbounded vector search
```
Prefer `select(...).limit(...)` before collecting; for large reads, stream in batches instead.
### Per-row writes
Avoid loops that write one row per call:
```typescript
for (const row of rows) {
await table.add([row]); // one commit + fragment per row
}
```
Each `add()` creates a new version and fragment. Pass the whole batch in a single call, or chunk very large inputs:
```typescript
await table.add(rows); // single commit
// for very large inputs, add in chunks of several thousand rows
```
### Drop-then-reuse the same table name (Enterprise/Cloud)
Avoid dropping or overwriting a remote table and then reusing that name right away:
```typescript
await db.dropTable("my_table");
const table = await db.createTable("my_table", rows); // reads 500 for ~5 min
const table = await db.createTable("my_table", rows, { mode: "overwrite" }); // same problem
```
Why: Enterprise/Cloud splits DDL (control plane) from query serving (data plane). The data plane caches the dataset behind a table name for up to `table_cache_ttl` (default 300s / 5 min), so after a drop/overwrite the DDL succeeds but queries against the reused name return `500 Internal Server Error` until the cache expires — and a fresh `describe` may still show the old schema. Instead, write to a **fresh name**, use `tableNames()` and fail if it already exists, then `renameTable(fresh, final)` onto the final name only after the old table's drop has propagated (~5 min). See the "Enterprise: never drop-then-reuse the same table name" section in `SKILL.md`. Local/OSS tables have no separate data plane — overwrite freely there.
### Guessing performance fixes
Avoid changing `nprobes`, `refineFactor`, `ef`, or index settings before checking `analyzePlan()` and `indexStats(...)`. Diagnose first, then tune one knob at a time.
plugins/lancedb/skills/lancedb/references/typescript/performance.md
-78
View File
@@ -1,78 +0,0 @@
# TypeScript Performance Guidance
Use this when writing TypeScript code that ingests data, queries large tables, builds indexes, or investigates latency.
## Ingestion
- Prefer bulk or batched writes.
- Avoid per-row write loops; they create many small commits/fragments.
- For generated data, accumulate reasonable batches before adding.
- For file-backed data, prefer APIs that stream from Arrow/Parquet-style inputs when available.
## Indexing
- Build a vector index once brute-force vector search becomes too slow. As a rule of thumb, local brute force is fine below roughly 100K vectors; beyond that, build an index.
- Use the general-purpose vector index defaults unless the task has explicit recall/latency requirements.
- Build scalar indexes for filtered columns and merge/upsert keys.
- Use full-text index phrase options only when phrase queries require them.
## Querying
Always be explicit:
```typescript
await table.search(queryVector).select(["id", "title"]).limit(20).toArray();
```
- `select()` reduces bytes read and transferred.
- `limit()` prevents accidental full-table collection.
- Pre-filtering is the default behavior. Use `postfilter()` only when fewer than `limit` results are acceptable.
## Recall Tuning
Tune one knob at a time:
- Quantized indexes: raise `refineFactor(...)` to rescore more candidates on full vectors.
- HNSW-backed indexes: raise `ef(...)`; start around `1.5 * k`, increase toward `10 * k` if recall is short.
- IVF candidate breadth: `nprobes(...)` is usually auto-tuned; override only when a selective pre-filter leaves too few neighbors.
## Maintenance
After every successful embedded OSS/local ingestion, call `table.optimize()`.
Do not add this to LanceDB Enterprise/Cloud remote table code; remote compaction
and cleanup are handled automatically based on the Enterprise cluster
configuration.
Why local maintenance is needed:
- Frequent writes can create many small fragments. Queries then need to scan across more files, which can increase latency.
- Updates, deletes, and appends create new table versions. Old versions are retained for time travel and rollback, which can grow disk usage.
- Indexes may have newly added rows that are not yet fully optimized into the index structure.
For local/OSS tables, run `optimize()` after the final successful ingestion
write. Also run it after later batches of update/delete operations or on a
regular maintenance schedule:
```typescript
await table.optimize();
```
If the user wants more aggressive local disk cleanup, pass a shorter cleanup retention window:
```typescript
const olderThan = new Date(Date.now() - 24 * 60 * 60 * 1000);
await table.optimize({ cleanupOlderThan: olderThan });
```
Do not use very short cleanup windows when the application depends on time travel, rollback, or old versions.
## Diagnostics
Before changing code or indexes, inspect:
```typescript
console.log(await table.search(queryVector).where("year > 2000").limit(10).analyzePlan());
console.log(await table.indexStats("vector_idx"));
```
Look for high scan cost, missing indexes, fragmented data, and unindexed rows.