rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 23:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: lead with what you do, add a Data Curation section

Browse Source 
The tab now runs write, read, curate, derive, train before it reaches the
management side: Ingestion, Semantic Search, Data Curation, Feature Engineering,
Training, then Management and Operations. Indexing joins Operations, where the
index maintenance page already was, and gains it as Build & Repair.

Ingestion gains a Create Table page. It was a section inside the overview, so
the one thing a new reader needs first was the one thing the sidebar did not
show.

Data Curation is new: Aggregate, Sampling, Join, Windowing. The SQL reference
answers what the syntax is; these answer why you would reach for it, framed as
the questions curation actually asks -- what do I have, what can I afford, what
else do I know, which one do I keep. Each links to the reference for the
complete grammar rather than repeating it.

They carry the things a reference page would not: that count(*) minus count(col)
is your coverage, that sampling both sides of a join gives a percent rather than
ten, that a window ordered by a value you recompute moves rows between train and
test, and that a one percent sample tells you about the head and nothing about
the tail.
This commit is contained in:
Jack Ye
2026-09-14 01:24:36 -07:00
parent 734eb39dc3
commit fd177e0aea
9 changed files with 864 additions and 361 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/curation/aggregate.mdx
+113
View File
@@ -0,0 +1,113 @@
---
title: "Profiling a dataset"
sidebarTitle: "Aggregate"
description: "Counting, grouping and summarizing — finding out what your data actually contains."
icon: chart-column
keywords: ["aggregate", "group by", "class balance", "count", "profiling", "distinct"]
---
<Badge color="red">Enterprise</Badge> Aggregation is SQL, and
[SQL needs a deployment](/basics/apis).
Before curating anything, find out what you have. Most of the surprises in a
training set — a class that is 90% of the rows, a column that is half null, a
"unique" id that is not — are one `GROUP BY` away from being visible.
## How big, and how complete {#size}
```sql
SELECT
count(*) AS rows,
count(embedding) AS embedded,
count(caption) AS captioned
FROM images
```
`count(*)` counts rows; `count(col)` counts **non-null values**. The gap between
them is your coverage, which is the fastest way to check how far a
[column refresh](/feature-engineering/refresh-column) has got without waiting
for the job to report.
## Class balance {#balance}
The single most useful curation query:
```sql
SELECT label, count(*) AS n
FROM images
GROUP BY label
ORDER BY n DESC
```
If the top class is an order of magnitude larger than the rest, sampling
uniformly will reproduce that imbalance. Two ways out: draw a fixed number per
class with [windowing](/curation/windowing#per-group), or oversample the rare
ones and accept the duplication.
## Distinct values {#distinct}
```sql
SELECT count(DISTINCT source) FROM images
```
On a high-cardinality column this is expensive, because exact distinctness means
holding every value seen. When an estimate will do:
```sql
SELECT approx_distinct(source) FROM images
```
Cheap enough to run on a table where the exact version is not worth the wait.
The other approximate functions — `approx_median`, `approx_percentile_cont` —
are useful for the same reason when profiling a numeric column.
## Conditional counts {#conditional}
`FILTER` narrows one aggregate without narrowing the query, which is how you get
several slices in one pass rather than one query each:
```sql
SELECT
label,
count(*) AS total,
count(*) FILTER (WHERE score > 0.9) AS confident,
count(*) FILTER (WHERE caption IS NULL) AS uncaptioned
FROM images
GROUP BY label
```
## Grouping by a bucket {#buckets}
Raw values are rarely the grouping you want. `date_bin` buckets timestamps, and
ordinary arithmetic buckets numbers:
```sql
SELECT date_bin(INTERVAL '1 day', created_at) AS day, count(*)
FROM images GROUP BY day ORDER BY day
```
A histogram of a score column is the same trick:
```sql
SELECT floor(score * 10) / 10 AS bucket, count(*)
FROM images GROUP BY bucket ORDER BY bucket
```
## Profiling a sample {#sampling}
On a very large table, profile a [sample](/curation/sampling) and scale:
```sql
SELECT label, count(*) * 100 AS estimated
FROM images TABLESAMPLE (1)
GROUP BY label
```
Treat the result as an estimate. A 1% sample will miss classes rarer than about
1 in 10,000 entirely, which is exactly the tail you are usually looking for — so
use this to check the shape of the head, not the existence of the tail.
## Reference {#reference}
The complete clause and function list is in
[Aggregation](/sql/aggregation).
docs/web/curation/index.mdx
+54
View File
@@ -0,0 +1,54 @@
---
title: "Data curation"
sidebarTitle: "Overview"
description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking."
icon: filter
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
---
Curation is the work between having data and having a *dataset*: finding out
what you actually have, drawing a subset you can afford to train on, attaching
labels that live elsewhere, and picking one row per thing when you have several.
These pages are task-oriented. Each shows the operation in the context you would
reach for it, and links to the [SQL reference](/sql) for the complete syntax.
<Note>
Most curation is analytical -- grouping, joining, ranking -- which the query
builders do not do. It is therefore SQL, and SQL needs a deployment. See
[SDKs and SQL](/basics/apis).
</Note>
## The four questions {#questions}
| | |
| --- | --- |
| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values |
| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed |
| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table |
| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits |
## A worked shape {#shape}
The four compose into the same pipeline most curation ends up being:
```sql
WITH ranked AS (
SELECT d.*, c.label,
row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn
FROM documents AS d
JOIN labels AS c ON c.doc_id = d.doc_id
)
SELECT * FROM ranked WHERE rn = 1
```
Join to bring in what you know, window to pick one row per document, filter to
keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger
than you want, and [aggregate](/curation/aggregate) to check the balance of what
you ended up with.
## Then what {#next}
A curated result you want to keep is a
[materialized view](/management/materialized-view) -- stored, refreshed, and
readable by a [data loader](/training) like any other table.
docs/web/curation/join.mdx
+91
View File
@@ -0,0 +1,91 @@
---
title: "Joining datasets"
sidebarTitle: "Join"
description: "Bringing labels, metadata and scores together with the rows they describe."
icon: link
keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
---
<Badge color="red">Enterprise</Badge> Joins are SQL, and
[SQL needs a deployment](/basics/apis).
Curation usually means combining things that were produced separately — images
in one table, human labels in another, model scores in a third. A join is how
they become one dataset.
## Attaching labels {#labels}
```sql
SELECT i.id, i.embedding, l.label
FROM images AS i
JOIN labels AS l ON l.image_id = i.id
```
An inner join keeps only rows that matched, which for curation is often exactly
what you want: unlabelled rows are not training data.
When you need to *see* what did not match, use a left join and filter:
```sql
SELECT i.id
FROM images AS i
LEFT JOIN labels AS l ON l.image_id = i.id
WHERE l.image_id IS NULL
```
That is your unlabelled backlog, which is the thing worth counting before you
plan an annotation round.
## Across namespaces {#namespaces}
Tables in different [namespaces](/management/namespace) join by their dotted
paths — the catalog is one catalog, so there is nothing to federate:
```sql
SELECT i.id, u.team
FROM prod.media.images AS i
JOIN prod.reference.uploaders AS u ON u.id = i.uploader_id
```
## Joining a search result {#search}
Full-text search is a table function, so its result is a relation and joins like
any other. This is how you ask which categories a query matches, rather than
just which rows:
```sql
SELECT c.name, count(*) AS hits
FROM fts('documents', 'puppy') AS m
JOIN categories AS c ON c.id = m.category_id
GROUP BY c.name
ORDER BY hits DESC
```
<Note>
Vector search has no SQL form, so a similarity result cannot be joined this way.
Run the search in an SDK, then join over its output — or write the result to a
table first.
</Note>
## Making it fast {#performance}
Two things dominate join cost on a lakehouse table.
**Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets
the planner look rows up instead of scanning for them. Check with
[`EXPLAIN`](/sql/explain): an indexed key shows a `ScalarIndexQuery`, an
unindexed one shows a `LanceScan` feeding the join.
**Do not carry wide columns through.** Select the columns you need before
joining, not after. Embeddings and blobs are the expensive part of a multimodal
table, and a join can materialize them many more times than the result has rows.
```sql
SELECT i.id, l.label
FROM (SELECT id, uploader_id FROM images) AS i
JOIN labels AS l ON l.image_id = i.id
```
## Reference {#reference}
The join types and their semantics are in [Joins](/sql/join).
docs/web/curation/sampling.mdx
+98
View File
@@ -0,0 +1,98 @@
---
title: "Sampling a dataset"
sidebarTitle: "Sampling"
description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible."
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
---
Most curation starts with a subset. A question you can answer in ten seconds on
1% of the data is worth asking twenty times; the same question on the whole
table is worth asking once.
## A quick look {#quick}
<Badge color="red">Enterprise</Badge>
```sql
SELECT * FROM images TABLESAMPLE (1) -- 1 percent
SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows
```
A bare number is a percentage; add `ROWS` for a count. Use it while you are
exploring — the cost of the query drops with the sample, and so does the cost
of being wrong about what you wanted to ask.
## Rows or fragments {#methods}
```sql
SELECT * FROM images TABLESAMPLE BERNOULLI (10) -- rows
SELECT * FROM images TABLESAMPLE SYSTEM (10) -- fragments
```
`BERNOULLI` considers each row independently and reads across the whole table.
`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O,
because it skips files rather than rows.
The trade is uniformity. A fragment sample is only as representative as your
write order. If rows arrived grouped by source, by date, or by label — which is
usual — then a fragment sample inherits that grouping.
<Note>
Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number
you compute has to be defensible. Omitting the method gives `BERNOULLI`.
</Note>
## A fixed evaluation set {#fixed}
Without a seed, every execution resamples. That is fine for exploring and wrong
for evaluation — a metric computed on different rows each run is not comparable
with itself.
```sql
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
```
The same seed against the same data gives the same rows, which is what makes a
sample usable as a held-out set. If the underlying data changes, so does the
draw — when you need the set frozen against that too, materialize it:
```sql
CREATE TABLE eval_set AS
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
```
A [materialized view](/management/materialized-view) is the other option when
you want it to track the source on a refresh rather than stay frozen.
## Sampling both sides of a join {#joins}
```sql
SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id
```
Each side samples independently, so a row survives only if both of its sides
were drawn. Sampling 10% of each side does **not** give 10% of the join — it
gives roughly 1%. Sample once, after the join, unless you specifically want the
smaller result.
## Shuffling for training {#training}
For a training loop rather than a query, the SDKs read a table in a shuffled
order without SQL at all:
```python
from lancedb.permutation import Permutation
permutation = Permutation.identity(table)
dataloader = torch.utils.data.DataLoader(permutation)
```
This is the right tool for epoch-level shuffling, where you want every row
eventually and only the order varies. `TABLESAMPLE` is for when you want fewer
rows. See [Training](/training).
## Reference {#reference}
The full clause, its options and what it refuses are in
[Sampling](/sql/sampling).
docs/web/curation/windowing.mdx
+124
View File
@@ -0,0 +1,124 @@
---
title: "Ranking and deduplication"
sidebarTitle: "Windowing"
description: "Picking one row per thing, ranking within a group, and splitting a dataset."
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
---
<Badge color="red">Enterprise</Badge> Window functions are SQL, and
[SQL needs a deployment](/basics/apis).
A window function computes across a set of related rows and returns a value
**per row** — nothing is collapsed. That is what makes it the tool for "which of
these do I keep", where an aggregate would throw away the rows you are choosing
between.
## One row per thing {#dedup}
The most common curation query there is. You have several rows per document —
revisions, re-uploads, repeated annotations — and want one.
```sql
WITH ranked AS (
SELECT *,
row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
FROM revisions
)
SELECT * FROM ranked WHERE rn = 1
```
`PARTITION BY` says what "per thing" means; `ORDER BY` says which one wins. The
outer query is not optional — a window function cannot appear in `WHERE`,
because windows are computed after filtering, so the predicate would have
nothing to read.
<Note>
Swap `row_number()` for `rank()` when ties should both survive. `row_number()`
always picks exactly one; `rank()` gives tied rows the same rank, so `rank = 1`
can return several.
</Note>
## A fixed number per class {#per-group}
Class imbalance found by [profiling](/curation/aggregate#balance) is fixed here:
rank within each class, then take the top N.
```sql
WITH ranked AS (
SELECT *, row_number() OVER (PARTITION BY label ORDER BY score DESC) AS rn
FROM images
)
SELECT * FROM ranked WHERE rn <= 1000
```
That gives at most 1,000 rows per label, taking the highest-scoring ones. Order
by something arbitrary but stable if you want an unbiased draw instead, or
combine with [sampling](/curation/sampling) to randomize within each class.
## Splitting a dataset {#splits}
`ntile` divides each partition into equal buckets, which is a train/test split
that stays balanced per class:
```sql
WITH split AS (
SELECT *, ntile(10) OVER (PARTITION BY label ORDER BY id) AS bucket
FROM images
)
SELECT * FROM split WHERE bucket <= 8 -- train
```
Buckets 9 and 10 are your validation and test sets. Because the partition is by
label, each split keeps the class proportions of the whole.
<Warning>
Order by something stable. Ordering by a value that changes — a score you
recompute, a timestamp that gets updated — means the split changes with it, and
a row can move from train to test between runs.
</Warning>
## Comparing a row to its neighbours {#offset}
`lag` and `lead` read the previous and next row in the window, which is how you
find gaps and changes without a self-join:
```sql
SELECT
id,
created_at,
created_at - lag(created_at) OVER (ORDER BY created_at) AS gap
FROM events
```
Useful for spotting ingestion gaps, duplicate timestamps, and runs of identical
values that suggest a broken upstream job.
## Running totals {#running}
Any [aggregate](/curation/aggregate) becomes a window function with `OVER`,
which is how cumulative coverage is expressed:
```sql
SELECT
label,
count(*) AS n,
sum(count(*)) OVER (ORDER BY count(*) DESC) AS cumulative
FROM images
GROUP BY label
```
Reading down the `cumulative` column tells you how many classes cover most of
your data — usually far fewer than you expect.
## Cost {#cost}
Window functions sort within each partition, so on a large table they are among
the more expensive things you can ask for. Filter first, partition on something
selective, and check the plan with
[Explain Analyze](/search/explain).
## Reference {#reference}
The full window syntax, framing, and function list are in
[Window functions](/sql/window).
docs/web/docs.json
+34 -23
View File
@@ -162,32 +162,11 @@
"basics/agentic-skills"
]
},
{
"group": "Management",
"pages": [
"management/index",
"management/database",
"management/namespace",
"management/table",
"tables/multimodal",
"tables/branching",
"management/materialized-view",
"management/function",
{
"group": "Indexing",
"pages": [
"indexing/index",
"indexing/vector-index",
"indexing/fts-index",
"indexing/scalar-index"
]
}
]
},
{
"group": "Ingestion",
"pages": [
"ingestion/index",
"ingestion/create-table",
"ingestion/insert",
"ingestion/merge-insert",
"ingestion/update",
@@ -206,6 +185,16 @@
"search/explain"
]
},
{
"group": "Data Curation",
"pages": [
"curation/index",
"curation/aggregate",
"curation/sampling",
"curation/join",
"curation/windowing"
]
},
{
"group": "Feature Engineering",
"pages": [
@@ -224,10 +213,32 @@
"training/torch"
]
},
{
"group": "Management",
"pages": [
"management/index",
"management/database",
"management/namespace",
"management/table",
"tables/multimodal",
"tables/branching",
"management/materialized-view",
"management/function"
]
},
{
"group": "Operations",
"pages": [
"indexing/reindexing",
{
"group": "Indexing",
"pages": [
"indexing/index",
"indexing/vector-index",
"indexing/fts-index",
"indexing/scalar-index",
"indexing/reindexing"
]
},
"maintenance/compaction",
{
"group": "Storage",
docs/web/indexing/reindexing.mdx
+1 -1
View File
@@ -1,6 +1,6 @@
---
title: "Building and repairing indexes"
sidebarTitle: "Indexing"
sidebarTitle: "Build & Repair"
description: "Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing."
icon: "refresh"
keywords: ["reindexing", "re-index", "incremental indexing", "reindex"]
docs/web/ingestion/create-table.mdx
+347
View File
@@ -0,0 +1,347 @@
---
title: "Creating a table"
sidebarTitle: "Create Table"
description: "Start here: turn the data you already have into a LanceDB table."
icon: table
keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"]
---
import {
TsConnect,
RsConnect,
} from '/snippets/connection.mdx';
import {
PyCreateTableFromDicts as CreateTableFromDicts,
TsCreateTableFromDicts as TsCreateTableFromDicts,
RsCreateTableFromDicts as RsCreateTableFromDicts,
PyCreateTableConflictHandling as CreateTableConflictHandling,
TsCreateTableConflictHandling as TsCreateTableConflictHandling,
RsCreateTableConflictHandling as RsCreateTableConflictHandling,
PyCreateTableFromPandas as CreateTableFromPandas,
PyCreateTableCustomSchema as CreateTableCustomSchema,
TsCreateTableCustomSchema as TsCreateTableCustomSchema,
RsCreateTableCustomSchema as RsCreateTableCustomSchema,
PyCreateTableFromPolars as CreateTableFromPolars,
PyCreateTableFromArrow as CreateTableFromArrow,
TsCreateTableFromArrow as TsCreateTableFromArrow,
RsCreateTableFromArrow as RsCreateTableFromArrow,
PyCreateTableFromPydantic as CreateTableFromPydantic,
PyCreateTableNestedSchema as CreateTableNestedSchema,
PyOpenExistingTable as OpenExistingTable,
TsOpenExistingTable as TsOpenExistingTable,
RsOpenExistingTable as RsOpenExistingTable,
PyCreateEmptyTable as CreateEmptyTable,
TsCreateEmptyTable as TsCreateEmptyTable,
RsCreateEmptyTable as RsCreateEmptyTable,
PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
PyDropTable as DropTable,
TsDropTable as TsDropTable,
RsDropTable as RsDropTable,
PyTablesBasicConnect as TablesBasicConnect,
PyTablesDocumentModel as TablesDocumentModel,
PyTablesTzValidator as TablesTzValidator,
} from '/snippets/tables.mdx';
A table is created from data you already have -- a list of objects, an Arrow
table, a pandas or polars frame, Pydantic models -- or from a schema alone when
the rows come later.
## Create a table with data {#create-a-table-with-data}
Initialize a LanceDB connection and create a table
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesBasicConnect}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConnect}
</CodeBlock>
</CodeGroup>
Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
### From list of objects {#from-list-of-objects}
You can provide a list of objects to create a table. The Python and TypeScript SDKs
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
In Python, pass a list or other batch-like object; a single bare `dict` or single
`LanceModel` is rejected.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromDicts}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableFromDicts}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableFromDicts}
</CodeBlock>
</CodeGroup>
### Handle existing tables {#handle-existing-tables}
By default, `create_table` raises an error if a table with the same name already exists.
You can change this behavior with two parameters that resolve the conflict in different ways:
- **Idempotent open**: return the existing table without modifying it. Use when your
code may run more than once (notebooks, retries, init scripts) and you want to reuse
the table on subsequent runs. The provided data is ignored, but the schema is
validated against the existing table and a mismatch raises an error.
- **Overwrite**: drop the existing table and create a new one with the provided data.
Use this for test fixtures or when you intentionally want to replace prior contents.
This permanently discards the old table's data.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableConflictHandling}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableConflictHandling}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableConflictHandling}
</CodeBlock>
</CodeGroup>
<Note>
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
</Note>
### From a custom schema {#from-a-custom-schema}
You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableCustomSchema}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableCustomSchema}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableCustomSchema}
</CodeBlock>
</CodeGroup>
An explicit schema is also where you control nullability. If later writes omit a
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.
### From an Arrow Table {#from-an-arrow-table}
You can also create LanceDB tables directly from Arrow tables.
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromArrow}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableFromArrow}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableFromArrow}
</CodeBlock>
</CodeGroup>
### From a Pandas DataFrame {#from-a-pandas-dataframe}
<Badge color="green">Python Only</Badge>
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPandas}
</CodeBlock>
</CodeGroup>
<Note title="Note">
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
</Note>
<Note title="Vector Column Type">
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
</Note>
### From a Polars DataFrame {#from-a-polars-dataframe}
<Badge color="green">Python Only</Badge>
LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
is on the way.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPolars}
</CodeBlock>
</CodeGroup>
### From Pydantic Models {#from-pydantic-models}
<Badge color="green">Python Only</Badge>
When you create an empty table without data, you must specify the table schema.
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
Pydantic model called `LanceModel`.
For example, the following Content model specifies a table with 5 columns:
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
pass the class as the value of the `schema` parameter to `create_table`.
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
can be configured with the vector dimensions. It is also important to note that
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
(which itself derives from `pydantic.BaseModel`).
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPydantic}
</CodeBlock>
</CodeGroup>
#### Nested schemas {#nested-schemas}
Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesDocumentModel}
</CodeBlock>
</CodeGroup>
This can be used as the type of a LanceDB table column:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableNestedSchema}
</CodeBlock>
</CodeGroup>
This creates a struct column called "document" that has two subfields
called "content" and "source":
```bash
In [28]: tbl.schema
Out[28]:
id: string not null
vector: fixed_size_list<item: float>[1536] not null
child 0, item: float
document: struct<content: string not null, source: string not null> not null
child 0, content: string not null
child 1, source: string not null
```
#### Validators {#validators}
Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
for a `created_at` field.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesTzValidator}
</CodeBlock>
</CodeGroup>
When you run this code it, should raise the `ValidationError`.
<Note>
Datasets too large to pass to `create_table` at once are covered separately --
see [Insert data](/ingestion/insert#large-datasets), which explains why
`table.add()` parallelizes where `create_table` does not.
</Note>
## Create empty table {#create-empty-table}
You can create an empty table for scenarios where you want to add data to the table later.
An example would be when you want to collect data from a stream/external file and then add it to a table in
batches.
An empty table can be initialized via an Arrow schema.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateEmptyTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateEmptyTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateEmptyTable}
</CodeBlock>
</CodeGroup>
Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
that has been extended to support LanceDB specific types like `Vector`.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateEmptyTablePydantic}
</CodeBlock>
</CodeGroup>
Once the empty table has been created, you can append to it or modify its contents,
as explained in [Ingesting data](/ingestion).
## Open an existing table {#open-an-existing-table}
You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
If you forget the name of your table, you can always get a listing of all table names.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{OpenExistingTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsOpenExistingTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsOpenExistingTable}
</CodeBlock>
</CodeGroup>
## Drop a table {#drop-a-table}
Use the `drop_table()` method on the database to remove a table.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{DropTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsDropTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsDropTable}
</CodeBlock>
</CodeGroup>
This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.
docs/web/ingestion/index.mdx
+2 -337
View File
@@ -7,44 +7,6 @@ keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars
---
import {
TsConnect,
RsConnect,
} from '/snippets/connection.mdx';
import {
PyCreateTableFromDicts as CreateTableFromDicts,
TsCreateTableFromDicts as TsCreateTableFromDicts,
RsCreateTableFromDicts as RsCreateTableFromDicts,
PyCreateTableConflictHandling as CreateTableConflictHandling,
TsCreateTableConflictHandling as TsCreateTableConflictHandling,
RsCreateTableConflictHandling as RsCreateTableConflictHandling,
PyCreateTableFromPandas as CreateTableFromPandas,
PyCreateTableCustomSchema as CreateTableCustomSchema,
TsCreateTableCustomSchema as TsCreateTableCustomSchema,
RsCreateTableCustomSchema as RsCreateTableCustomSchema,
PyCreateTableFromPolars as CreateTableFromPolars,
PyCreateTableFromArrow as CreateTableFromArrow,
TsCreateTableFromArrow as TsCreateTableFromArrow,
RsCreateTableFromArrow as RsCreateTableFromArrow,
PyCreateTableFromPydantic as CreateTableFromPydantic,
PyCreateTableNestedSchema as CreateTableNestedSchema,
PyAddFromDataset as AddFromDataset,
PyCreateTableFromIterator as CreateTableFromIterator,
TsCreateTableFromIterator as TsCreateTableFromIterator,
RsCreateTableFromIterator as RsCreateTableFromIterator,
TsAddProgress,
PyOpenExistingTable as OpenExistingTable,
TsOpenExistingTable as TsOpenExistingTable,
RsOpenExistingTable as RsOpenExistingTable,
PyCreateEmptyTable as CreateEmptyTable,
TsCreateEmptyTable as TsCreateEmptyTable,
RsCreateEmptyTable as RsCreateEmptyTable,
PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
PyDropTable as DropTable,
TsDropTable as TsDropTable,
RsDropTable as RsDropTable,
PyTablesBasicConnect as TablesBasicConnect,
PyTablesDocumentModel as TablesDocumentModel,
PyTablesTzValidator as TablesTzValidator,
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
PyUpdateConnectLocal as UpdateConnectLocal,
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
@@ -71,305 +33,8 @@ second.
Every operation here has a SQL form -- see [DML](/sql/dml).
</Note>
## Create a table with data {#create-a-table-with-data}
Initialize a LanceDB connection and create a table
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesBasicConnect}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsConnect}
</CodeBlock>
</CodeGroup>
Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
### From list of objects {#from-list-of-objects}
You can provide a list of objects to create a table. The Python and TypeScript SDKs
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
In Python, pass a list or other batch-like object; a single bare `dict` or single
`LanceModel` is rejected.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromDicts}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableFromDicts}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableFromDicts}
</CodeBlock>
</CodeGroup>
### Handle existing tables {#handle-existing-tables}
By default, `create_table` raises an error if a table with the same name already exists.
You can change this behavior with two parameters that resolve the conflict in different ways:
- **Idempotent open**: return the existing table without modifying it. Use when your
code may run more than once (notebooks, retries, init scripts) and you want to reuse
the table on subsequent runs. The provided data is ignored, but the schema is
validated against the existing table and a mismatch raises an error.
- **Overwrite**: drop the existing table and create a new one with the provided data.
Use this for test fixtures or when you intentionally want to replace prior contents.
This permanently discards the old table's data.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableConflictHandling}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableConflictHandling}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableConflictHandling}
</CodeBlock>
</CodeGroup>
<Note>
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
</Note>
### From a custom schema {#from-a-custom-schema}
You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableCustomSchema}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableCustomSchema}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableCustomSchema}
</CodeBlock>
</CodeGroup>
An explicit schema is also where you control nullability. If later writes omit a
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.
### From an Arrow Table {#from-an-arrow-table}
You can also create LanceDB tables directly from Arrow tables.
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromArrow}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateTableFromArrow}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateTableFromArrow}
</CodeBlock>
</CodeGroup>
### From a Pandas DataFrame {#from-a-pandas-dataframe}
<Badge color="green">Python Only</Badge>
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPandas}
</CodeBlock>
</CodeGroup>
<Note title="Note">
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
</Note>
<Note title="Vector Column Type">
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
</Note>
### From a Polars DataFrame {#from-a-polars-dataframe}
<Badge color="green">Python Only</Badge>
LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
is on the way.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPolars}
</CodeBlock>
</CodeGroup>
### From Pydantic Models {#from-pydantic-models}
<Badge color="green">Python Only</Badge>
When you create an empty table without data, you must specify the table schema.
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
Pydantic model called `LanceModel`.
For example, the following Content model specifies a table with 5 columns:
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
pass the class as the value of the `schema` parameter to `create_table`.
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
can be configured with the vector dimensions. It is also important to note that
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
(which itself derives from `pydantic.BaseModel`).
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableFromPydantic}
</CodeBlock>
</CodeGroup>
#### Nested schemas {#nested-schemas}
Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesDocumentModel}
</CodeBlock>
</CodeGroup>
This can be used as the type of a LanceDB table column:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateTableNestedSchema}
</CodeBlock>
</CodeGroup>
This creates a struct column called "document" that has two subfields
called "content" and "source":
```bash
In [28]: tbl.schema
Out[28]:
id: string not null
vector: fixed_size_list<item: float>[1536] not null
child 0, item: float
document: struct<content: string not null, source: string not null> not null
child 0, content: string not null
child 1, source: string not null
```
#### Validators {#validators}
Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
for a `created_at` field.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{TablesTzValidator}
</CodeBlock>
</CodeGroup>
When you run this code it, should raise the `ValidationError`.
<Note>
Datasets too large to pass to `create_table` at once are covered separately --
see [Insert data](/ingestion/insert#large-datasets), which explains why
`table.add()` parallelizes where `create_table` does not.
</Note>
## Create empty table {#create-empty-table}
You can create an empty table for scenarios where you want to add data to the table later.
An example would be when you want to collect data from a stream/external file and then add it to a table in
batches.
An empty table can be initialized via an Arrow schema.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateEmptyTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsCreateEmptyTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsCreateEmptyTable}
</CodeBlock>
</CodeGroup>
Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
that has been extended to support LanceDB specific types like `Vector`.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{CreateEmptyTablePydantic}
</CodeBlock>
</CodeGroup>
Once the empty table has been created, you can append to it or modify its contents,
as explained in [Ingesting data](/ingestion).
## Open an existing table {#open-an-existing-table}
You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
If you forget the name of your table, you can always get a listing of all table names.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{OpenExistingTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsOpenExistingTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsOpenExistingTable}
</CodeBlock>
</CodeGroup>
## Drop a table {#drop-a-table}
Use the `drop_table()` method on the database to remove a table.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{DropTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsDropTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsDropTable}
</CodeBlock>
</CodeGroup>
This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.
Start by [creating a table](/ingestion/create-table); the rest of this
section writes to one that exists.
## Choose a write method {#choose-a-write-method}