diff --git a/docs/web/curation/aggregate.mdx b/docs/web/curation/aggregate.mdx new file mode 100644 index 000000000..7fab2c662 --- /dev/null +++ b/docs/web/curation/aggregate.mdx @@ -0,0 +1,113 @@ +--- +title: "Profiling a dataset" +sidebarTitle: "Aggregate" +description: "Counting, grouping and summarizing — finding out what your data actually contains." +icon: chart-column +keywords: ["aggregate", "group by", "class balance", "count", "profiling", "distinct"] +--- + +Enterprise Aggregation is SQL, and +[SQL needs a deployment](/basics/apis). + +Before curating anything, find out what you have. Most of the surprises in a +training set — a class that is 90% of the rows, a column that is half null, a +"unique" id that is not — are one `GROUP BY` away from being visible. + +## How big, and how complete {#size} + +```sql +SELECT + count(*) AS rows, + count(embedding) AS embedded, + count(caption) AS captioned +FROM images +``` + +`count(*)` counts rows; `count(col)` counts **non-null values**. The gap between +them is your coverage, which is the fastest way to check how far a +[column refresh](/feature-engineering/refresh-column) has got without waiting +for the job to report. + +## Class balance {#balance} + +The single most useful curation query: + +```sql +SELECT label, count(*) AS n +FROM images +GROUP BY label +ORDER BY n DESC +``` + +If the top class is an order of magnitude larger than the rest, sampling +uniformly will reproduce that imbalance. Two ways out: draw a fixed number per +class with [windowing](/curation/windowing#per-group), or oversample the rare +ones and accept the duplication. + +## Distinct values {#distinct} + +```sql +SELECT count(DISTINCT source) FROM images +``` + +On a high-cardinality column this is expensive, because exact distinctness means +holding every value seen. When an estimate will do: + +```sql +SELECT approx_distinct(source) FROM images +``` + +Cheap enough to run on a table where the exact version is not worth the wait. +The other approximate functions — `approx_median`, `approx_percentile_cont` — +are useful for the same reason when profiling a numeric column. + +## Conditional counts {#conditional} + +`FILTER` narrows one aggregate without narrowing the query, which is how you get +several slices in one pass rather than one query each: + +```sql +SELECT + label, + count(*) AS total, + count(*) FILTER (WHERE score > 0.9) AS confident, + count(*) FILTER (WHERE caption IS NULL) AS uncaptioned +FROM images +GROUP BY label +``` + +## Grouping by a bucket {#buckets} + +Raw values are rarely the grouping you want. `date_bin` buckets timestamps, and +ordinary arithmetic buckets numbers: + +```sql +SELECT date_bin(INTERVAL '1 day', created_at) AS day, count(*) +FROM images GROUP BY day ORDER BY day +``` + +A histogram of a score column is the same trick: + +```sql +SELECT floor(score * 10) / 10 AS bucket, count(*) +FROM images GROUP BY bucket ORDER BY bucket +``` + +## Profiling a sample {#sampling} + +On a very large table, profile a [sample](/curation/sampling) and scale: + +```sql +SELECT label, count(*) * 100 AS estimated +FROM images TABLESAMPLE (1) +GROUP BY label +``` + +Treat the result as an estimate. A 1% sample will miss classes rarer than about +1 in 10,000 entirely, which is exactly the tail you are usually looking for — so +use this to check the shape of the head, not the existence of the tail. + +## Reference {#reference} + +The complete clause and function list is in +[Aggregation](/sql/aggregation). diff --git a/docs/web/curation/index.mdx b/docs/web/curation/index.mdx new file mode 100644 index 000000000..216149c71 --- /dev/null +++ b/docs/web/curation/index.mdx @@ -0,0 +1,54 @@ +--- +title: "Data curation" +sidebarTitle: "Overview" +description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking." +icon: filter +keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"] +--- + +Curation is the work between having data and having a *dataset*: finding out +what you actually have, drawing a subset you can afford to train on, attaching +labels that live elsewhere, and picking one row per thing when you have several. + +These pages are task-oriented. Each shows the operation in the context you would +reach for it, and links to the [SQL reference](/sql) for the complete syntax. + + +Most curation is analytical -- grouping, joining, ranking -- which the query +builders do not do. It is therefore SQL, and SQL needs a deployment. See +[SDKs and SQL](/basics/apis). + + +## The four questions {#questions} + +| | | +| --- | --- | +| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values | +| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed | +| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table | +| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits | + +## A worked shape {#shape} + +The four compose into the same pipeline most curation ends up being: + +```sql +WITH ranked AS ( + SELECT d.*, c.label, + row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn + FROM documents AS d + JOIN labels AS c ON c.doc_id = d.doc_id +) +SELECT * FROM ranked WHERE rn = 1 +``` + +Join to bring in what you know, window to pick one row per document, filter to +keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger +than you want, and [aggregate](/curation/aggregate) to check the balance of what +you ended up with. + +## Then what {#next} + +A curated result you want to keep is a +[materialized view](/management/materialized-view) -- stored, refreshed, and +readable by a [data loader](/training) like any other table. diff --git a/docs/web/curation/join.mdx b/docs/web/curation/join.mdx new file mode 100644 index 000000000..f8d55d93c --- /dev/null +++ b/docs/web/curation/join.mdx @@ -0,0 +1,91 @@ +--- +title: "Joining datasets" +sidebarTitle: "Join" +description: "Bringing labels, metadata and scores together with the rows they describe." +icon: link +keywords: ["join", "labels", "metadata", "enrichment", "namespace"] +--- + +Enterprise Joins are SQL, and +[SQL needs a deployment](/basics/apis). + +Curation usually means combining things that were produced separately — images +in one table, human labels in another, model scores in a third. A join is how +they become one dataset. + +## Attaching labels {#labels} + +```sql +SELECT i.id, i.embedding, l.label +FROM images AS i +JOIN labels AS l ON l.image_id = i.id +``` + +An inner join keeps only rows that matched, which for curation is often exactly +what you want: unlabelled rows are not training data. + +When you need to *see* what did not match, use a left join and filter: + +```sql +SELECT i.id +FROM images AS i +LEFT JOIN labels AS l ON l.image_id = i.id +WHERE l.image_id IS NULL +``` + +That is your unlabelled backlog, which is the thing worth counting before you +plan an annotation round. + +## Across namespaces {#namespaces} + +Tables in different [namespaces](/management/namespace) join by their dotted +paths — the catalog is one catalog, so there is nothing to federate: + +```sql +SELECT i.id, u.team +FROM prod.media.images AS i +JOIN prod.reference.uploaders AS u ON u.id = i.uploader_id +``` + +## Joining a search result {#search} + +Full-text search is a table function, so its result is a relation and joins like +any other. This is how you ask which categories a query matches, rather than +just which rows: + +```sql +SELECT c.name, count(*) AS hits +FROM fts('documents', 'puppy') AS m +JOIN categories AS c ON c.id = m.category_id +GROUP BY c.name +ORDER BY hits DESC +``` + + +Vector search has no SQL form, so a similarity result cannot be joined this way. +Run the search in an SDK, then join over its output — or write the result to a +table first. + + +## Making it fast {#performance} + +Two things dominate join cost on a lakehouse table. + +**Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets +the planner look rows up instead of scanning for them. Check with +[`EXPLAIN`](/sql/explain): an indexed key shows a `ScalarIndexQuery`, an +unindexed one shows a `LanceScan` feeding the join. + +**Do not carry wide columns through.** Select the columns you need before +joining, not after. Embeddings and blobs are the expensive part of a multimodal +table, and a join can materialize them many more times than the result has rows. + +```sql +SELECT i.id, l.label +FROM (SELECT id, uploader_id FROM images) AS i +JOIN labels AS l ON l.image_id = i.id +``` + +## Reference {#reference} + +The join types and their semantics are in [Joins](/sql/join). diff --git a/docs/web/curation/sampling.mdx b/docs/web/curation/sampling.mdx new file mode 100644 index 000000000..f38517379 --- /dev/null +++ b/docs/web/curation/sampling.mdx @@ -0,0 +1,98 @@ +--- +title: "Sampling a dataset" +sidebarTitle: "Sampling" +description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible." +icon: dice +keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"] +--- + +Most curation starts with a subset. A question you can answer in ten seconds on +1% of the data is worth asking twenty times; the same question on the whole +table is worth asking once. + +## A quick look {#quick} + +Enterprise + +```sql +SELECT * FROM images TABLESAMPLE (1) -- 1 percent +SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows +``` + +A bare number is a percentage; add `ROWS` for a count. Use it while you are +exploring — the cost of the query drops with the sample, and so does the cost +of being wrong about what you wanted to ask. + +## Rows or fragments {#methods} + +```sql +SELECT * FROM images TABLESAMPLE BERNOULLI (10) -- rows +SELECT * FROM images TABLESAMPLE SYSTEM (10) -- fragments +``` + +`BERNOULLI` considers each row independently and reads across the whole table. +`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O, +because it skips files rather than rows. + +The trade is uniformity. A fragment sample is only as representative as your +write order. If rows arrived grouped by source, by date, or by label — which is +usual — then a fragment sample inherits that grouping. + + +Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number +you compute has to be defensible. Omitting the method gives `BERNOULLI`. + + +## A fixed evaluation set {#fixed} + +Without a seed, every execution resamples. That is fine for exploring and wrong +for evaluation — a metric computed on different rows each run is not comparable +with itself. + +```sql +SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42) +``` + +The same seed against the same data gives the same rows, which is what makes a +sample usable as a held-out set. If the underlying data changes, so does the +draw — when you need the set frozen against that too, materialize it: + +```sql +CREATE TABLE eval_set AS +SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42) +``` + +A [materialized view](/management/materialized-view) is the other option when +you want it to track the source on a refresh rather than stay frozen. + +## Sampling both sides of a join {#joins} + +```sql +SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id +``` + +Each side samples independently, so a row survives only if both of its sides +were drawn. Sampling 10% of each side does **not** give 10% of the join — it +gives roughly 1%. Sample once, after the join, unless you specifically want the +smaller result. + +## Shuffling for training {#training} + +For a training loop rather than a query, the SDKs read a table in a shuffled +order without SQL at all: + +```python +from lancedb.permutation import Permutation + +permutation = Permutation.identity(table) +dataloader = torch.utils.data.DataLoader(permutation) +``` + +This is the right tool for epoch-level shuffling, where you want every row +eventually and only the order varies. `TABLESAMPLE` is for when you want fewer +rows. See [Training](/training). + +## Reference {#reference} + +The full clause, its options and what it refuses are in +[Sampling](/sql/sampling). diff --git a/docs/web/curation/windowing.mdx b/docs/web/curation/windowing.mdx new file mode 100644 index 000000000..2386ece3e --- /dev/null +++ b/docs/web/curation/windowing.mdx @@ -0,0 +1,124 @@ +--- +title: "Ranking and deduplication" +sidebarTitle: "Windowing" +description: "Picking one row per thing, ranking within a group, and splitting a dataset." +icon: chart-simple +keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"] +--- + +Enterprise Window functions are SQL, and +[SQL needs a deployment](/basics/apis). + +A window function computes across a set of related rows and returns a value +**per row** — nothing is collapsed. That is what makes it the tool for "which of +these do I keep", where an aggregate would throw away the rows you are choosing +between. + +## One row per thing {#dedup} + +The most common curation query there is. You have several rows per document — +revisions, re-uploads, repeated annotations — and want one. + +```sql +WITH ranked AS ( + SELECT *, + row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn + FROM revisions +) +SELECT * FROM ranked WHERE rn = 1 +``` + +`PARTITION BY` says what "per thing" means; `ORDER BY` says which one wins. The +outer query is not optional — a window function cannot appear in `WHERE`, +because windows are computed after filtering, so the predicate would have +nothing to read. + + +Swap `row_number()` for `rank()` when ties should both survive. `row_number()` +always picks exactly one; `rank()` gives tied rows the same rank, so `rank = 1` +can return several. + + +## A fixed number per class {#per-group} + +Class imbalance found by [profiling](/curation/aggregate#balance) is fixed here: +rank within each class, then take the top N. + +```sql +WITH ranked AS ( + SELECT *, row_number() OVER (PARTITION BY label ORDER BY score DESC) AS rn + FROM images +) +SELECT * FROM ranked WHERE rn <= 1000 +``` + +That gives at most 1,000 rows per label, taking the highest-scoring ones. Order +by something arbitrary but stable if you want an unbiased draw instead, or +combine with [sampling](/curation/sampling) to randomize within each class. + +## Splitting a dataset {#splits} + +`ntile` divides each partition into equal buckets, which is a train/test split +that stays balanced per class: + +```sql +WITH split AS ( + SELECT *, ntile(10) OVER (PARTITION BY label ORDER BY id) AS bucket + FROM images +) +SELECT * FROM split WHERE bucket <= 8 -- train +``` + +Buckets 9 and 10 are your validation and test sets. Because the partition is by +label, each split keeps the class proportions of the whole. + + +Order by something stable. Ordering by a value that changes — a score you +recompute, a timestamp that gets updated — means the split changes with it, and +a row can move from train to test between runs. + + +## Comparing a row to its neighbours {#offset} + +`lag` and `lead` read the previous and next row in the window, which is how you +find gaps and changes without a self-join: + +```sql +SELECT + id, + created_at, + created_at - lag(created_at) OVER (ORDER BY created_at) AS gap +FROM events +``` + +Useful for spotting ingestion gaps, duplicate timestamps, and runs of identical +values that suggest a broken upstream job. + +## Running totals {#running} + +Any [aggregate](/curation/aggregate) becomes a window function with `OVER`, +which is how cumulative coverage is expressed: + +```sql +SELECT + label, + count(*) AS n, + sum(count(*)) OVER (ORDER BY count(*) DESC) AS cumulative +FROM images +GROUP BY label +``` + +Reading down the `cumulative` column tells you how many classes cover most of +your data — usually far fewer than you expect. + +## Cost {#cost} + +Window functions sort within each partition, so on a large table they are among +the more expensive things you can ask for. Filter first, partition on something +selective, and check the plan with +[Explain Analyze](/search/explain). + +## Reference {#reference} + +The full window syntax, framing, and function list are in +[Window functions](/sql/window). diff --git a/docs/web/docs.json b/docs/web/docs.json index 25c7133ca..2360b00cb 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -162,32 +162,11 @@ "basics/agentic-skills" ] }, - { - "group": "Management", - "pages": [ - "management/index", - "management/database", - "management/namespace", - "management/table", - "tables/multimodal", - "tables/branching", - "management/materialized-view", - "management/function", - { - "group": "Indexing", - "pages": [ - "indexing/index", - "indexing/vector-index", - "indexing/fts-index", - "indexing/scalar-index" - ] - } - ] - }, { "group": "Ingestion", "pages": [ "ingestion/index", + "ingestion/create-table", "ingestion/insert", "ingestion/merge-insert", "ingestion/update", @@ -206,6 +185,16 @@ "search/explain" ] }, + { + "group": "Data Curation", + "pages": [ + "curation/index", + "curation/aggregate", + "curation/sampling", + "curation/join", + "curation/windowing" + ] + }, { "group": "Feature Engineering", "pages": [ @@ -224,10 +213,32 @@ "training/torch" ] }, + { + "group": "Management", + "pages": [ + "management/index", + "management/database", + "management/namespace", + "management/table", + "tables/multimodal", + "tables/branching", + "management/materialized-view", + "management/function" + ] + }, { "group": "Operations", "pages": [ - "indexing/reindexing", + { + "group": "Indexing", + "pages": [ + "indexing/index", + "indexing/vector-index", + "indexing/fts-index", + "indexing/scalar-index", + "indexing/reindexing" + ] + }, "maintenance/compaction", { "group": "Storage", diff --git a/docs/web/indexing/reindexing.mdx b/docs/web/indexing/reindexing.mdx index d55ebf9bd..e427a328e 100644 --- a/docs/web/indexing/reindexing.mdx +++ b/docs/web/indexing/reindexing.mdx @@ -1,6 +1,6 @@ --- title: "Building and repairing indexes" -sidebarTitle: "Indexing" +sidebarTitle: "Build & Repair" description: "Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing." icon: "refresh" keywords: ["reindexing", "re-index", "incremental indexing", "reindex"] diff --git a/docs/web/ingestion/create-table.mdx b/docs/web/ingestion/create-table.mdx new file mode 100644 index 000000000..fce23ac65 --- /dev/null +++ b/docs/web/ingestion/create-table.mdx @@ -0,0 +1,347 @@ +--- +title: "Creating a table" +sidebarTitle: "Create Table" +description: "Start here: turn the data you already have into a LanceDB table." +icon: table +keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"] +--- + +import { + TsConnect, + RsConnect, +} from '/snippets/connection.mdx'; +import { + PyCreateTableFromDicts as CreateTableFromDicts, + TsCreateTableFromDicts as TsCreateTableFromDicts, + RsCreateTableFromDicts as RsCreateTableFromDicts, + PyCreateTableConflictHandling as CreateTableConflictHandling, + TsCreateTableConflictHandling as TsCreateTableConflictHandling, + RsCreateTableConflictHandling as RsCreateTableConflictHandling, + PyCreateTableFromPandas as CreateTableFromPandas, + PyCreateTableCustomSchema as CreateTableCustomSchema, + TsCreateTableCustomSchema as TsCreateTableCustomSchema, + RsCreateTableCustomSchema as RsCreateTableCustomSchema, + PyCreateTableFromPolars as CreateTableFromPolars, + PyCreateTableFromArrow as CreateTableFromArrow, + TsCreateTableFromArrow as TsCreateTableFromArrow, + RsCreateTableFromArrow as RsCreateTableFromArrow, + PyCreateTableFromPydantic as CreateTableFromPydantic, + PyCreateTableNestedSchema as CreateTableNestedSchema, + PyOpenExistingTable as OpenExistingTable, + TsOpenExistingTable as TsOpenExistingTable, + RsOpenExistingTable as RsOpenExistingTable, + PyCreateEmptyTable as CreateEmptyTable, + TsCreateEmptyTable as TsCreateEmptyTable, + RsCreateEmptyTable as RsCreateEmptyTable, + PyCreateEmptyTablePydantic as CreateEmptyTablePydantic, + PyDropTable as DropTable, + TsDropTable as TsDropTable, + RsDropTable as RsDropTable, + PyTablesBasicConnect as TablesBasicConnect, + PyTablesDocumentModel as TablesDocumentModel, + PyTablesTzValidator as TablesTzValidator, +} from '/snippets/tables.mdx'; + +A table is created from data you already have -- a list of objects, an Arrow +table, a pandas or polars frame, Pydantic models -- or from a schema alone when +the rows come later. + +## Create a table with data {#create-a-table-with-data} + +Initialize a LanceDB connection and create a table + + + + {TablesBasicConnect} + + + + {TsConnect} + + + + {RsConnect} + + + +Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns. + +### From list of objects {#from-list-of-objects} + +You can provide a list of objects to create a table. The Python and TypeScript SDKs +support lists/arrays of dictionaries, while the Rust SDK supports lists of structs. +In Python, pass a list or other batch-like object; a single bare `dict` or single +`LanceModel` is rejected. + + + + {CreateTableFromDicts} + + + + {TsCreateTableFromDicts} + + + + {RsCreateTableFromDicts} + + + +### Handle existing tables {#handle-existing-tables} + +By default, `create_table` raises an error if a table with the same name already exists. +You can change this behavior with two parameters that resolve the conflict in different ways: + +- **Idempotent open**: return the existing table without modifying it. Use when your + code may run more than once (notebooks, retries, init scripts) and you want to reuse + the table on subsequent runs. The provided data is ignored, but the schema is + validated against the existing table and a mismatch raises an error. +- **Overwrite**: drop the existing table and create a new one with the provided data. + Use this for test fixtures or when you intentionally want to replace prior contents. + This permanently discards the old table's data. + + + + {CreateTableConflictHandling} + + + + {TsCreateTableConflictHandling} + + + + {RsCreateTableConflictHandling} + + + + +`exist_ok` / `existOk` does not append the provided data to an existing table. Use +[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and* +contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table). + + +### From a custom schema {#from-a-custom-schema} + +You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata. + + + + {CreateTableCustomSchema} + + + + {TsCreateTableCustomSchema} + + + + {RsCreateTableCustomSchema} + + + +An explicit schema is also where you control nullability. If later writes omit a +non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be +omitted or written with null values. Without an explicit schema, Python infers list-like vector +values as fixed-size `float32` vector fields from the observed dimension. + +For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length, +null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes +`fill_value`, and `"null"` writes nulls. + +### From an Arrow Table {#from-an-arrow-table} +You can also create LanceDB tables directly from Arrow tables. +Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow. + + + + {CreateTableFromArrow} + + + + {TsCreateTableFromArrow} + + + + {RsCreateTableFromArrow} + + + + +### From a Pandas DataFrame {#from-a-pandas-dataframe} +Python Only + + + + {CreateTableFromPandas} + + + + +Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly. + + + +The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type. + + +### From a Polars DataFrame {#from-a-polars-dataframe} +Python Only + +LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library +written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow +under the hood. A deeper integration between LanceDB Tables and Polars DataFrames +is on the way. + + + + {CreateTableFromPolars} + + + +### From Pydantic Models {#from-pydantic-models} +Python Only + +When you create an empty table without data, you must specify the table schema. +LanceDB supports creating tables by specifying a PyArrow schema or a specialized +Pydantic model called `LanceModel`. + +For example, the following Content model specifies a table with 5 columns: +`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can +pass the class as the value of the `schema` parameter to `create_table`. +The `vector` column is a `Vector` type, which is a specialized Pydantic type that +can be configured with the vector dimensions. It is also important to note that +LanceDB only understands subclasses of `lancedb.pydantic.LanceModel` +(which itself derives from `pydantic.BaseModel`). + + + + {CreateTableFromPydantic} + + + +#### Nested schemas {#nested-schemas} + +Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object: + + + + {TablesDocumentModel} + + + +This can be used as the type of a LanceDB table column: + + + + {CreateTableNestedSchema} + + + +This creates a struct column called "document" that has two subfields +called "content" and "source": + +```bash +In [28]: tbl.schema +Out[28]: +id: string not null +vector: fixed_size_list[1536] not null + child 0, item: float +document: struct not null + child 0, content: string not null + child 1, source: string not null +``` + +#### Validators {#validators} + +Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's +[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example +below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used +for a `created_at` field. + + + + {TablesTzValidator} + + + +When you run this code it, should raise the `ValidationError`. + + +Datasets too large to pass to `create_table` at once are covered separately -- +see [Insert data](/ingestion/insert#large-datasets), which explains why +`table.add()` parallelizes where `create_table` does not. + + +## Create empty table {#create-empty-table} +You can create an empty table for scenarios where you want to add data to the table later. +An example would be when you want to collect data from a stream/external file and then add it to a table in +batches. + +An empty table can be initialized via an Arrow schema. + + + + {CreateEmptyTable} + + + + {TsCreateEmptyTable} + + + + {RsCreateEmptyTable} + + + +Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not +directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel` +that has been extended to support LanceDB specific types like `Vector`. + + + + {CreateEmptyTablePydantic} + + + +Once the empty table has been created, you can append to it or modify its contents, +as explained in [Ingesting data](/ingestion). + +## Open an existing table {#open-an-existing-table} + +You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method. +If you forget the name of your table, you can always get a listing of all table names. + + + + {OpenExistingTable} + + + + {TsOpenExistingTable} + + + + {RsOpenExistingTable} + + + +## Drop a table {#drop-a-table} + +Use the `drop_table()` method on the database to remove a table. + + + + {DropTable} + + + + {TsDropTable} + + + + {RsDropTable} + + + +This permanently removes the table and is not recoverable, unlike deleting rows. +By default, if the table does not exist an exception is raised. To suppress this, +you can pass in `ignore_missing=True`. diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx index 504810758..1a8a94955 100644 --- a/docs/web/ingestion/index.mdx +++ b/docs/web/ingestion/index.mdx @@ -7,44 +7,6 @@ keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars --- import { - TsConnect, - RsConnect, -} from '/snippets/connection.mdx'; -import { - PyCreateTableFromDicts as CreateTableFromDicts, - TsCreateTableFromDicts as TsCreateTableFromDicts, - RsCreateTableFromDicts as RsCreateTableFromDicts, - PyCreateTableConflictHandling as CreateTableConflictHandling, - TsCreateTableConflictHandling as TsCreateTableConflictHandling, - RsCreateTableConflictHandling as RsCreateTableConflictHandling, - PyCreateTableFromPandas as CreateTableFromPandas, - PyCreateTableCustomSchema as CreateTableCustomSchema, - TsCreateTableCustomSchema as TsCreateTableCustomSchema, - RsCreateTableCustomSchema as RsCreateTableCustomSchema, - PyCreateTableFromPolars as CreateTableFromPolars, - PyCreateTableFromArrow as CreateTableFromArrow, - TsCreateTableFromArrow as TsCreateTableFromArrow, - RsCreateTableFromArrow as RsCreateTableFromArrow, - PyCreateTableFromPydantic as CreateTableFromPydantic, - PyCreateTableNestedSchema as CreateTableNestedSchema, - PyAddFromDataset as AddFromDataset, - PyCreateTableFromIterator as CreateTableFromIterator, - TsCreateTableFromIterator as TsCreateTableFromIterator, - RsCreateTableFromIterator as RsCreateTableFromIterator, - TsAddProgress, - PyOpenExistingTable as OpenExistingTable, - TsOpenExistingTable as TsOpenExistingTable, - RsOpenExistingTable as RsOpenExistingTable, - PyCreateEmptyTable as CreateEmptyTable, - TsCreateEmptyTable as TsCreateEmptyTable, - RsCreateEmptyTable as RsCreateEmptyTable, - PyCreateEmptyTablePydantic as CreateEmptyTablePydantic, - PyDropTable as DropTable, - TsDropTable as TsDropTable, - RsDropTable as RsDropTable, - PyTablesBasicConnect as TablesBasicConnect, - PyTablesDocumentModel as TablesDocumentModel, - PyTablesTzValidator as TablesTzValidator, PyUpdateConnectEnterprise as UpdateConnectEnterprise, PyUpdateConnectLocal as UpdateConnectLocal, PyUpdateExampleTableSetup as UpdateExampleTableSetup, @@ -71,305 +33,8 @@ second. Every operation here has a SQL form -- see [DML](/sql/dml). -## Create a table with data {#create-a-table-with-data} - -Initialize a LanceDB connection and create a table - - - - {TablesBasicConnect} - - - - {TsConnect} - - - - {RsConnect} - - - -Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns. - -### From list of objects {#from-list-of-objects} - -You can provide a list of objects to create a table. The Python and TypeScript SDKs -support lists/arrays of dictionaries, while the Rust SDK supports lists of structs. -In Python, pass a list or other batch-like object; a single bare `dict` or single -`LanceModel` is rejected. - - - - {CreateTableFromDicts} - - - - {TsCreateTableFromDicts} - - - - {RsCreateTableFromDicts} - - - -### Handle existing tables {#handle-existing-tables} - -By default, `create_table` raises an error if a table with the same name already exists. -You can change this behavior with two parameters that resolve the conflict in different ways: - -- **Idempotent open**: return the existing table without modifying it. Use when your - code may run more than once (notebooks, retries, init scripts) and you want to reuse - the table on subsequent runs. The provided data is ignored, but the schema is - validated against the existing table and a mismatch raises an error. -- **Overwrite**: drop the existing table and create a new one with the provided data. - Use this for test fixtures or when you intentionally want to replace prior contents. - This permanently discards the old table's data. - - - - {CreateTableConflictHandling} - - - - {TsCreateTableConflictHandling} - - - - {RsCreateTableConflictHandling} - - - - -`exist_ok` / `existOk` does not append the provided data to an existing table. Use -[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and* -contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table). - - -### From a custom schema {#from-a-custom-schema} - -You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata. - - - - {CreateTableCustomSchema} - - - - {TsCreateTableCustomSchema} - - - - {RsCreateTableCustomSchema} - - - -An explicit schema is also where you control nullability. If later writes omit a -non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be -omitted or written with null values. Without an explicit schema, Python infers list-like vector -values as fixed-size `float32` vector fields from the observed dimension. - -For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length, -null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes -`fill_value`, and `"null"` writes nulls. - -### From an Arrow Table {#from-an-arrow-table} -You can also create LanceDB tables directly from Arrow tables. -Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow. - - - - {CreateTableFromArrow} - - - - {TsCreateTableFromArrow} - - - - {RsCreateTableFromArrow} - - - - -### From a Pandas DataFrame {#from-a-pandas-dataframe} -Python Only - - - - {CreateTableFromPandas} - - - - -Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly. - - - -The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type. - - -### From a Polars DataFrame {#from-a-polars-dataframe} -Python Only - -LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library -written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow -under the hood. A deeper integration between LanceDB Tables and Polars DataFrames -is on the way. - - - - {CreateTableFromPolars} - - - -### From Pydantic Models {#from-pydantic-models} -Python Only - -When you create an empty table without data, you must specify the table schema. -LanceDB supports creating tables by specifying a PyArrow schema or a specialized -Pydantic model called `LanceModel`. - -For example, the following Content model specifies a table with 5 columns: -`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can -pass the class as the value of the `schema` parameter to `create_table`. -The `vector` column is a `Vector` type, which is a specialized Pydantic type that -can be configured with the vector dimensions. It is also important to note that -LanceDB only understands subclasses of `lancedb.pydantic.LanceModel` -(which itself derives from `pydantic.BaseModel`). - - - - {CreateTableFromPydantic} - - - -#### Nested schemas {#nested-schemas} - -Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object: - - - - {TablesDocumentModel} - - - -This can be used as the type of a LanceDB table column: - - - - {CreateTableNestedSchema} - - - -This creates a struct column called "document" that has two subfields -called "content" and "source": - -```bash -In [28]: tbl.schema -Out[28]: -id: string not null -vector: fixed_size_list[1536] not null - child 0, item: float -document: struct not null - child 0, content: string not null - child 1, source: string not null -``` - -#### Validators {#validators} - -Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's -[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example -below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used -for a `created_at` field. - - - - {TablesTzValidator} - - - -When you run this code it, should raise the `ValidationError`. - - -Datasets too large to pass to `create_table` at once are covered separately -- -see [Insert data](/ingestion/insert#large-datasets), which explains why -`table.add()` parallelizes where `create_table` does not. - - -## Create empty table {#create-empty-table} -You can create an empty table for scenarios where you want to add data to the table later. -An example would be when you want to collect data from a stream/external file and then add it to a table in -batches. - -An empty table can be initialized via an Arrow schema. - - - - {CreateEmptyTable} - - - - {TsCreateEmptyTable} - - - - {RsCreateEmptyTable} - - - -Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not -directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel` -that has been extended to support LanceDB specific types like `Vector`. - - - - {CreateEmptyTablePydantic} - - - -Once the empty table has been created, you can append to it or modify its contents, -as explained in [Ingesting data](/ingestion). - -## Open an existing table {#open-an-existing-table} - -You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method. -If you forget the name of your table, you can always get a listing of all table names. - - - - {OpenExistingTable} - - - - {TsOpenExistingTable} - - - - {RsOpenExistingTable} - - - -## Drop a table {#drop-a-table} - -Use the `drop_table()` method on the database to remove a table. - - - - {DropTable} - - - - {TsDropTable} - - - - {RsDropTable} - - - -This permanently removes the table and is not recoverable, unlike deleting rows. -By default, if the table does not exist an exception is raised. To suppress this, -you can pass in `ignore_missing=True`. +Start by [creating a table](/ingestion/create-table); the rest of this +section writes to one that exists. ## Choose a write method {#choose-a-write-method}