mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
docs: move versioning and branching under Table, split schema from data evolution
Versioning and branching are properties of a table, not steps in getting data in, so they join the Table group. Consistency stays with Ingestion, where the question it answers -- when do other writers' rows become visible -- is asked. The Ingesting Data page was in the Table group while an Ingestion overview existed separately, so the two are merged. The large-dataset section was duplicated between it and the insert page; the insert page keeps it. Schema and Data Evolution covered two different things under one title. Changing a table's shape stays in Table management; deriving values from data already there becomes a Feature engineering page, next to computed columns and functions, which are the same job at larger scale.
This commit is contained in:
Jack Ye
+8
-4
@@ -70,6 +70,10 @@
|
||||
{
|
||||
"source": "/feature-engineering/functions",
|
||||
"destination": "/management/function"
|
||||
},
|
||||
{
|
||||
"source": "/tables/create",
|
||||
"destination": "/ingestion"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -110,9 +114,10 @@
|
||||
"group": "Table",
|
||||
"pages": [
|
||||
"tables/index",
|
||||
"tables/create",
|
||||
"tables/schema",
|
||||
"tables/multimodal"
|
||||
"tables/multimodal",
|
||||
"tables/versioning",
|
||||
"tables/branching"
|
||||
]
|
||||
},
|
||||
"management/function"
|
||||
@@ -126,8 +131,6 @@
|
||||
"ingestion/merge-insert",
|
||||
"ingestion/update",
|
||||
"ingestion/delete",
|
||||
"tables/versioning",
|
||||
"tables/branching",
|
||||
"tables/consistency"
|
||||
]
|
||||
},
|
||||
@@ -169,6 +172,7 @@
|
||||
"group": "Feature engineering",
|
||||
"pages": [
|
||||
"feature-engineering/index",
|
||||
"feature-engineering/data-evolution",
|
||||
"feature-engineering/add-computed-column",
|
||||
"feature-engineering/refresh-column",
|
||||
"feature-engineering/materialized-views"
|
||||
|
||||
@@ -59,7 +59,7 @@ For best results:
|
||||
- **Avoid inserting one row at a time.** Each insert creates a new data fragment on
|
||||
disk. Batch your data into Arrow tables, DataFrames, or use iterators.
|
||||
|
||||
See [Loading Large Datasets](/tables/create#loading-large-datasets) for full examples.
|
||||
See [Ingesting large datasets](/ingestion/insert#large-datasets) for full examples.
|
||||
|
||||
### Do I need to set a refine factor when using an index? {#do-i-need-to-set-a-refine-factor-when-using-an-index}
|
||||
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: "Data evolution"
|
||||
sidebarTitle: "Data Evolution"
|
||||
description: "Derive new columns from data already in the table, without rewriting what is already there."
|
||||
icon: wand-magic-sparkles
|
||||
keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
|
||||
---
|
||||
|
||||
import {
|
||||
PySchemaAddSetup as SchemaAddSetup,
|
||||
TsSchemaAddSetup as TsSchemaAddSetup,
|
||||
RsSchemaAddSetup as RsSchemaAddSetup,
|
||||
PyAddColumnsCalculated as AddColumnsCalculated,
|
||||
TsAddColumnsCalculated as TsAddColumnsCalculated,
|
||||
RsAddColumnsCalculated as RsAddColumnsCalculated,
|
||||
PyAddFeatureColumnsSql as AddFeatureColumnsSql,
|
||||
TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
|
||||
RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
Data evolution is adding **values** derived from what a table already holds.
|
||||
[Schema evolution](/tables/schema) changes the table's shape; this fills the
|
||||
shape with something new.
|
||||
|
||||
The reason it is cheap is the format. Adding a derived column writes only that
|
||||
column -- the existing data, which is the expensive part on a multimodal table,
|
||||
is never rewritten. See [why that works](/basics/lance#evolution).
|
||||
|
||||
## Set up the example table {#set-up-the-example-table}
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSchemaAddSetup}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Add derived columns {#add-derived-columns}
|
||||
|
||||
You can add new columns that are derived from existing data using SQL expressions.
|
||||
For feature engineering on large existing tables, group related derived features into
|
||||
one `add_columns` operation instead of running many separate writes. This creates one
|
||||
new table version for the schema change and computes the new columns from the existing
|
||||
rows, which avoids growing the table's version history with many small updates.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
The same call can add multiple derived columns at once. For example, if you are
|
||||
building several lightweight features from existing product fields, pass all of the
|
||||
new column expressions together:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Info>
|
||||
LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
|
||||
PyArrow `RecordBatch` iterators for populating new columns. New column values must be
|
||||
defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
|
||||
If your transformation cannot be expressed in SQL, compute the values outside
|
||||
`add_columns` before writing them back through another workflow.
|
||||
</Info>
|
||||
|
||||
## When SQL is not enough {#beyond-sql}
|
||||
|
||||
The expressions above are evaluated once, as part of the call. Two other shapes
|
||||
exist when that does not fit:
|
||||
|
||||
- A [computed column](/feature-engineering/add-computed-column) declares the
|
||||
expression and leaves the column empty, so you choose when the work happens.
|
||||
That is the right default on a large table.
|
||||
- A [function](/management/function) fills a column with Python, for values SQL
|
||||
cannot express -- an embedding, a model call, an image decode.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
|
||||
Declare now, fill later.
|
||||
</Card>
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/tables/schema">
|
||||
Change the table's shape instead of its contents.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,11 +1,50 @@
|
||||
---
|
||||
title: "Ingesting data"
|
||||
sidebarTitle: "Overview"
|
||||
description: "The four write operations, and the example table the rest of this section uses."
|
||||
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
|
||||
icon: inbox
|
||||
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
||||
---
|
||||
|
||||
import {
|
||||
TsConnect,
|
||||
RsConnect,
|
||||
} from '/snippets/connection.mdx';
|
||||
import {
|
||||
PyCreateTableFromDicts as CreateTableFromDicts,
|
||||
TsCreateTableFromDicts as TsCreateTableFromDicts,
|
||||
RsCreateTableFromDicts as RsCreateTableFromDicts,
|
||||
PyCreateTableConflictHandling as CreateTableConflictHandling,
|
||||
TsCreateTableConflictHandling as TsCreateTableConflictHandling,
|
||||
RsCreateTableConflictHandling as RsCreateTableConflictHandling,
|
||||
PyCreateTableFromPandas as CreateTableFromPandas,
|
||||
PyCreateTableCustomSchema as CreateTableCustomSchema,
|
||||
TsCreateTableCustomSchema as TsCreateTableCustomSchema,
|
||||
RsCreateTableCustomSchema as RsCreateTableCustomSchema,
|
||||
PyCreateTableFromPolars as CreateTableFromPolars,
|
||||
PyCreateTableFromArrow as CreateTableFromArrow,
|
||||
TsCreateTableFromArrow as TsCreateTableFromArrow,
|
||||
RsCreateTableFromArrow as RsCreateTableFromArrow,
|
||||
PyCreateTableFromPydantic as CreateTableFromPydantic,
|
||||
PyCreateTableNestedSchema as CreateTableNestedSchema,
|
||||
PyAddFromDataset as AddFromDataset,
|
||||
PyCreateTableFromIterator as CreateTableFromIterator,
|
||||
TsCreateTableFromIterator as TsCreateTableFromIterator,
|
||||
RsCreateTableFromIterator as RsCreateTableFromIterator,
|
||||
TsAddProgress,
|
||||
PyOpenExistingTable as OpenExistingTable,
|
||||
TsOpenExistingTable as TsOpenExistingTable,
|
||||
RsOpenExistingTable as RsOpenExistingTable,
|
||||
PyCreateEmptyTable as CreateEmptyTable,
|
||||
TsCreateEmptyTable as TsCreateEmptyTable,
|
||||
RsCreateEmptyTable as RsCreateEmptyTable,
|
||||
PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
|
||||
PyDropTable as DropTable,
|
||||
TsDropTable as TsDropTable,
|
||||
RsDropTable as RsDropTable,
|
||||
PyTablesBasicConnect as TablesBasicConnect,
|
||||
PyTablesDocumentModel as TablesDocumentModel,
|
||||
PyTablesTzValidator as TablesTzValidator,
|
||||
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
|
||||
PyUpdateConnectLocal as UpdateConnectLocal,
|
||||
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
|
||||
@@ -18,6 +57,316 @@ import {
|
||||
RsUpdateMakeUsersReader,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
In LanceDB, tables store records with a defined schema that specifies column
|
||||
names and types. Across the SDKs, you can create tables from row-oriented data
|
||||
and Apache Arrow data structures. The Python SDK additionally supports PyArrow
|
||||
schemas for explicit schema control, and `LanceModel` for Pydantic-based
|
||||
validation.
|
||||
|
||||
Getting data in has two halves: creating a table from data you already have, and
|
||||
writing to a table that exists. This page covers the first and introduces the
|
||||
second.
|
||||
|
||||
## Create a table with data {#create-a-table-with-data}
|
||||
|
||||
Initialize a LanceDB connection and create a table
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesBasicConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsConnect}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
|
||||
|
||||
### From list of objects {#from-list-of-objects}
|
||||
|
||||
You can provide a list of objects to create a table. The Python and TypeScript SDKs
|
||||
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
|
||||
In Python, pass a list or other batch-like object; a single bare `dict` or single
|
||||
`LanceModel` is rejected.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Handle existing tables {#handle-existing-tables}
|
||||
|
||||
By default, `create_table` raises an error if a table with the same name already exists.
|
||||
You can change this behavior with two parameters that resolve the conflict in different ways:
|
||||
|
||||
- **Idempotent open**: return the existing table without modifying it. Use when your
|
||||
code may run more than once (notebooks, retries, init scripts) and you want to reuse
|
||||
the table on subsequent runs. The provided data is ignored, but the schema is
|
||||
validated against the existing table and a mismatch raises an error.
|
||||
- **Overwrite**: drop the existing table and create a new one with the provided data.
|
||||
Use this for test fixtures or when you intentionally want to replace prior contents.
|
||||
This permanently discards the old table's data.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
|
||||
[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
|
||||
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
|
||||
</Note>
|
||||
|
||||
### From a custom schema {#from-a-custom-schema}
|
||||
|
||||
You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
An explicit schema is also where you control nullability. If later writes omit a
|
||||
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
|
||||
omitted or written with null values. Without an explicit schema, Python infers list-like vector
|
||||
values as fixed-size `float32` vector fields from the observed dimension.
|
||||
|
||||
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
|
||||
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
|
||||
`fill_value`, and `"null"` writes nulls.
|
||||
|
||||
### From an Arrow Table {#from-an-arrow-table}
|
||||
You can also create LanceDB tables directly from Arrow tables.
|
||||
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
### From a Pandas DataFrame {#from-a-pandas-dataframe}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note title="Note">
|
||||
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
|
||||
</Note>
|
||||
|
||||
<Note title="Vector Column Type">
|
||||
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
|
||||
</Note>
|
||||
|
||||
### From a Polars DataFrame {#from-a-polars-dataframe}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
|
||||
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
|
||||
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
|
||||
is on the way.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPolars}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### From Pydantic Models {#from-pydantic-models}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
When you create an empty table without data, you must specify the table schema.
|
||||
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
|
||||
Pydantic model called `LanceModel`.
|
||||
|
||||
For example, the following Content model specifies a table with 5 columns:
|
||||
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
|
||||
pass the class as the value of the `schema` parameter to `create_table`.
|
||||
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
|
||||
can be configured with the vector dimensions. It is also important to note that
|
||||
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
|
||||
(which itself derives from `pydantic.BaseModel`).
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPydantic}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Nested schemas {#nested-schemas}
|
||||
|
||||
Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesDocumentModel}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This can be used as the type of a LanceDB table column:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableNestedSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This creates a struct column called "document" that has two subfields
|
||||
called "content" and "source":
|
||||
|
||||
```bash
|
||||
In [28]: tbl.schema
|
||||
Out[28]:
|
||||
id: string not null
|
||||
vector: fixed_size_list<item: float>[1536] not null
|
||||
child 0, item: float
|
||||
document: struct<content: string not null, source: string not null> not null
|
||||
child 0, content: string not null
|
||||
child 1, source: string not null
|
||||
```
|
||||
|
||||
#### Validators {#validators}
|
||||
|
||||
Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
|
||||
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
|
||||
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
|
||||
for a `created_at` field.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesTzValidator}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
When you run this code it, should raise the `ValidationError`.
|
||||
|
||||
<Note>
|
||||
Datasets too large to pass to `create_table` at once are covered separately --
|
||||
see [Insert data](/ingestion/insert#large-datasets), which explains why
|
||||
`table.add()` parallelizes where `create_table` does not.
|
||||
</Note>
|
||||
|
||||
## Create empty table {#create-empty-table}
|
||||
You can create an empty table for scenarios where you want to add data to the table later.
|
||||
An example would be when you want to collect data from a stream/external file and then add it to a table in
|
||||
batches.
|
||||
|
||||
An empty table can be initialized via an Arrow schema.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
|
||||
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
|
||||
that has been extended to support LanceDB specific types like `Vector`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateEmptyTablePydantic}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Once the empty table has been created, you can append to it or modify its contents,
|
||||
as explained in [Ingesting data](/ingestion).
|
||||
|
||||
## Open an existing table {#open-an-existing-table}
|
||||
|
||||
You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
|
||||
If you forget the name of your table, you can always get a listing of all table names.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{OpenExistingTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsOpenExistingTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsOpenExistingTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Drop a table {#drop-a-table}
|
||||
|
||||
Use the `drop_table()` method on the database to remove a table.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This permanently removes the table and is not recoverable, unlike deleting rows.
|
||||
By default, if the table does not exist an exception is raised. To suppress this,
|
||||
you can pass in `ignore_missing=True`.
|
||||
|
||||
## Choose a write method {#choose-a-write-method}
|
||||
|
||||
| Family | Method | Use this when... |
|
||||
@@ -104,4 +453,4 @@ Expected table contents:
|
||||
| 2 | Bob | 20 |
|
||||
|
||||
The example above shows a PyArrow schema. You can just as well create the table using other
|
||||
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Creating a table](/tables/create) for more details.
|
||||
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [above](#create-a-table-with-data).
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: plus
|
||||
---
|
||||
|
||||
Rows are appended with `table.add()`. It accepts the same shapes
|
||||
[table creation](/tables/create) does — lists of objects, Arrow tables, pandas
|
||||
[table creation](/ingestion) does — lists of objects, Arrow tables, pandas
|
||||
and polars frames, Pydantic models — and commits a new table version.
|
||||
|
||||
These examples use the table set up in [Ingesting data](/ingestion).
|
||||
|
||||
@@ -64,7 +64,7 @@ well as on LanceDB Enterprise (remote) tables.
|
||||
|
||||
The branch API is identical no matter how you connect — only the connection
|
||||
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
|
||||
a `table` (see [Create a table](/tables/create)), then use the same branch calls
|
||||
a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
|
||||
in every example that follows.
|
||||
|
||||
### LanceDB OSS {#lancedb-oss}
|
||||
|
||||
@@ -1,442 +0,0 @@
|
||||
---
|
||||
title: Ingesting Data
|
||||
sidebarTitle: "Ingesting data"
|
||||
description: Learn about different methods to ingest data into tables in LanceDB, including from various data sources and empty tables.
|
||||
icon: "cookie"
|
||||
---
|
||||
import { TsConnect, RsConnect } from '/snippets/connection.mdx';
|
||||
import {
|
||||
PyCreateTableFromDicts as CreateTableFromDicts,
|
||||
TsCreateTableFromDicts as TsCreateTableFromDicts,
|
||||
RsCreateTableFromDicts as RsCreateTableFromDicts,
|
||||
PyCreateTableConflictHandling as CreateTableConflictHandling,
|
||||
TsCreateTableConflictHandling as TsCreateTableConflictHandling,
|
||||
RsCreateTableConflictHandling as RsCreateTableConflictHandling,
|
||||
PyCreateTableFromPandas as CreateTableFromPandas,
|
||||
PyCreateTableCustomSchema as CreateTableCustomSchema,
|
||||
TsCreateTableCustomSchema as TsCreateTableCustomSchema,
|
||||
RsCreateTableCustomSchema as RsCreateTableCustomSchema,
|
||||
PyCreateTableFromPolars as CreateTableFromPolars,
|
||||
PyCreateTableFromArrow as CreateTableFromArrow,
|
||||
TsCreateTableFromArrow as TsCreateTableFromArrow,
|
||||
RsCreateTableFromArrow as RsCreateTableFromArrow,
|
||||
PyCreateTableFromPydantic as CreateTableFromPydantic,
|
||||
PyCreateTableNestedSchema as CreateTableNestedSchema,
|
||||
PyAddFromDataset as AddFromDataset,
|
||||
PyCreateTableFromIterator as CreateTableFromIterator,
|
||||
TsCreateTableFromIterator as TsCreateTableFromIterator,
|
||||
RsCreateTableFromIterator as RsCreateTableFromIterator,
|
||||
TsAddProgress,
|
||||
PyOpenExistingTable as OpenExistingTable,
|
||||
TsOpenExistingTable as TsOpenExistingTable,
|
||||
RsOpenExistingTable as RsOpenExistingTable,
|
||||
PyCreateEmptyTable as CreateEmptyTable,
|
||||
TsCreateEmptyTable as TsCreateEmptyTable,
|
||||
RsCreateEmptyTable as RsCreateEmptyTable,
|
||||
PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
|
||||
PyDropTable as DropTable,
|
||||
TsDropTable as TsDropTable,
|
||||
RsDropTable as RsDropTable,
|
||||
PyTablesBasicConnect as TablesBasicConnect,
|
||||
PyTablesDocumentModel as TablesDocumentModel,
|
||||
PyTablesTzValidator as TablesTzValidator,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
In LanceDB, tables store records with a defined schema that specifies column names and types. Across the SDKs, you can create tables from row-oriented data and Apache Arrow data structures. The Python SDK additionally supports:
|
||||
|
||||
- PyArrow schemas for explicit schema control
|
||||
- `LanceModel` for Pydantic-based validation
|
||||
|
||||
## Create a table with data {#create-a-table-with-data}
|
||||
|
||||
Initialize a LanceDB connection and create a table
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesBasicConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsConnect}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
|
||||
|
||||
### From list of objects {#from-list-of-objects}
|
||||
|
||||
You can provide a list of objects to create a table. The Python and TypeScript SDKs
|
||||
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
|
||||
In Python, pass a list or other batch-like object; a single bare `dict` or single
|
||||
`LanceModel` is rejected.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableFromDicts}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Handle existing tables {#handle-existing-tables}
|
||||
|
||||
By default, `create_table` raises an error if a table with the same name already exists.
|
||||
You can change this behavior with two parameters that resolve the conflict in different ways:
|
||||
|
||||
- **Idempotent open**: return the existing table without modifying it. Use when your
|
||||
code may run more than once (notebooks, retries, init scripts) and you want to reuse
|
||||
the table on subsequent runs. The provided data is ignored, but the schema is
|
||||
validated against the existing table and a mismatch raises an error.
|
||||
- **Overwrite**: drop the existing table and create a new one with the provided data.
|
||||
Use this for test fixtures or when you intentionally want to replace prior contents.
|
||||
This permanently discards the old table's data.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableConflictHandling}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
|
||||
[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
|
||||
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
|
||||
</Note>
|
||||
|
||||
### From a custom schema {#from-a-custom-schema}
|
||||
|
||||
You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableCustomSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
An explicit schema is also where you control nullability. If later writes omit a
|
||||
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
|
||||
omitted or written with null values. Without an explicit schema, Python infers list-like vector
|
||||
values as fixed-size `float32` vector fields from the observed dimension.
|
||||
|
||||
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
|
||||
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
|
||||
`fill_value`, and `"null"` writes nulls.
|
||||
|
||||
### From an Arrow Table {#from-an-arrow-table}
|
||||
You can also create LanceDB tables directly from Arrow tables.
|
||||
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableFromArrow}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
### From a Pandas DataFrame {#from-a-pandas-dataframe}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note title="Note">
|
||||
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
|
||||
</Note>
|
||||
|
||||
<Note title="Vector Column Type">
|
||||
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
|
||||
</Note>
|
||||
|
||||
### From a Polars DataFrame {#from-a-polars-dataframe}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
|
||||
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
|
||||
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
|
||||
is on the way.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPolars}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### From Pydantic Models {#from-pydantic-models}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
When you create an empty table without data, you must specify the table schema.
|
||||
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
|
||||
Pydantic model called `LanceModel`.
|
||||
|
||||
For example, the following Content model specifies a table with 5 columns:
|
||||
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
|
||||
pass the class as the value of the `schema` parameter to `create_table`.
|
||||
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
|
||||
can be configured with the vector dimensions. It is also important to note that
|
||||
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
|
||||
(which itself derives from `pydantic.BaseModel`).
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromPydantic}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Nested schemas {#nested-schemas}
|
||||
|
||||
Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesDocumentModel}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This can be used as the type of a LanceDB table column:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableNestedSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This creates a struct column called "document" that has two subfields
|
||||
called "content" and "source":
|
||||
|
||||
```bash
|
||||
In [28]: tbl.schema
|
||||
Out[28]:
|
||||
id: string not null
|
||||
vector: fixed_size_list<item: float>[1536] not null
|
||||
child 0, item: float
|
||||
document: struct<content: string not null, source: string not null> not null
|
||||
child 0, content: string not null
|
||||
child 1, source: string not null
|
||||
```
|
||||
|
||||
#### Validators {#validators}
|
||||
|
||||
Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
|
||||
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
|
||||
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
|
||||
for a `created_at` field.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{TablesTzValidator}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
When you run this code it, should raise the `ValidationError`.
|
||||
|
||||
### Loading Large Datasets {#loading-large-datasets}
|
||||
|
||||
When ingesting large datasets, use `table.add()` on an existing table rather than
|
||||
passing all data to `create_table()`. The `add()` method auto-parallelizes large
|
||||
writes, while `create_table(name, data)` does not.
|
||||
|
||||
<Tip>
|
||||
For best performance with large datasets, create an empty table first and then call
|
||||
`table.add()`. This enables automatic write parallelism for materialized data sources.
|
||||
</Tip>
|
||||
|
||||
#### From files (Parquet, CSV, etc.) {#from-files-parquet-csv-etc}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams
|
||||
data from disk without loading the entire dataset into memory.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddFromDataset}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
`pa.dataset()` input is currently Python-only. TypeScript and Rust support for
|
||||
file-based dataset ingestion is tracked in
|
||||
[lancedb#3173](https://github.com/lancedb/lancedb/issues/3173).
|
||||
</Note>
|
||||
|
||||
#### From iterators (custom batch generation) {#from-iterators-custom-batch-generation}
|
||||
|
||||
When you need custom batch logic — generating embeddings on the fly, transforming
|
||||
rows from an external source, etc. — use an iterator of `RecordBatch` objects.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateTableFromIterator}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateTableFromIterator}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateTableFromIterator}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Use this pattern when:
|
||||
|
||||
- Your source data already arrives in Arrow batches, readers, datasets, or streams.
|
||||
- Materializing the entire ingest as one giant in-memory list or array would be too expensive.
|
||||
- You want to control chunk size explicitly during ingestion.
|
||||
|
||||
Python can also consume iterators of other supported types like Pandas DataFrames or Python lists.
|
||||
|
||||
#### Write parallelism {#write-parallelism}
|
||||
|
||||
<Note title="Automatic parallelism">
|
||||
For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB
|
||||
automatically parallelizes large writes — no configuration needed. Auto-parallelism
|
||||
targets approximately 1M rows or 2GB per write partition.
|
||||
|
||||
For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine
|
||||
total size upfront. A `parallelism` parameter to control this manually is planned
|
||||
but not yet exposed in Python or TypeScript
|
||||
([tracking issue](https://github.com/lancedb/lancedb/issues/3173)).
|
||||
</Note>
|
||||
|
||||
#### Tracking ingestion progress {#tracking-ingestion-progress}
|
||||
<Badge color="green">TypeScript Only</Badge>
|
||||
|
||||
For long-running writes, pass a `progress` callback to `table.add()` to surface
|
||||
per-batch progress in your UI, logs, or metrics pipeline. The callback fires
|
||||
once per batch written and once more with `done: true` when the write completes.
|
||||
|
||||
Each invocation receives a `WriteProgress` object:
|
||||
|
||||
| Field | Description |
|
||||
|:------|:------------|
|
||||
| `outputRows` | Rows written so far. |
|
||||
| `outputBytes` | Bytes written so far. |
|
||||
| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. |
|
||||
| `elapsedSeconds` | Wall-clock seconds since the write started. |
|
||||
| `activeTasks` | Parallel write tasks currently in flight. |
|
||||
| `totalTasks` | Total parallel write tasks (the write parallelism). |
|
||||
| `done` | `true` only for the final callback. |
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddProgress}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
A few things to know before you wire this up:
|
||||
|
||||
- Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere.
|
||||
- Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow.
|
||||
- Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written.
|
||||
|
||||
## Create empty table {#create-empty-table}
|
||||
You can create an empty table for scenarios where you want to add data to the table later.
|
||||
An example would be when you want to collect data from a stream/external file and then add it to a table in
|
||||
batches.
|
||||
|
||||
An empty table can be initialized via an Arrow schema.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
|
||||
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
|
||||
that has been extended to support LanceDB specific types like `Vector`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateEmptyTablePydantic}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Once the empty table has been created, you can append to it or modify its contents,
|
||||
as explained in [Ingesting data](/ingestion).
|
||||
|
||||
## Open an existing table {#open-an-existing-table}
|
||||
|
||||
You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
|
||||
If you forget the name of your table, you can always get a listing of all table names.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{OpenExistingTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsOpenExistingTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsOpenExistingTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Drop a table {#drop-a-table}
|
||||
|
||||
Use the `drop_table()` method on the database to remove a table.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDropTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This permanently removes the table and is not recoverable, unlike deleting rows.
|
||||
By default, if the table does not exist an exception is raised. To suppress this,
|
||||
you can pass in `ignore_missing=True`.
|
||||
@@ -267,12 +267,12 @@ reuse the existing table instead of writing the supplied rows again.
|
||||
If you want to avoid overwriting an existing table, omit the overwrite mode. For append-only
|
||||
ingestion into a table that already exists, open the table and call `add(...)` instead of
|
||||
`create_table(...)`. For repeatable setup with `exist_ok` / `existOk`, see
|
||||
[Handle existing tables](/tables/create#handle-existing-tables).
|
||||
[Handle existing tables](/ingestion#handle-existing-tables).
|
||||
</Info>
|
||||
|
||||
For more ingestion patterns, including PyArrow tables, Python `pyarrow.dataset.Dataset` inputs,
|
||||
empty tables, and Python `LanceModel` schemas with nested fields, see
|
||||
[Ingesting data](/tables/create/).
|
||||
[Ingesting data](/ingestion).
|
||||
|
||||
### From Pandas DataFrames {#from-pandas-dataframes}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
@@ -644,7 +644,7 @@ advanced features of LanceDB. Below are some suggested next pages.
|
||||
<Card
|
||||
title="Ingesting data"
|
||||
icon="cookie"
|
||||
href="/tables/create/"
|
||||
href="/ingestion"
|
||||
>
|
||||
Learn the different approaches to creating and ingesting data into LanceDB tables from various sources.
|
||||
</Card>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Schema and Data Evolution"
|
||||
sidebarTitle: "Schema & data evolution"
|
||||
title: "Schema evolution"
|
||||
sidebarTitle: "Schema Evolution"
|
||||
description: Learn how to manage table schemas in LanceDB, including adding, altering, and dropping columns.
|
||||
icon: "boxes-stacked"
|
||||
---
|
||||
@@ -8,18 +8,12 @@ import {
|
||||
PySchemaAddSetup as SchemaAddSetup,
|
||||
TsSchemaAddSetup as TsSchemaAddSetup,
|
||||
RsSchemaAddSetup as RsSchemaAddSetup,
|
||||
PyAddColumnsCalculated as AddColumnsCalculated,
|
||||
TsAddColumnsCalculated as TsAddColumnsCalculated,
|
||||
RsAddColumnsCalculated as RsAddColumnsCalculated,
|
||||
PyAddColumnsDefaultValues as AddColumnsDefaultValues,
|
||||
TsAddColumnsDefaultValues as TsAddColumnsDefaultValues,
|
||||
RsAddColumnsDefaultValues as RsAddColumnsDefaultValues,
|
||||
PyAddColumnsNullable as AddColumnsNullable,
|
||||
TsAddColumnsNullable as TsAddColumnsNullable,
|
||||
RsAddColumnsNullable as RsAddColumnsNullable,
|
||||
PyAddFeatureColumnsSql as AddFeatureColumnsSql,
|
||||
TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
|
||||
RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
|
||||
PySchemaAlterSetup as SchemaAlterSetup,
|
||||
TsSchemaAlterSetup as TsSchemaAlterSetup,
|
||||
RsSchemaAlterSetup as RsSchemaAlterSetup,
|
||||
@@ -108,53 +102,11 @@ First, let's create a sample table with product data to demonstrate schema evolu
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Add derived columns {#add-derived-columns}
|
||||
|
||||
You can add new columns that are derived from existing data using SQL expressions.
|
||||
For feature engineering on large existing tables, group related derived features into
|
||||
one `add_columns` operation instead of running many separate writes. This creates one
|
||||
new table version for the schema change and computes the new columns from the existing
|
||||
rows, which avoids growing the table's version history with many small updates.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddColumnsCalculated}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
The same call can add multiple derived columns at once. For example, if you are
|
||||
building several lightweight features from existing product fields, pass all of the
|
||||
new column expressions together:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{AddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsAddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsAddFeatureColumnsSql}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Info>
|
||||
LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
|
||||
PyArrow `RecordBatch` iterators for populating new columns. New column values must be
|
||||
defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
|
||||
If your transformation cannot be expressed in SQL, compute the values outside
|
||||
`add_columns` before writing them back through another workflow.
|
||||
</Info>
|
||||
<Note>
|
||||
Columns derived from existing data -- SQL expressions over other columns -- are
|
||||
covered in [Data evolution](/feature-engineering/data-evolution), because
|
||||
deriving values is a feature-engineering job rather than a schema one.
|
||||
</Note>
|
||||
|
||||
### Add columns with default values {#add-columns-with-default-values}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user