diff --git a/docs/web/docs.json b/docs/web/docs.json
index 60212b6d1..b5152fe24 100644
--- a/docs/web/docs.json
+++ b/docs/web/docs.json
@@ -70,6 +70,10 @@
{
"source": "/feature-engineering/functions",
"destination": "/management/function"
+ },
+ {
+ "source": "/tables/create",
+ "destination": "/ingestion"
}
],
"navigation": {
@@ -110,9 +114,10 @@
"group": "Table",
"pages": [
"tables/index",
- "tables/create",
"tables/schema",
- "tables/multimodal"
+ "tables/multimodal",
+ "tables/versioning",
+ "tables/branching"
]
},
"management/function"
@@ -126,8 +131,6 @@
"ingestion/merge-insert",
"ingestion/update",
"ingestion/delete",
- "tables/versioning",
- "tables/branching",
"tables/consistency"
]
},
@@ -169,6 +172,7 @@
"group": "Feature engineering",
"pages": [
"feature-engineering/index",
+ "feature-engineering/data-evolution",
"feature-engineering/add-computed-column",
"feature-engineering/refresh-column",
"feature-engineering/materialized-views"
diff --git a/docs/web/faq/faq-oss.mdx b/docs/web/faq/faq-oss.mdx
index 7c1b4ceaf..94a8eb6c2 100644
--- a/docs/web/faq/faq-oss.mdx
+++ b/docs/web/faq/faq-oss.mdx
@@ -59,7 +59,7 @@ For best results:
- **Avoid inserting one row at a time.** Each insert creates a new data fragment on
disk. Batch your data into Arrow tables, DataFrames, or use iterators.
-See [Loading Large Datasets](/tables/create#loading-large-datasets) for full examples.
+See [Ingesting large datasets](/ingestion/insert#large-datasets) for full examples.
### Do I need to set a refine factor when using an index? {#do-i-need-to-set-a-refine-factor-when-using-an-index}
diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx
new file mode 100644
index 000000000..a30b97d43
--- /dev/null
+++ b/docs/web/feature-engineering/data-evolution.mdx
@@ -0,0 +1,113 @@
+---
+title: "Data evolution"
+sidebarTitle: "Data Evolution"
+description: "Derive new columns from data already in the table, without rewriting what is already there."
+icon: wand-magic-sparkles
+keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
+---
+
+import {
+ PySchemaAddSetup as SchemaAddSetup,
+ TsSchemaAddSetup as TsSchemaAddSetup,
+ RsSchemaAddSetup as RsSchemaAddSetup,
+ PyAddColumnsCalculated as AddColumnsCalculated,
+ TsAddColumnsCalculated as TsAddColumnsCalculated,
+ RsAddColumnsCalculated as RsAddColumnsCalculated,
+ PyAddFeatureColumnsSql as AddFeatureColumnsSql,
+ TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
+ RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
+} from '/snippets/tables.mdx';
+
+Data evolution is adding **values** derived from what a table already holds.
+[Schema evolution](/tables/schema) changes the table's shape; this fills the
+shape with something new.
+
+The reason it is cheap is the format. Adding a derived column writes only that
+column -- the existing data, which is the expensive part on a multimodal table,
+is never rewritten. See [why that works](/basics/lance#evolution).
+
+## Set up the example table {#set-up-the-example-table}
+
+
+
+ {SchemaAddSetup}
+
+
+
+ {TsSchemaAddSetup}
+
+
+
+ {RsSchemaAddSetup}
+
+
+
+## Add derived columns {#add-derived-columns}
+
+You can add new columns that are derived from existing data using SQL expressions.
+For feature engineering on large existing tables, group related derived features into
+one `add_columns` operation instead of running many separate writes. This creates one
+new table version for the schema change and computes the new columns from the existing
+rows, which avoids growing the table's version history with many small updates.
+
+
+
+ {AddColumnsCalculated}
+
+
+
+ {TsAddColumnsCalculated}
+
+
+
+ {RsAddColumnsCalculated}
+
+
+
+The same call can add multiple derived columns at once. For example, if you are
+building several lightweight features from existing product fields, pass all of the
+new column expressions together:
+
+
+
+ {AddFeatureColumnsSql}
+
+
+
+ {TsAddFeatureColumnsSql}
+
+
+
+ {RsAddFeatureColumnsSql}
+
+
+
+
+LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
+PyArrow `RecordBatch` iterators for populating new columns. New column values must be
+defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
+If your transformation cannot be expressed in SQL, compute the values outside
+`add_columns` before writing them back through another workflow.
+
+
+## When SQL is not enough {#beyond-sql}
+
+The expressions above are evaluated once, as part of the call. Two other shapes
+exist when that does not fit:
+
+- A [computed column](/feature-engineering/add-computed-column) declares the
+ expression and leaves the column empty, so you choose when the work happens.
+ That is the right default on a large table.
+- A [function](/management/function) fills a column with Python, for values SQL
+ cannot express -- an embedding, a model call, an image decode.
+
+## What to read next {#next}
+
+
+
+ Declare now, fill later.
+
+
+ Change the table's shape instead of its contents.
+
+
diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx
index 7b79ae323..1676ce525 100644
--- a/docs/web/ingestion/index.mdx
+++ b/docs/web/ingestion/index.mdx
@@ -1,11 +1,50 @@
---
title: "Ingesting data"
sidebarTitle: "Overview"
-description: "The four write operations, and the example table the rest of this section uses."
+description: "Getting data into a table: creating one from your data, and the four write operations that follow."
icon: inbox
+keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
---
import {
+ TsConnect,
+ RsConnect,
+} from '/snippets/connection.mdx';
+import {
+ PyCreateTableFromDicts as CreateTableFromDicts,
+ TsCreateTableFromDicts as TsCreateTableFromDicts,
+ RsCreateTableFromDicts as RsCreateTableFromDicts,
+ PyCreateTableConflictHandling as CreateTableConflictHandling,
+ TsCreateTableConflictHandling as TsCreateTableConflictHandling,
+ RsCreateTableConflictHandling as RsCreateTableConflictHandling,
+ PyCreateTableFromPandas as CreateTableFromPandas,
+ PyCreateTableCustomSchema as CreateTableCustomSchema,
+ TsCreateTableCustomSchema as TsCreateTableCustomSchema,
+ RsCreateTableCustomSchema as RsCreateTableCustomSchema,
+ PyCreateTableFromPolars as CreateTableFromPolars,
+ PyCreateTableFromArrow as CreateTableFromArrow,
+ TsCreateTableFromArrow as TsCreateTableFromArrow,
+ RsCreateTableFromArrow as RsCreateTableFromArrow,
+ PyCreateTableFromPydantic as CreateTableFromPydantic,
+ PyCreateTableNestedSchema as CreateTableNestedSchema,
+ PyAddFromDataset as AddFromDataset,
+ PyCreateTableFromIterator as CreateTableFromIterator,
+ TsCreateTableFromIterator as TsCreateTableFromIterator,
+ RsCreateTableFromIterator as RsCreateTableFromIterator,
+ TsAddProgress,
+ PyOpenExistingTable as OpenExistingTable,
+ TsOpenExistingTable as TsOpenExistingTable,
+ RsOpenExistingTable as RsOpenExistingTable,
+ PyCreateEmptyTable as CreateEmptyTable,
+ TsCreateEmptyTable as TsCreateEmptyTable,
+ RsCreateEmptyTable as RsCreateEmptyTable,
+ PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
+ PyDropTable as DropTable,
+ TsDropTable as TsDropTable,
+ RsDropTable as RsDropTable,
+ PyTablesBasicConnect as TablesBasicConnect,
+ PyTablesDocumentModel as TablesDocumentModel,
+ PyTablesTzValidator as TablesTzValidator,
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
PyUpdateConnectLocal as UpdateConnectLocal,
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
@@ -18,6 +57,316 @@ import {
RsUpdateMakeUsersReader,
} from '/snippets/tables.mdx';
+In LanceDB, tables store records with a defined schema that specifies column
+names and types. Across the SDKs, you can create tables from row-oriented data
+and Apache Arrow data structures. The Python SDK additionally supports PyArrow
+schemas for explicit schema control, and `LanceModel` for Pydantic-based
+validation.
+
+Getting data in has two halves: creating a table from data you already have, and
+writing to a table that exists. This page covers the first and introduces the
+second.
+
+## Create a table with data {#create-a-table-with-data}
+
+Initialize a LanceDB connection and create a table
+
+
+
+ {TablesBasicConnect}
+
+
+
+ {TsConnect}
+
+
+
+ {RsConnect}
+
+
+
+Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
+
+### From list of objects {#from-list-of-objects}
+
+You can provide a list of objects to create a table. The Python and TypeScript SDKs
+support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
+In Python, pass a list or other batch-like object; a single bare `dict` or single
+`LanceModel` is rejected.
+
+
+
+ {CreateTableFromDicts}
+
+
+
+ {TsCreateTableFromDicts}
+
+
+
+ {RsCreateTableFromDicts}
+
+
+
+### Handle existing tables {#handle-existing-tables}
+
+By default, `create_table` raises an error if a table with the same name already exists.
+You can change this behavior with two parameters that resolve the conflict in different ways:
+
+- **Idempotent open**: return the existing table without modifying it. Use when your
+ code may run more than once (notebooks, retries, init scripts) and you want to reuse
+ the table on subsequent runs. The provided data is ignored, but the schema is
+ validated against the existing table and a mismatch raises an error.
+- **Overwrite**: drop the existing table and create a new one with the provided data.
+ Use this for test fixtures or when you intentionally want to replace prior contents.
+ This permanently discards the old table's data.
+
+
+
+ {CreateTableConflictHandling}
+
+
+
+ {TsCreateTableConflictHandling}
+
+
+
+ {RsCreateTableConflictHandling}
+
+
+
+
+`exist_ok` / `existOk` does not append the provided data to an existing table. Use
+[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
+contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
+
+
+### From a custom schema {#from-a-custom-schema}
+
+You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
+
+
+
+ {CreateTableCustomSchema}
+
+
+
+ {TsCreateTableCustomSchema}
+
+
+
+ {RsCreateTableCustomSchema}
+
+
+
+An explicit schema is also where you control nullability. If later writes omit a
+non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
+omitted or written with null values. Without an explicit schema, Python infers list-like vector
+values as fixed-size `float32` vector fields from the observed dimension.
+
+For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
+null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
+`fill_value`, and `"null"` writes nulls.
+
+### From an Arrow Table {#from-an-arrow-table}
+You can also create LanceDB tables directly from Arrow tables.
+Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
+
+
+
+ {CreateTableFromArrow}
+
+
+
+ {TsCreateTableFromArrow}
+
+
+
+ {RsCreateTableFromArrow}
+
+
+
+
+### From a Pandas DataFrame {#from-a-pandas-dataframe}
+Python Only
+
+
+
+ {CreateTableFromPandas}
+
+
+
+
+Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
+
+
+
+The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
+
+
+### From a Polars DataFrame {#from-a-polars-dataframe}
+Python Only
+
+LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
+written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
+under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
+is on the way.
+
+
+
+ {CreateTableFromPolars}
+
+
+
+### From Pydantic Models {#from-pydantic-models}
+Python Only
+
+When you create an empty table without data, you must specify the table schema.
+LanceDB supports creating tables by specifying a PyArrow schema or a specialized
+Pydantic model called `LanceModel`.
+
+For example, the following Content model specifies a table with 5 columns:
+`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
+pass the class as the value of the `schema` parameter to `create_table`.
+The `vector` column is a `Vector` type, which is a specialized Pydantic type that
+can be configured with the vector dimensions. It is also important to note that
+LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
+(which itself derives from `pydantic.BaseModel`).
+
+
+
+ {CreateTableFromPydantic}
+
+
+
+#### Nested schemas {#nested-schemas}
+
+Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
+
+
+
+ {TablesDocumentModel}
+
+
+
+This can be used as the type of a LanceDB table column:
+
+
+
+ {CreateTableNestedSchema}
+
+
+
+This creates a struct column called "document" that has two subfields
+called "content" and "source":
+
+```bash
+In [28]: tbl.schema
+Out[28]:
+id: string not null
+vector: fixed_size_list[1536] not null
+ child 0, item: float
+document: struct not null
+ child 0, content: string not null
+ child 1, source: string not null
+```
+
+#### Validators {#validators}
+
+Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
+[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
+below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
+for a `created_at` field.
+
+
+
+ {TablesTzValidator}
+
+
+
+When you run this code it, should raise the `ValidationError`.
+
+
+Datasets too large to pass to `create_table` at once are covered separately --
+see [Insert data](/ingestion/insert#large-datasets), which explains why
+`table.add()` parallelizes where `create_table` does not.
+
+
+## Create empty table {#create-empty-table}
+You can create an empty table for scenarios where you want to add data to the table later.
+An example would be when you want to collect data from a stream/external file and then add it to a table in
+batches.
+
+An empty table can be initialized via an Arrow schema.
+
+
+
+ {CreateEmptyTable}
+
+
+
+ {TsCreateEmptyTable}
+
+
+
+ {RsCreateEmptyTable}
+
+
+
+Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
+directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
+that has been extended to support LanceDB specific types like `Vector`.
+
+
+
+ {CreateEmptyTablePydantic}
+
+
+
+Once the empty table has been created, you can append to it or modify its contents,
+as explained in [Ingesting data](/ingestion).
+
+## Open an existing table {#open-an-existing-table}
+
+You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
+If you forget the name of your table, you can always get a listing of all table names.
+
+
+
+ {OpenExistingTable}
+
+
+
+ {TsOpenExistingTable}
+
+
+
+ {RsOpenExistingTable}
+
+
+
+## Drop a table {#drop-a-table}
+
+Use the `drop_table()` method on the database to remove a table.
+
+
+
+ {DropTable}
+
+
+
+ {TsDropTable}
+
+
+
+ {RsDropTable}
+
+
+
+This permanently removes the table and is not recoverable, unlike deleting rows.
+By default, if the table does not exist an exception is raised. To suppress this,
+you can pass in `ignore_missing=True`.
+
## Choose a write method {#choose-a-write-method}
| Family | Method | Use this when... |
@@ -104,4 +453,4 @@ Expected table contents:
| 2 | Bob | 20 |
The example above shows a PyArrow schema. You can just as well create the table using other
-table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Creating a table](/tables/create) for more details.
+table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [above](#create-a-table-with-data).
diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx
index cc49b354c..5290b17c6 100644
--- a/docs/web/ingestion/insert.mdx
+++ b/docs/web/ingestion/insert.mdx
@@ -6,7 +6,7 @@ icon: plus
---
Rows are appended with `table.add()`. It accepts the same shapes
-[table creation](/tables/create) does — lists of objects, Arrow tables, pandas
+[table creation](/ingestion) does — lists of objects, Arrow tables, pandas
and polars frames, Pydantic models — and commits a new table version.
These examples use the table set up in [Ingesting data](/ingestion).
diff --git a/docs/web/tables/branching.mdx b/docs/web/tables/branching.mdx
index 9e0d06339..a7bd677d3 100644
--- a/docs/web/tables/branching.mdx
+++ b/docs/web/tables/branching.mdx
@@ -64,7 +64,7 @@ well as on LanceDB Enterprise (remote) tables.
The branch API is identical no matter how you connect — only the connection
itself differs between OSS and Enterprise. Establish a connection (`db`) and open
-a `table` (see [Create a table](/tables/create)), then use the same branch calls
+a `table` (see [Ingesting data](/ingestion)), then use the same branch calls
in every example that follows.
### LanceDB OSS {#lancedb-oss}
diff --git a/docs/web/tables/create.mdx b/docs/web/tables/create.mdx
deleted file mode 100644
index 735b1ceaf..000000000
--- a/docs/web/tables/create.mdx
+++ /dev/null
@@ -1,442 +0,0 @@
----
-title: Ingesting Data
-sidebarTitle: "Ingesting data"
-description: Learn about different methods to ingest data into tables in LanceDB, including from various data sources and empty tables.
-icon: "cookie"
----
-import { TsConnect, RsConnect } from '/snippets/connection.mdx';
-import {
- PyCreateTableFromDicts as CreateTableFromDicts,
- TsCreateTableFromDicts as TsCreateTableFromDicts,
- RsCreateTableFromDicts as RsCreateTableFromDicts,
- PyCreateTableConflictHandling as CreateTableConflictHandling,
- TsCreateTableConflictHandling as TsCreateTableConflictHandling,
- RsCreateTableConflictHandling as RsCreateTableConflictHandling,
- PyCreateTableFromPandas as CreateTableFromPandas,
- PyCreateTableCustomSchema as CreateTableCustomSchema,
- TsCreateTableCustomSchema as TsCreateTableCustomSchema,
- RsCreateTableCustomSchema as RsCreateTableCustomSchema,
- PyCreateTableFromPolars as CreateTableFromPolars,
- PyCreateTableFromArrow as CreateTableFromArrow,
- TsCreateTableFromArrow as TsCreateTableFromArrow,
- RsCreateTableFromArrow as RsCreateTableFromArrow,
- PyCreateTableFromPydantic as CreateTableFromPydantic,
- PyCreateTableNestedSchema as CreateTableNestedSchema,
- PyAddFromDataset as AddFromDataset,
- PyCreateTableFromIterator as CreateTableFromIterator,
- TsCreateTableFromIterator as TsCreateTableFromIterator,
- RsCreateTableFromIterator as RsCreateTableFromIterator,
- TsAddProgress,
- PyOpenExistingTable as OpenExistingTable,
- TsOpenExistingTable as TsOpenExistingTable,
- RsOpenExistingTable as RsOpenExistingTable,
- PyCreateEmptyTable as CreateEmptyTable,
- TsCreateEmptyTable as TsCreateEmptyTable,
- RsCreateEmptyTable as RsCreateEmptyTable,
- PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
- PyDropTable as DropTable,
- TsDropTable as TsDropTable,
- RsDropTable as RsDropTable,
- PyTablesBasicConnect as TablesBasicConnect,
- PyTablesDocumentModel as TablesDocumentModel,
- PyTablesTzValidator as TablesTzValidator,
-} from '/snippets/tables.mdx';
-
-In LanceDB, tables store records with a defined schema that specifies column names and types. Across the SDKs, you can create tables from row-oriented data and Apache Arrow data structures. The Python SDK additionally supports:
-
-- PyArrow schemas for explicit schema control
-- `LanceModel` for Pydantic-based validation
-
-## Create a table with data {#create-a-table-with-data}
-
-Initialize a LanceDB connection and create a table
-
-
-
- {TablesBasicConnect}
-
-
-
- {TsConnect}
-
-
-
- {RsConnect}
-
-
-
-Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.
-
-### From list of objects {#from-list-of-objects}
-
-You can provide a list of objects to create a table. The Python and TypeScript SDKs
-support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
-In Python, pass a list or other batch-like object; a single bare `dict` or single
-`LanceModel` is rejected.
-
-
-
- {CreateTableFromDicts}
-
-
-
- {TsCreateTableFromDicts}
-
-
-
- {RsCreateTableFromDicts}
-
-
-
-### Handle existing tables {#handle-existing-tables}
-
-By default, `create_table` raises an error if a table with the same name already exists.
-You can change this behavior with two parameters that resolve the conflict in different ways:
-
-- **Idempotent open**: return the existing table without modifying it. Use when your
- code may run more than once (notebooks, retries, init scripts) and you want to reuse
- the table on subsequent runs. The provided data is ignored, but the schema is
- validated against the existing table and a mismatch raises an error.
-- **Overwrite**: drop the existing table and create a new one with the provided data.
- Use this for test fixtures or when you intentionally want to replace prior contents.
- This permanently discards the old table's data.
-
-
-
- {CreateTableConflictHandling}
-
-
-
- {TsCreateTableConflictHandling}
-
-
-
- {RsCreateTableConflictHandling}
-
-
-
-
-`exist_ok` / `existOk` does not append the provided data to an existing table. Use
-[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
-contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
-
-
-### From a custom schema {#from-a-custom-schema}
-
-You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.
-
-
-
- {CreateTableCustomSchema}
-
-
-
- {TsCreateTableCustomSchema}
-
-
-
- {RsCreateTableCustomSchema}
-
-
-
-An explicit schema is also where you control nullability. If later writes omit a
-non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
-omitted or written with null values. Without an explicit schema, Python infers list-like vector
-values as fixed-size `float32` vector fields from the observed dimension.
-
-For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
-null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
-`fill_value`, and `"null"` writes nulls.
-
-### From an Arrow Table {#from-an-arrow-table}
-You can also create LanceDB tables directly from Arrow tables.
-Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.
-
-
-
- {CreateTableFromArrow}
-
-
-
- {TsCreateTableFromArrow}
-
-
-
- {RsCreateTableFromArrow}
-
-
-
-
-### From a Pandas DataFrame {#from-a-pandas-dataframe}
-Python Only
-
-
-
- {CreateTableFromPandas}
-
-
-
-
-Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
-
-
-
-The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
-
-
-### From a Polars DataFrame {#from-a-polars-dataframe}
-Python Only
-
-LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
-written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
-under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
-is on the way.
-
-
-
- {CreateTableFromPolars}
-
-
-
-### From Pydantic Models {#from-pydantic-models}
-Python Only
-
-When you create an empty table without data, you must specify the table schema.
-LanceDB supports creating tables by specifying a PyArrow schema or a specialized
-Pydantic model called `LanceModel`.
-
-For example, the following Content model specifies a table with 5 columns:
-`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
-pass the class as the value of the `schema` parameter to `create_table`.
-The `vector` column is a `Vector` type, which is a specialized Pydantic type that
-can be configured with the vector dimensions. It is also important to note that
-LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
-(which itself derives from `pydantic.BaseModel`).
-
-
-
- {CreateTableFromPydantic}
-
-
-
-#### Nested schemas {#nested-schemas}
-
-Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:
-
-
-
- {TablesDocumentModel}
-
-
-
-This can be used as the type of a LanceDB table column:
-
-
-
- {CreateTableNestedSchema}
-
-
-
-This creates a struct column called "document" that has two subfields
-called "content" and "source":
-
-```bash
-In [28]: tbl.schema
-Out[28]:
-id: string not null
-vector: fixed_size_list[1536] not null
- child 0, item: float
-document: struct not null
- child 0, content: string not null
- child 1, source: string not null
-```
-
-#### Validators {#validators}
-
-Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
-[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
-below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
-for a `created_at` field.
-
-
-
- {TablesTzValidator}
-
-
-
-When you run this code it, should raise the `ValidationError`.
-
-### Loading Large Datasets {#loading-large-datasets}
-
-When ingesting large datasets, use `table.add()` on an existing table rather than
-passing all data to `create_table()`. The `add()` method auto-parallelizes large
-writes, while `create_table(name, data)` does not.
-
-
-For best performance with large datasets, create an empty table first and then call
-`table.add()`. This enables automatic write parallelism for materialized data sources.
-
-
-#### From files (Parquet, CSV, etc.) {#from-files-parquet-csv-etc}
-Python Only
-
-For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams
-data from disk without loading the entire dataset into memory.
-
-
-
- {AddFromDataset}
-
-
-
-
-`pa.dataset()` input is currently Python-only. TypeScript and Rust support for
-file-based dataset ingestion is tracked in
-[lancedb#3173](https://github.com/lancedb/lancedb/issues/3173).
-
-
-#### From iterators (custom batch generation) {#from-iterators-custom-batch-generation}
-
-When you need custom batch logic — generating embeddings on the fly, transforming
-rows from an external source, etc. — use an iterator of `RecordBatch` objects.
-
-
-
- {CreateTableFromIterator}
-
-
-
- {TsCreateTableFromIterator}
-
-
-
- {RsCreateTableFromIterator}
-
-
-
-Use this pattern when:
-
-- Your source data already arrives in Arrow batches, readers, datasets, or streams.
-- Materializing the entire ingest as one giant in-memory list or array would be too expensive.
-- You want to control chunk size explicitly during ingestion.
-
-Python can also consume iterators of other supported types like Pandas DataFrames or Python lists.
-
-#### Write parallelism {#write-parallelism}
-
-
-For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB
-automatically parallelizes large writes — no configuration needed. Auto-parallelism
-targets approximately 1M rows or 2GB per write partition.
-
-For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine
-total size upfront. A `parallelism` parameter to control this manually is planned
-but not yet exposed in Python or TypeScript
-([tracking issue](https://github.com/lancedb/lancedb/issues/3173)).
-
-
-#### Tracking ingestion progress {#tracking-ingestion-progress}
-TypeScript Only
-
-For long-running writes, pass a `progress` callback to `table.add()` to surface
-per-batch progress in your UI, logs, or metrics pipeline. The callback fires
-once per batch written and once more with `done: true` when the write completes.
-
-Each invocation receives a `WriteProgress` object:
-
-| Field | Description |
-|:------|:------------|
-| `outputRows` | Rows written so far. |
-| `outputBytes` | Bytes written so far. |
-| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. |
-| `elapsedSeconds` | Wall-clock seconds since the write started. |
-| `activeTasks` | Parallel write tasks currently in flight. |
-| `totalTasks` | Total parallel write tasks (the write parallelism). |
-| `done` | `true` only for the final callback. |
-
-
-
- {TsAddProgress}
-
-
-
-A few things to know before you wire this up:
-
-- Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere.
-- Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow.
-- Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written.
-
-## Create empty table {#create-empty-table}
-You can create an empty table for scenarios where you want to add data to the table later.
-An example would be when you want to collect data from a stream/external file and then add it to a table in
-batches.
-
-An empty table can be initialized via an Arrow schema.
-
-
-
- {CreateEmptyTable}
-
-
-
- {TsCreateEmptyTable}
-
-
-
- {RsCreateEmptyTable}
-
-
-
-Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
-directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
-that has been extended to support LanceDB specific types like `Vector`.
-
-
-
- {CreateEmptyTablePydantic}
-
-
-
-Once the empty table has been created, you can append to it or modify its contents,
-as explained in [Ingesting data](/ingestion).
-
-## Open an existing table {#open-an-existing-table}
-
-You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
-If you forget the name of your table, you can always get a listing of all table names.
-
-
-
- {OpenExistingTable}
-
-
-
- {TsOpenExistingTable}
-
-
-
- {RsOpenExistingTable}
-
-
-
-## Drop a table {#drop-a-table}
-
-Use the `drop_table()` method on the database to remove a table.
-
-
-
- {DropTable}
-
-
-
- {TsDropTable}
-
-
-
- {RsDropTable}
-
-
-
-This permanently removes the table and is not recoverable, unlike deleting rows.
-By default, if the table does not exist an exception is raised. To suppress this,
-you can pass in `ignore_missing=True`.
diff --git a/docs/web/tables/index.mdx b/docs/web/tables/index.mdx
index 3c2f5e689..a731cee52 100644
--- a/docs/web/tables/index.mdx
+++ b/docs/web/tables/index.mdx
@@ -267,12 +267,12 @@ reuse the existing table instead of writing the supplied rows again.
If you want to avoid overwriting an existing table, omit the overwrite mode. For append-only
ingestion into a table that already exists, open the table and call `add(...)` instead of
`create_table(...)`. For repeatable setup with `exist_ok` / `existOk`, see
-[Handle existing tables](/tables/create#handle-existing-tables).
+[Handle existing tables](/ingestion#handle-existing-tables).
For more ingestion patterns, including PyArrow tables, Python `pyarrow.dataset.Dataset` inputs,
empty tables, and Python `LanceModel` schemas with nested fields, see
-[Ingesting data](/tables/create/).
+[Ingesting data](/ingestion).
### From Pandas DataFrames {#from-pandas-dataframes}
Python Only
@@ -644,7 +644,7 @@ advanced features of LanceDB. Below are some suggested next pages.
Learn the different approaches to creating and ingesting data into LanceDB tables from various sources.
diff --git a/docs/web/tables/schema.mdx b/docs/web/tables/schema.mdx
index ef8db1906..9b110abd9 100644
--- a/docs/web/tables/schema.mdx
+++ b/docs/web/tables/schema.mdx
@@ -1,6 +1,6 @@
---
-title: "Schema and Data Evolution"
-sidebarTitle: "Schema & data evolution"
+title: "Schema evolution"
+sidebarTitle: "Schema Evolution"
description: Learn how to manage table schemas in LanceDB, including adding, altering, and dropping columns.
icon: "boxes-stacked"
---
@@ -8,18 +8,12 @@ import {
PySchemaAddSetup as SchemaAddSetup,
TsSchemaAddSetup as TsSchemaAddSetup,
RsSchemaAddSetup as RsSchemaAddSetup,
- PyAddColumnsCalculated as AddColumnsCalculated,
- TsAddColumnsCalculated as TsAddColumnsCalculated,
- RsAddColumnsCalculated as RsAddColumnsCalculated,
PyAddColumnsDefaultValues as AddColumnsDefaultValues,
TsAddColumnsDefaultValues as TsAddColumnsDefaultValues,
RsAddColumnsDefaultValues as RsAddColumnsDefaultValues,
PyAddColumnsNullable as AddColumnsNullable,
TsAddColumnsNullable as TsAddColumnsNullable,
RsAddColumnsNullable as RsAddColumnsNullable,
- PyAddFeatureColumnsSql as AddFeatureColumnsSql,
- TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
- RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
PySchemaAlterSetup as SchemaAlterSetup,
TsSchemaAlterSetup as TsSchemaAlterSetup,
RsSchemaAlterSetup as RsSchemaAlterSetup,
@@ -108,53 +102,11 @@ First, let's create a sample table with product data to demonstrate schema evolu
-### Add derived columns {#add-derived-columns}
-
-You can add new columns that are derived from existing data using SQL expressions.
-For feature engineering on large existing tables, group related derived features into
-one `add_columns` operation instead of running many separate writes. This creates one
-new table version for the schema change and computes the new columns from the existing
-rows, which avoids growing the table's version history with many small updates.
-
-
-
- {AddColumnsCalculated}
-
-
-
- {TsAddColumnsCalculated}
-
-
-
- {RsAddColumnsCalculated}
-
-
-
-The same call can add multiple derived columns at once. For example, if you are
-building several lightweight features from existing product fields, pass all of the
-new column expressions together:
-
-
-
- {AddFeatureColumnsSql}
-
-
-
- {TsAddFeatureColumnsSql}
-
-
-
- {RsAddFeatureColumnsSql}
-
-
-
-
-LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
-PyArrow `RecordBatch` iterators for populating new columns. New column values must be
-defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
-If your transformation cannot be expressed in SQL, compute the values outside
-`add_columns` before writing them back through another workflow.
-
+
+Columns derived from existing data -- SQL expressions over other columns -- are
+covered in [Data evolution](/feature-engineering/data-evolution), because
+deriving values is a feature-engineering job rather than a schema one.
+
### Add columns with default values {#add-columns-with-default-values}