From 139138c22e693a8e491d234e4ffe5aa191c3068c Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Wed, 9 Sep 2026 23:23:14 -0700 Subject: [PATCH] docs: move versioning and branching under Table, split schema from data evolution Versioning and branching are properties of a table, not steps in getting data in, so they join the Table group. Consistency stays with Ingestion, where the question it answers -- when do other writers' rows become visible -- is asked. The Ingesting Data page was in the Table group while an Ingestion overview existed separately, so the two are merged. The large-dataset section was duplicated between it and the insert page; the insert page keeps it. Schema and Data Evolution covered two different things under one title. Changing a table's shape stays in Table management; deriving values from data already there becomes a Feature engineering page, next to computed columns and functions, which are the same job at larger scale. --- docs/web/docs.json | 12 +- docs/web/faq/faq-oss.mdx | 2 +- .../feature-engineering/data-evolution.mdx | 113 +++++ docs/web/ingestion/index.mdx | 353 +++++++++++++- docs/web/ingestion/insert.mdx | 2 +- docs/web/tables/branching.mdx | 2 +- docs/web/tables/create.mdx | 442 ------------------ docs/web/tables/index.mdx | 6 +- docs/web/tables/schema.mdx | 62 +-- 9 files changed, 485 insertions(+), 509 deletions(-) create mode 100644 docs/web/feature-engineering/data-evolution.mdx delete mode 100644 docs/web/tables/create.mdx diff --git a/docs/web/docs.json b/docs/web/docs.json index 60212b6d1..b5152fe24 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -70,6 +70,10 @@ { "source": "/feature-engineering/functions", "destination": "/management/function" + }, + { + "source": "/tables/create", + "destination": "/ingestion" } ], "navigation": { @@ -110,9 +114,10 @@ "group": "Table", "pages": [ "tables/index", - "tables/create", "tables/schema", - "tables/multimodal" + "tables/multimodal", + "tables/versioning", + "tables/branching" ] }, "management/function" @@ -126,8 +131,6 @@ "ingestion/merge-insert", "ingestion/update", "ingestion/delete", - "tables/versioning", - "tables/branching", "tables/consistency" ] }, @@ -169,6 +172,7 @@ "group": "Feature engineering", "pages": [ "feature-engineering/index", + "feature-engineering/data-evolution", "feature-engineering/add-computed-column", "feature-engineering/refresh-column", "feature-engineering/materialized-views" diff --git a/docs/web/faq/faq-oss.mdx b/docs/web/faq/faq-oss.mdx index 7c1b4ceaf..94a8eb6c2 100644 --- a/docs/web/faq/faq-oss.mdx +++ b/docs/web/faq/faq-oss.mdx @@ -59,7 +59,7 @@ For best results: - **Avoid inserting one row at a time.** Each insert creates a new data fragment on disk. Batch your data into Arrow tables, DataFrames, or use iterators. -See [Loading Large Datasets](/tables/create#loading-large-datasets) for full examples. +See [Ingesting large datasets](/ingestion/insert#large-datasets) for full examples. ### Do I need to set a refine factor when using an index? {#do-i-need-to-set-a-refine-factor-when-using-an-index} diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx new file mode 100644 index 000000000..a30b97d43 --- /dev/null +++ b/docs/web/feature-engineering/data-evolution.mdx @@ -0,0 +1,113 @@ +--- +title: "Data evolution" +sidebarTitle: "Data Evolution" +description: "Derive new columns from data already in the table, without rewriting what is already there." +icon: wand-magic-sparkles +keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"] +--- + +import { + PySchemaAddSetup as SchemaAddSetup, + TsSchemaAddSetup as TsSchemaAddSetup, + RsSchemaAddSetup as RsSchemaAddSetup, + PyAddColumnsCalculated as AddColumnsCalculated, + TsAddColumnsCalculated as TsAddColumnsCalculated, + RsAddColumnsCalculated as RsAddColumnsCalculated, + PyAddFeatureColumnsSql as AddFeatureColumnsSql, + TsAddFeatureColumnsSql as TsAddFeatureColumnsSql, + RsAddFeatureColumnsSql as RsAddFeatureColumnsSql, +} from '/snippets/tables.mdx'; + +Data evolution is adding **values** derived from what a table already holds. +[Schema evolution](/tables/schema) changes the table's shape; this fills the +shape with something new. + +The reason it is cheap is the format. Adding a derived column writes only that +column -- the existing data, which is the expensive part on a multimodal table, +is never rewritten. See [why that works](/basics/lance#evolution). + +## Set up the example table {#set-up-the-example-table} + + + + {SchemaAddSetup} + + + + {TsSchemaAddSetup} + + + + {RsSchemaAddSetup} + + + +## Add derived columns {#add-derived-columns} + +You can add new columns that are derived from existing data using SQL expressions. +For feature engineering on large existing tables, group related derived features into +one `add_columns` operation instead of running many separate writes. This creates one +new table version for the schema change and computes the new columns from the existing +rows, which avoids growing the table's version history with many small updates. + + + + {AddColumnsCalculated} + + + + {TsAddColumnsCalculated} + + + + {RsAddColumnsCalculated} + + + +The same call can add multiple derived columns at once. For example, if you are +building several lightweight features from existing product fields, pass all of the +new column expressions together: + + + + {AddFeatureColumnsSql} + + + + {TsAddFeatureColumnsSql} + + + + {RsAddFeatureColumnsSql} + + + + +LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or +PyArrow `RecordBatch` iterators for populating new columns. New column values must be +defined with SQL expressions, or added as NULL columns from an Arrow field or schema. +If your transformation cannot be expressed in SQL, compute the values outside +`add_columns` before writing them back through another workflow. + + +## When SQL is not enough {#beyond-sql} + +The expressions above are evaluated once, as part of the call. Two other shapes +exist when that does not fit: + +- A [computed column](/feature-engineering/add-computed-column) declares the + expression and leaves the column empty, so you choose when the work happens. + That is the right default on a large table. +- A [function](/management/function) fills a column with Python, for values SQL + cannot express -- an embedding, a model call, an image decode. + +## What to read next {#next} + + + + Declare now, fill later. + + + Change the table's shape instead of its contents. + + diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx index 7b79ae323..1676ce525 100644 --- a/docs/web/ingestion/index.mdx +++ b/docs/web/ingestion/index.mdx @@ -1,11 +1,50 @@ --- title: "Ingesting data" sidebarTitle: "Overview" -description: "The four write operations, and the example table the rest of this section uses." +description: "Getting data into a table: creating one from your data, and the four write operations that follow." icon: inbox +keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"] --- import { + TsConnect, + RsConnect, +} from '/snippets/connection.mdx'; +import { + PyCreateTableFromDicts as CreateTableFromDicts, + TsCreateTableFromDicts as TsCreateTableFromDicts, + RsCreateTableFromDicts as RsCreateTableFromDicts, + PyCreateTableConflictHandling as CreateTableConflictHandling, + TsCreateTableConflictHandling as TsCreateTableConflictHandling, + RsCreateTableConflictHandling as RsCreateTableConflictHandling, + PyCreateTableFromPandas as CreateTableFromPandas, + PyCreateTableCustomSchema as CreateTableCustomSchema, + TsCreateTableCustomSchema as TsCreateTableCustomSchema, + RsCreateTableCustomSchema as RsCreateTableCustomSchema, + PyCreateTableFromPolars as CreateTableFromPolars, + PyCreateTableFromArrow as CreateTableFromArrow, + TsCreateTableFromArrow as TsCreateTableFromArrow, + RsCreateTableFromArrow as RsCreateTableFromArrow, + PyCreateTableFromPydantic as CreateTableFromPydantic, + PyCreateTableNestedSchema as CreateTableNestedSchema, + PyAddFromDataset as AddFromDataset, + PyCreateTableFromIterator as CreateTableFromIterator, + TsCreateTableFromIterator as TsCreateTableFromIterator, + RsCreateTableFromIterator as RsCreateTableFromIterator, + TsAddProgress, + PyOpenExistingTable as OpenExistingTable, + TsOpenExistingTable as TsOpenExistingTable, + RsOpenExistingTable as RsOpenExistingTable, + PyCreateEmptyTable as CreateEmptyTable, + TsCreateEmptyTable as TsCreateEmptyTable, + RsCreateEmptyTable as RsCreateEmptyTable, + PyCreateEmptyTablePydantic as CreateEmptyTablePydantic, + PyDropTable as DropTable, + TsDropTable as TsDropTable, + RsDropTable as RsDropTable, + PyTablesBasicConnect as TablesBasicConnect, + PyTablesDocumentModel as TablesDocumentModel, + PyTablesTzValidator as TablesTzValidator, PyUpdateConnectEnterprise as UpdateConnectEnterprise, PyUpdateConnectLocal as UpdateConnectLocal, PyUpdateExampleTableSetup as UpdateExampleTableSetup, @@ -18,6 +57,316 @@ import { RsUpdateMakeUsersReader, } from '/snippets/tables.mdx'; +In LanceDB, tables store records with a defined schema that specifies column +names and types. Across the SDKs, you can create tables from row-oriented data +and Apache Arrow data structures. The Python SDK additionally supports PyArrow +schemas for explicit schema control, and `LanceModel` for Pydantic-based +validation. + +Getting data in has two halves: creating a table from data you already have, and +writing to a table that exists. This page covers the first and introduces the +second. + +## Create a table with data {#create-a-table-with-data} + +Initialize a LanceDB connection and create a table + + + + {TablesBasicConnect} + + + + {TsConnect} + + + + {RsConnect} + + + +Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns. + +### From list of objects {#from-list-of-objects} + +You can provide a list of objects to create a table. The Python and TypeScript SDKs +support lists/arrays of dictionaries, while the Rust SDK supports lists of structs. +In Python, pass a list or other batch-like object; a single bare `dict` or single +`LanceModel` is rejected. + + + + {CreateTableFromDicts} + + + + {TsCreateTableFromDicts} + + + + {RsCreateTableFromDicts} + + + +### Handle existing tables {#handle-existing-tables} + +By default, `create_table` raises an error if a table with the same name already exists. +You can change this behavior with two parameters that resolve the conflict in different ways: + +- **Idempotent open**: return the existing table without modifying it. Use when your + code may run more than once (notebooks, retries, init scripts) and you want to reuse + the table on subsequent runs. The provided data is ignored, but the schema is + validated against the existing table and a mismatch raises an error. +- **Overwrite**: drop the existing table and create a new one with the provided data. + Use this for test fixtures or when you intentionally want to replace prior contents. + This permanently discards the old table's data. + + + + {CreateTableConflictHandling} + + + + {TsCreateTableConflictHandling} + + + + {RsCreateTableConflictHandling} + + + + +`exist_ok` / `existOk` does not append the provided data to an existing table. Use +[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and* +contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table). + + +### From a custom schema {#from-a-custom-schema} + +You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata. + + + + {CreateTableCustomSchema} + + + + {TsCreateTableCustomSchema} + + + + {RsCreateTableCustomSchema} + + + +An explicit schema is also where you control nullability. If later writes omit a +non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be +omitted or written with null values. Without an explicit schema, Python infers list-like vector +values as fixed-size `float32` vector fields from the observed dimension. + +For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length, +null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes +`fill_value`, and `"null"` writes nulls. + +### From an Arrow Table {#from-an-arrow-table} +You can also create LanceDB tables directly from Arrow tables. +Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow. + + + + {CreateTableFromArrow} + + + + {TsCreateTableFromArrow} + + + + {RsCreateTableFromArrow} + + + + +### From a Pandas DataFrame {#from-a-pandas-dataframe} +Python Only + + + + {CreateTableFromPandas} + + + + +Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly. + + + +The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type. + + +### From a Polars DataFrame {#from-a-polars-dataframe} +Python Only + +LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library +written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow +under the hood. A deeper integration between LanceDB Tables and Polars DataFrames +is on the way. + + + + {CreateTableFromPolars} + + + +### From Pydantic Models {#from-pydantic-models} +Python Only + +When you create an empty table without data, you must specify the table schema. +LanceDB supports creating tables by specifying a PyArrow schema or a specialized +Pydantic model called `LanceModel`. + +For example, the following Content model specifies a table with 5 columns: +`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can +pass the class as the value of the `schema` parameter to `create_table`. +The `vector` column is a `Vector` type, which is a specialized Pydantic type that +can be configured with the vector dimensions. It is also important to note that +LanceDB only understands subclasses of `lancedb.pydantic.LanceModel` +(which itself derives from `pydantic.BaseModel`). + + + + {CreateTableFromPydantic} + + + +#### Nested schemas {#nested-schemas} + +Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object: + + + + {TablesDocumentModel} + + + +This can be used as the type of a LanceDB table column: + + + + {CreateTableNestedSchema} + + + +This creates a struct column called "document" that has two subfields +called "content" and "source": + +```bash +In [28]: tbl.schema +Out[28]: +id: string not null +vector: fixed_size_list[1536] not null + child 0, item: float +document: struct not null + child 0, content: string not null + child 1, source: string not null +``` + +#### Validators {#validators} + +Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's +[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example +below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used +for a `created_at` field. + + + + {TablesTzValidator} + + + +When you run this code it, should raise the `ValidationError`. + + +Datasets too large to pass to `create_table` at once are covered separately -- +see [Insert data](/ingestion/insert#large-datasets), which explains why +`table.add()` parallelizes where `create_table` does not. + + +## Create empty table {#create-empty-table} +You can create an empty table for scenarios where you want to add data to the table later. +An example would be when you want to collect data from a stream/external file and then add it to a table in +batches. + +An empty table can be initialized via an Arrow schema. + + + + {CreateEmptyTable} + + + + {TsCreateEmptyTable} + + + + {RsCreateEmptyTable} + + + +Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not +directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel` +that has been extended to support LanceDB specific types like `Vector`. + + + + {CreateEmptyTablePydantic} + + + +Once the empty table has been created, you can append to it or modify its contents, +as explained in [Ingesting data](/ingestion). + +## Open an existing table {#open-an-existing-table} + +You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method. +If you forget the name of your table, you can always get a listing of all table names. + + + + {OpenExistingTable} + + + + {TsOpenExistingTable} + + + + {RsOpenExistingTable} + + + +## Drop a table {#drop-a-table} + +Use the `drop_table()` method on the database to remove a table. + + + + {DropTable} + + + + {TsDropTable} + + + + {RsDropTable} + + + +This permanently removes the table and is not recoverable, unlike deleting rows. +By default, if the table does not exist an exception is raised. To suppress this, +you can pass in `ignore_missing=True`. + ## Choose a write method {#choose-a-write-method} | Family | Method | Use this when... | @@ -104,4 +453,4 @@ Expected table contents: | 2 | Bob | 20 | The example above shows a PyArrow schema. You can just as well create the table using other -table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Creating a table](/tables/create) for more details. +table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [above](#create-a-table-with-data). diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx index cc49b354c..5290b17c6 100644 --- a/docs/web/ingestion/insert.mdx +++ b/docs/web/ingestion/insert.mdx @@ -6,7 +6,7 @@ icon: plus --- Rows are appended with `table.add()`. It accepts the same shapes -[table creation](/tables/create) does — lists of objects, Arrow tables, pandas +[table creation](/ingestion) does — lists of objects, Arrow tables, pandas and polars frames, Pydantic models — and commits a new table version. These examples use the table set up in [Ingesting data](/ingestion). diff --git a/docs/web/tables/branching.mdx b/docs/web/tables/branching.mdx index 9e0d06339..a7bd677d3 100644 --- a/docs/web/tables/branching.mdx +++ b/docs/web/tables/branching.mdx @@ -64,7 +64,7 @@ well as on LanceDB Enterprise (remote) tables. The branch API is identical no matter how you connect — only the connection itself differs between OSS and Enterprise. Establish a connection (`db`) and open -a `table` (see [Create a table](/tables/create)), then use the same branch calls +a `table` (see [Ingesting data](/ingestion)), then use the same branch calls in every example that follows. ### LanceDB OSS {#lancedb-oss} diff --git a/docs/web/tables/create.mdx b/docs/web/tables/create.mdx deleted file mode 100644 index 735b1ceaf..000000000 --- a/docs/web/tables/create.mdx +++ /dev/null @@ -1,442 +0,0 @@ ---- -title: Ingesting Data -sidebarTitle: "Ingesting data" -description: Learn about different methods to ingest data into tables in LanceDB, including from various data sources and empty tables. -icon: "cookie" ---- -import { TsConnect, RsConnect } from '/snippets/connection.mdx'; -import { - PyCreateTableFromDicts as CreateTableFromDicts, - TsCreateTableFromDicts as TsCreateTableFromDicts, - RsCreateTableFromDicts as RsCreateTableFromDicts, - PyCreateTableConflictHandling as CreateTableConflictHandling, - TsCreateTableConflictHandling as TsCreateTableConflictHandling, - RsCreateTableConflictHandling as RsCreateTableConflictHandling, - PyCreateTableFromPandas as CreateTableFromPandas, - PyCreateTableCustomSchema as CreateTableCustomSchema, - TsCreateTableCustomSchema as TsCreateTableCustomSchema, - RsCreateTableCustomSchema as RsCreateTableCustomSchema, - PyCreateTableFromPolars as CreateTableFromPolars, - PyCreateTableFromArrow as CreateTableFromArrow, - TsCreateTableFromArrow as TsCreateTableFromArrow, - RsCreateTableFromArrow as RsCreateTableFromArrow, - PyCreateTableFromPydantic as CreateTableFromPydantic, - PyCreateTableNestedSchema as CreateTableNestedSchema, - PyAddFromDataset as AddFromDataset, - PyCreateTableFromIterator as CreateTableFromIterator, - TsCreateTableFromIterator as TsCreateTableFromIterator, - RsCreateTableFromIterator as RsCreateTableFromIterator, - TsAddProgress, - PyOpenExistingTable as OpenExistingTable, - TsOpenExistingTable as TsOpenExistingTable, - RsOpenExistingTable as RsOpenExistingTable, - PyCreateEmptyTable as CreateEmptyTable, - TsCreateEmptyTable as TsCreateEmptyTable, - RsCreateEmptyTable as RsCreateEmptyTable, - PyCreateEmptyTablePydantic as CreateEmptyTablePydantic, - PyDropTable as DropTable, - TsDropTable as TsDropTable, - RsDropTable as RsDropTable, - PyTablesBasicConnect as TablesBasicConnect, - PyTablesDocumentModel as TablesDocumentModel, - PyTablesTzValidator as TablesTzValidator, -} from '/snippets/tables.mdx'; - -In LanceDB, tables store records with a defined schema that specifies column names and types. Across the SDKs, you can create tables from row-oriented data and Apache Arrow data structures. The Python SDK additionally supports: - -- PyArrow schemas for explicit schema control -- `LanceModel` for Pydantic-based validation - -## Create a table with data {#create-a-table-with-data} - -Initialize a LanceDB connection and create a table - - - - {TablesBasicConnect} - - - - {TsConnect} - - - - {RsConnect} - - - -Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns. - -### From list of objects {#from-list-of-objects} - -You can provide a list of objects to create a table. The Python and TypeScript SDKs -support lists/arrays of dictionaries, while the Rust SDK supports lists of structs. -In Python, pass a list or other batch-like object; a single bare `dict` or single -`LanceModel` is rejected. - - - - {CreateTableFromDicts} - - - - {TsCreateTableFromDicts} - - - - {RsCreateTableFromDicts} - - - -### Handle existing tables {#handle-existing-tables} - -By default, `create_table` raises an error if a table with the same name already exists. -You can change this behavior with two parameters that resolve the conflict in different ways: - -- **Idempotent open**: return the existing table without modifying it. Use when your - code may run more than once (notebooks, retries, init scripts) and you want to reuse - the table on subsequent runs. The provided data is ignored, but the schema is - validated against the existing table and a mismatch raises an error. -- **Overwrite**: drop the existing table and create a new one with the provided data. - Use this for test fixtures or when you intentionally want to replace prior contents. - This permanently discards the old table's data. - - - - {CreateTableConflictHandling} - - - - {TsCreateTableConflictHandling} - - - - {RsCreateTableConflictHandling} - - - - -`exist_ok` / `existOk` does not append the provided data to an existing table. Use -[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and* -contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table). - - -### From a custom schema {#from-a-custom-schema} - -You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata. - - - - {CreateTableCustomSchema} - - - - {TsCreateTableCustomSchema} - - - - {RsCreateTableCustomSchema} - - - -An explicit schema is also where you control nullability. If later writes omit a -non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be -omitted or written with null values. Without an explicit schema, Python infers list-like vector -values as fixed-size `float32` vector fields from the observed dimension. - -For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length, -null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes -`fill_value`, and `"null"` writes nulls. - -### From an Arrow Table {#from-an-arrow-table} -You can also create LanceDB tables directly from Arrow tables. -Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow. - - - - {CreateTableFromArrow} - - - - {TsCreateTableFromArrow} - - - - {RsCreateTableFromArrow} - - - - -### From a Pandas DataFrame {#from-a-pandas-dataframe} -Python Only - - - - {CreateTableFromPandas} - - - - -Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly. - - - -The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type. - - -### From a Polars DataFrame {#from-a-polars-dataframe} -Python Only - -LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library -written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow -under the hood. A deeper integration between LanceDB Tables and Polars DataFrames -is on the way. - - - - {CreateTableFromPolars} - - - -### From Pydantic Models {#from-pydantic-models} -Python Only - -When you create an empty table without data, you must specify the table schema. -LanceDB supports creating tables by specifying a PyArrow schema or a specialized -Pydantic model called `LanceModel`. - -For example, the following Content model specifies a table with 5 columns: -`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can -pass the class as the value of the `schema` parameter to `create_table`. -The `vector` column is a `Vector` type, which is a specialized Pydantic type that -can be configured with the vector dimensions. It is also important to note that -LanceDB only understands subclasses of `lancedb.pydantic.LanceModel` -(which itself derives from `pydantic.BaseModel`). - - - - {CreateTableFromPydantic} - - - -#### Nested schemas {#nested-schemas} - -Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object: - - - - {TablesDocumentModel} - - - -This can be used as the type of a LanceDB table column: - - - - {CreateTableNestedSchema} - - - -This creates a struct column called "document" that has two subfields -called "content" and "source": - -```bash -In [28]: tbl.schema -Out[28]: -id: string not null -vector: fixed_size_list[1536] not null - child 0, item: float -document: struct not null - child 0, content: string not null - child 1, source: string not null -``` - -#### Validators {#validators} - -Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's -[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example -below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used -for a `created_at` field. - - - - {TablesTzValidator} - - - -When you run this code it, should raise the `ValidationError`. - -### Loading Large Datasets {#loading-large-datasets} - -When ingesting large datasets, use `table.add()` on an existing table rather than -passing all data to `create_table()`. The `add()` method auto-parallelizes large -writes, while `create_table(name, data)` does not. - - -For best performance with large datasets, create an empty table first and then call -`table.add()`. This enables automatic write parallelism for materialized data sources. - - -#### From files (Parquet, CSV, etc.) {#from-files-parquet-csv-etc} -Python Only - -For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams -data from disk without loading the entire dataset into memory. - - - - {AddFromDataset} - - - - -`pa.dataset()` input is currently Python-only. TypeScript and Rust support for -file-based dataset ingestion is tracked in -[lancedb#3173](https://github.com/lancedb/lancedb/issues/3173). - - -#### From iterators (custom batch generation) {#from-iterators-custom-batch-generation} - -When you need custom batch logic — generating embeddings on the fly, transforming -rows from an external source, etc. — use an iterator of `RecordBatch` objects. - - - - {CreateTableFromIterator} - - - - {TsCreateTableFromIterator} - - - - {RsCreateTableFromIterator} - - - -Use this pattern when: - -- Your source data already arrives in Arrow batches, readers, datasets, or streams. -- Materializing the entire ingest as one giant in-memory list or array would be too expensive. -- You want to control chunk size explicitly during ingestion. - -Python can also consume iterators of other supported types like Pandas DataFrames or Python lists. - -#### Write parallelism {#write-parallelism} - - -For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB -automatically parallelizes large writes — no configuration needed. Auto-parallelism -targets approximately 1M rows or 2GB per write partition. - -For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine -total size upfront. A `parallelism` parameter to control this manually is planned -but not yet exposed in Python or TypeScript -([tracking issue](https://github.com/lancedb/lancedb/issues/3173)). - - -#### Tracking ingestion progress {#tracking-ingestion-progress} -TypeScript Only - -For long-running writes, pass a `progress` callback to `table.add()` to surface -per-batch progress in your UI, logs, or metrics pipeline. The callback fires -once per batch written and once more with `done: true` when the write completes. - -Each invocation receives a `WriteProgress` object: - -| Field | Description | -|:------|:------------| -| `outputRows` | Rows written so far. | -| `outputBytes` | Bytes written so far. | -| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. | -| `elapsedSeconds` | Wall-clock seconds since the write started. | -| `activeTasks` | Parallel write tasks currently in flight. | -| `totalTasks` | Total parallel write tasks (the write parallelism). | -| `done` | `true` only for the final callback. | - - - - {TsAddProgress} - - - -A few things to know before you wire this up: - -- Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere. -- Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow. -- Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written. - -## Create empty table {#create-empty-table} -You can create an empty table for scenarios where you want to add data to the table later. -An example would be when you want to collect data from a stream/external file and then add it to a table in -batches. - -An empty table can be initialized via an Arrow schema. - - - - {CreateEmptyTable} - - - - {TsCreateEmptyTable} - - - - {RsCreateEmptyTable} - - - -Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not -directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel` -that has been extended to support LanceDB specific types like `Vector`. - - - - {CreateEmptyTablePydantic} - - - -Once the empty table has been created, you can append to it or modify its contents, -as explained in [Ingesting data](/ingestion). - -## Open an existing table {#open-an-existing-table} - -You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method. -If you forget the name of your table, you can always get a listing of all table names. - - - - {OpenExistingTable} - - - - {TsOpenExistingTable} - - - - {RsOpenExistingTable} - - - -## Drop a table {#drop-a-table} - -Use the `drop_table()` method on the database to remove a table. - - - - {DropTable} - - - - {TsDropTable} - - - - {RsDropTable} - - - -This permanently removes the table and is not recoverable, unlike deleting rows. -By default, if the table does not exist an exception is raised. To suppress this, -you can pass in `ignore_missing=True`. diff --git a/docs/web/tables/index.mdx b/docs/web/tables/index.mdx index 3c2f5e689..a731cee52 100644 --- a/docs/web/tables/index.mdx +++ b/docs/web/tables/index.mdx @@ -267,12 +267,12 @@ reuse the existing table instead of writing the supplied rows again. If you want to avoid overwriting an existing table, omit the overwrite mode. For append-only ingestion into a table that already exists, open the table and call `add(...)` instead of `create_table(...)`. For repeatable setup with `exist_ok` / `existOk`, see -[Handle existing tables](/tables/create#handle-existing-tables). +[Handle existing tables](/ingestion#handle-existing-tables). For more ingestion patterns, including PyArrow tables, Python `pyarrow.dataset.Dataset` inputs, empty tables, and Python `LanceModel` schemas with nested fields, see -[Ingesting data](/tables/create/). +[Ingesting data](/ingestion). ### From Pandas DataFrames {#from-pandas-dataframes} Python Only @@ -644,7 +644,7 @@ advanced features of LanceDB. Below are some suggested next pages. Learn the different approaches to creating and ingesting data into LanceDB tables from various sources. diff --git a/docs/web/tables/schema.mdx b/docs/web/tables/schema.mdx index ef8db1906..9b110abd9 100644 --- a/docs/web/tables/schema.mdx +++ b/docs/web/tables/schema.mdx @@ -1,6 +1,6 @@ --- -title: "Schema and Data Evolution" -sidebarTitle: "Schema & data evolution" +title: "Schema evolution" +sidebarTitle: "Schema Evolution" description: Learn how to manage table schemas in LanceDB, including adding, altering, and dropping columns. icon: "boxes-stacked" --- @@ -8,18 +8,12 @@ import { PySchemaAddSetup as SchemaAddSetup, TsSchemaAddSetup as TsSchemaAddSetup, RsSchemaAddSetup as RsSchemaAddSetup, - PyAddColumnsCalculated as AddColumnsCalculated, - TsAddColumnsCalculated as TsAddColumnsCalculated, - RsAddColumnsCalculated as RsAddColumnsCalculated, PyAddColumnsDefaultValues as AddColumnsDefaultValues, TsAddColumnsDefaultValues as TsAddColumnsDefaultValues, RsAddColumnsDefaultValues as RsAddColumnsDefaultValues, PyAddColumnsNullable as AddColumnsNullable, TsAddColumnsNullable as TsAddColumnsNullable, RsAddColumnsNullable as RsAddColumnsNullable, - PyAddFeatureColumnsSql as AddFeatureColumnsSql, - TsAddFeatureColumnsSql as TsAddFeatureColumnsSql, - RsAddFeatureColumnsSql as RsAddFeatureColumnsSql, PySchemaAlterSetup as SchemaAlterSetup, TsSchemaAlterSetup as TsSchemaAlterSetup, RsSchemaAlterSetup as RsSchemaAlterSetup, @@ -108,53 +102,11 @@ First, let's create a sample table with product data to demonstrate schema evolu -### Add derived columns {#add-derived-columns} - -You can add new columns that are derived from existing data using SQL expressions. -For feature engineering on large existing tables, group related derived features into -one `add_columns` operation instead of running many separate writes. This creates one -new table version for the schema change and computes the new columns from the existing -rows, which avoids growing the table's version history with many small updates. - - - - {AddColumnsCalculated} - - - - {TsAddColumnsCalculated} - - - - {RsAddColumnsCalculated} - - - -The same call can add multiple derived columns at once. For example, if you are -building several lightweight features from existing product fields, pass all of the -new column expressions together: - - - - {AddFeatureColumnsSql} - - - - {TsAddFeatureColumnsSql} - - - - {RsAddFeatureColumnsSql} - - - - -LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or -PyArrow `RecordBatch` iterators for populating new columns. New column values must be -defined with SQL expressions, or added as NULL columns from an Arrow field or schema. -If your transformation cannot be expressed in SQL, compute the values outside -`add_columns` before writing them back through another workflow. - + +Columns derived from existing data -- SQL expressions over other columns -- are +covered in [Data evolution](/feature-engineering/data-evolution), because +deriving values is a feature-engineering job rather than a schema one. + ### Add columns with default values {#add-columns-with-default-values}