lancedb/docs/web/ingestion/index.mdx

---
title: "Ingesting data"
sidebarTitle: "Overview"
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
icon: inbox
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
---

import {
    TsConnect,
    RsConnect,
} from '/snippets/connection.mdx';
import {
    PyCreateTableFromDicts as CreateTableFromDicts,
    TsCreateTableFromDicts as TsCreateTableFromDicts,
    RsCreateTableFromDicts as RsCreateTableFromDicts,
    PyCreateTableConflictHandling as CreateTableConflictHandling,
    TsCreateTableConflictHandling as TsCreateTableConflictHandling,
    RsCreateTableConflictHandling as RsCreateTableConflictHandling,
    PyCreateTableFromPandas as CreateTableFromPandas,
    PyCreateTableCustomSchema as CreateTableCustomSchema,
    TsCreateTableCustomSchema as TsCreateTableCustomSchema,
    RsCreateTableCustomSchema as RsCreateTableCustomSchema,
    PyCreateTableFromPolars as CreateTableFromPolars,
    PyCreateTableFromArrow as CreateTableFromArrow,
    TsCreateTableFromArrow as TsCreateTableFromArrow,
    RsCreateTableFromArrow as RsCreateTableFromArrow,
    PyCreateTableFromPydantic as CreateTableFromPydantic,
    PyCreateTableNestedSchema as CreateTableNestedSchema,
    PyAddFromDataset as AddFromDataset,
    PyCreateTableFromIterator as CreateTableFromIterator,
    TsCreateTableFromIterator as TsCreateTableFromIterator,
    RsCreateTableFromIterator as RsCreateTableFromIterator,
    TsAddProgress,
    PyOpenExistingTable as OpenExistingTable,
    TsOpenExistingTable as TsOpenExistingTable,
    RsOpenExistingTable as RsOpenExistingTable,
    PyCreateEmptyTable as CreateEmptyTable,
    TsCreateEmptyTable as TsCreateEmptyTable,
    RsCreateEmptyTable as RsCreateEmptyTable,
    PyCreateEmptyTablePydantic as CreateEmptyTablePydantic,
    PyDropTable as DropTable,
    TsDropTable as TsDropTable,
    RsDropTable as RsDropTable,
    PyTablesBasicConnect as TablesBasicConnect,
    PyTablesDocumentModel as TablesDocumentModel,
    PyTablesTzValidator as TablesTzValidator,
    PyUpdateConnectEnterprise as UpdateConnectEnterprise,
    PyUpdateConnectLocal as UpdateConnectLocal,
    PyUpdateExampleTableSetup as UpdateExampleTableSetup,
    TsUpdateConnectEnterprise,
    TsUpdateConnectLocal,
    TsUpdateExampleTableSetup,
    RsUpdateConnectEnterprise,
    RsUpdateConnectLocal,
    RsUpdateExampleTableSetup,
    RsUpdateMakeUsersReader,
} from '/snippets/tables.mdx';

In LanceDB, tables store records with a defined schema that specifies column
names and types. Across the SDKs, you can create tables from row-oriented data
and Apache Arrow data structures. The Python SDK additionally supports PyArrow
schemas for explicit schema control, and `LanceModel` for Pydantic-based
validation.

Getting data in has two halves: creating a table from data you already have, and
writing to a table that exists. This page covers the first and introduces the
second.

## Create a table with data {#create-a-table-with-data}

Initialize a LanceDB connection and create a table

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {TablesBasicConnect}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsConnect}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsConnect}
    </CodeBlock>
</CodeGroup>

Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.

### From list of objects {#from-list-of-objects}

You can provide a list of objects to create a table. The Python and TypeScript SDKs
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
In Python, pass a list or other batch-like object; a single bare `dict` or single
`LanceModel` is rejected.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableFromDicts}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsCreateTableFromDicts}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsCreateTableFromDicts}
    </CodeBlock>
</CodeGroup>

### Handle existing tables {#handle-existing-tables}

By default, `create_table` raises an error if a table with the same name already exists.
You can change this behavior with two parameters that resolve the conflict in different ways:

- **Idempotent open**: return the existing table without modifying it. Use when your
  code may run more than once (notebooks, retries, init scripts) and you want to reuse
  the table on subsequent runs. The provided data is ignored, but the schema is
  validated against the existing table and a mismatch raises an error.
- **Overwrite**: drop the existing table and create a new one with the provided data.
  Use this for test fixtures or when you intentionally want to replace prior contents.
  This permanently discards the old table's data.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableConflictHandling}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsCreateTableConflictHandling}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsCreateTableConflictHandling}
    </CodeBlock>
</CodeGroup>

<Note>
`exist_ok` / `existOk` does not append the provided data to an existing table. Use
[`table.add()`](/ingestion/insert) for that. If you need to ensure a table exists *and*
contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).
</Note>

### From a custom schema {#from-a-custom-schema}

You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableCustomSchema}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsCreateTableCustomSchema}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsCreateTableCustomSchema}
    </CodeBlock>
</CodeGroup>

An explicit schema is also where you control nullability. If later writes omit a
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.

For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.

### From an Arrow Table {#from-an-arrow-table}
You can also create LanceDB tables directly from Arrow tables.
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableFromArrow}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsCreateTableFromArrow}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsCreateTableFromArrow}
    </CodeBlock>
</CodeGroup>


### From a Pandas DataFrame {#from-a-pandas-dataframe}
<Badge color="green">Python Only</Badge>

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableFromPandas}
    </CodeBlock>
</CodeGroup>

<Note title="Note">
Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.
</Note>

<Note title="Vector Column Type">
The **`vector`** column needs to be a [Vector](/integrations/data/pydantic#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.
</Note>

### From a Polars DataFrame {#from-a-polars-dataframe}
<Badge color="green">Python Only</Badge>

LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
is on the way.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableFromPolars}
    </CodeBlock>
</CodeGroup>

### From Pydantic Models {#from-pydantic-models}
<Badge color="green">Python Only</Badge>

When you create an empty table without data, you must specify the table schema.
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
Pydantic model called `LanceModel`.

For example, the following Content model specifies a table with 5 columns:
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
pass the class as the value of the `schema` parameter to `create_table`.
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
can be configured with the vector dimensions. It is also important to note that
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
(which itself derives from `pydantic.BaseModel`).

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableFromPydantic}
    </CodeBlock>
</CodeGroup>

#### Nested schemas {#nested-schemas}

Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {TablesDocumentModel}
    </CodeBlock>
</CodeGroup>

This can be used as the type of a LanceDB table column:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateTableNestedSchema}
    </CodeBlock>
</CodeGroup>

This creates a struct column called "document" that has two subfields
called "content" and "source":

```bash
In [28]: tbl.schema
Out[28]:
id: string not null
vector: fixed_size_list<item: float>[1536] not null
    child 0, item: float
document: struct<content: string not null, source: string not null> not null
    child 0, content: string not null
    child 1, source: string not null
```

#### Validators {#validators}

Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
for a `created_at` field.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {TablesTzValidator}
    </CodeBlock>
</CodeGroup>

When you run this code it, should raise the `ValidationError`.

<Note>
Datasets too large to pass to `create_table` at once are covered separately --
see [Insert data](/ingestion/insert#large-datasets), which explains why
`table.add()` parallelizes where `create_table` does not.
</Note>

## Create empty table {#create-empty-table}
You can create an empty table for scenarios where you want to add data to the table later.
An example would be when you want to collect data from a stream/external file and then add it to a table in
batches.

An empty table can be initialized via an Arrow schema.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateEmptyTable}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsCreateEmptyTable}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsCreateEmptyTable}
    </CodeBlock>
</CodeGroup>

Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
that has been extended to support LanceDB specific types like `Vector`.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {CreateEmptyTablePydantic}
    </CodeBlock>
</CodeGroup>

Once the empty table has been created, you can append to it or modify its contents,
as explained in [Ingesting data](/ingestion).

## Open an existing table {#open-an-existing-table}

You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
If you forget the name of your table, you can always get a listing of all table names.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {OpenExistingTable}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsOpenExistingTable}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsOpenExistingTable}
    </CodeBlock>
</CodeGroup>

## Drop a table {#drop-a-table}

Use the `drop_table()` method on the database to remove a table.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {DropTable}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsDropTable}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsDropTable}
    </CodeBlock>
</CodeGroup>

This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.

## Choose a write method {#choose-a-write-method}

| Family |  Method | Use this when... |
| --------------- | --------------- | ---------------- |
| [`add`](/ingestion/insert) | `add(data)` | You are appending new rows and do not need to match against existing ones. |
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. |

Write operations return status metadata. For example, `update` reports updated row count and
committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports
inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table
handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest`
before modifying data.
The committed `version` advances even for writes that affect zero rows, such as a delete predicate
that matches nothing.

## Connect to LanceDB {#connect-to-lancedb}

Connect to your local LanceDB instance:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {UpdateConnectLocal}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsUpdateConnectLocal}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsUpdateConnectLocal}
    </CodeBlock>
</CodeGroup>

Or, connect to LanceDB Enterprise:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {UpdateConnectEnterprise}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsUpdateConnectEnterprise}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsUpdateConnectEnterprise}
    </CodeBlock>
</CodeGroup>

In the Rust snippets, a `make_users_reader` helper is used  to build Arrow input data.

<CodeGroup>
    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsUpdateMakeUsersReader}
    </CodeBlock>
</CodeGroup>

## Create the example table {#create-the-example-table}

We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table.
<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {UpdateExampleTableSetup}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsUpdateExampleTableSetup}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsUpdateExampleTableSetup}
    </CodeBlock>
</CodeGroup>

Expected table contents:

| id | name | login_count |
| --- | --- | --- |
| 1 | Alice | 10 |
| 2 | Bob | 20 |

The example above shows a PyArrow schema. You can just as well create the table using other
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [above](#create-a-table-with-data).