lancedb/docs/web/ingestion/insert.mdx

---
title: "Insert data"
sidebarTitle: "Insert"
description: "Append records, load file-backed data or batches, and verify ingestion."
icon: plus
availability:
  embedded: available
  enterprise: available
---

import {
    TsCreateTableFromIterator as TsCreateTableFromIterator,
    RsCreateTableFromIterator as RsCreateTableFromIterator,
} from '/snippets/tables.mdx';
import {
    PyAddFromDataset as AddFromDataset,
    PyCreateTableFromIterator as CreateTableFromIterator,
} from '/snippets/ingestion_examples.mdx';

import { RsExample as RsAddProgress } from '/snippets/ingestion_progress.mdx';
import { PyAddProgress, TsAddProgressBody, TsAddProgressTail } from '/snippets/ingestion_examples.mdx';
import { PyAppendRows } from '/snippets/ingestion_examples.mdx';
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';

Use `add()` to append records to an existing table. It does not compare incoming
keys with stored keys. Choose [merge insert](/ingestion/merge-insert) when the
same record may arrive again and should update an existing row.

## Append a first batch {#append}

Install the Python SDK with `pip install lancedb`. This complete example starts
with Alice, appends Bob and Charlie, and prints all three records.

<CodeBlock filename="append_rows.py" language="python" icon="python">
{PyAppendRows}
</CodeBlock>

The program uses temporary local storage. For a complete TypeScript or Rust
example, see [Complete write workflow](#complete-example).
Enterprise SQL offers [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT`.

## Prepare the input {#prepare}

Match the table's schema, including required fields and vector dimensions.
Supported input formats differ by SDK; see
[Create a table](/ingestion/create-table). Inspect conversion errors before
retrying, and verify the final row count and representative values.

## Choose input for a large load {#large-datasets}

Avoid building one large in-memory list when records already live in files
or arrive as batches. Create a table with the intended schema, then append
through the appropriate input path.

<span id="bulk-ingestion-for-data-you-already-have"></span>
### Read from files {#from-files-parquet-csv-etc}

Python accepts a `pyarrow.dataset.Dataset`. This example writes a small Parquet
file locally, creates an empty table with its schema, and appends the two records.
For your own data, point the dataset reader at the input files instead.

<CodeBlock filename="load_parquet.py" language="python" icon="python">
{AddFromDataset}
</CodeBlock>

A file-backed source can be scanned without first loading the whole dataset
into application memory. This Python input API is distinct from converting
files to Arrow batches yourself in another language.

<span id="iterator-ingestion-for-data-you-transform-on-the-fly"></span>
### Produce batches during ingestion {#from-iterators-custom-batch-generation}

Use batches when processing the input as it arrives. The Python example creates
an empty table and appends ten records produced by a batch iterator.

<CodeGroup>
<CodeBlock filename="Python" language="python" icon="python">
{CreateTableFromIterator}
</CodeBlock>
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsCreateTableFromIterator}
</CodeBlock>
<CodeBlock filename="Rust" language="rust" icon="rust">
{RsCreateTableFromIterator}
</CodeBlock>
</CodeGroup>

The language APIs differ: the TypeScript example appends each constructed Arrow
batch, while the Rust example supplies an Arrow reader when creating the table.
These are input-format examples, not equivalent parallel-write benchmarks.

### Control write parallelism {#write-parallelism}

Python's `add()` can estimate parallelism for materialized inputs. A streaming
reader may not expose its total size or support replay. Use `write_parallelism`
when controlling the number of concurrent partitions; increasing it can also
increase memory use. TypeScript does not expose that manual option.

Measure with realistic row sizes and storage. Avoid one write per row when a
batch will do, and avoid a large worker count for a small input.

## Track completion and progress {#tracking-ingestion-progress}

Python, TypeScript and Rust support progress callbacks when adding data.
These examples start with Alice, append Bob and Charlie, and print the number
of rows written as progress updates arrive. After the write succeeds, they
query the table to show all three records.

Each program uses a temporary local database and removes it when finished.

- **Python:** install `lancedb`, save the code as `insert_progress.py`, and run
  `python insert_progress.py`.
- **TypeScript:** install `@lancedb/lancedb`, save the code as `insert_progress.mts`,
  and run `node insert_progress.mts` with Node.js 22.18 or later.
- **Rust:** download the [Rust examples](https://raw.githubusercontent.com/lancedb/lancedb/2af85bea8999c8cd23ec01ff62ba110c54652c10/docs/web/static/code/ingestion-feedback-rust.zip),
  extract the project, and run `cargo run --locked --bin progress`.

<CodeGroup>
<CodeBlock filename="Python" language="python" icon="python">
{PyAddProgress}
</CodeBlock>
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsAddProgressBody}
{TsAddProgressTail}
</CodeBlock>
<CodeBlock filename="Rust" language="rust" icon="rust">
{RsAddProgress}
</CodeBlock>
</CodeGroup>

For a progress bar in Python, install `tqdm` and pass `progress=True` instead
of a callback.

### Read progress updates

Updates include row and byte counts, elapsed time, and parallel write activity.
The field names differ by language:

| Meaning | Python dictionary key | TypeScript property | Rust method |
|:--------|:----------------------|:--------------------|:------------|
| Rows written so far | `output_rows` | `outputRows` | `output_rows()` |
| Bytes written so far | `output_bytes` | `outputBytes` | `output_bytes()` |
| Expected row count, when known | `total_rows` | `totalRows` | `total_rows()` |
| Elapsed time | `elapsed_seconds` | `elapsedSeconds` | `elapsed()` |
| Active write tasks | `active_tasks` | `activeTasks` | `active_tasks()` |
| Total write tasks | `total_tasks` | `totalTasks` | `total_tasks()` |
| Final update | `done` | `done` | `done()` |

Python and TypeScript report elapsed time in seconds; Rust returns a `Duration`.
The expected row count may be unknown for streaming input. On the final update,
it falls back to the number of rows written.

For remote multipart writes, rows count input scanning, while bytes count
acknowledged IPC parts. Bytes start at zero and a retried part is counted once,
after acknowledgement. Scanned rows and staged bytes do not imply a commit.

Keep callbacks short and use the write operation's result to determine success.
The final `done` update means progress reporting has ended; it can also occur
after a write fails or is cancelled. Input errors before writing starts may
produce no updates.

TypeScript delivers callbacks asynchronously. Slow callbacks can queue up,
and the `add()` promise may resolve before every callback has run. Exceptions
from a TypeScript callback are logged and do not abort the write.

## Verify writes and handle retries {#performance}

Wait for the write operation's result, then check the expected records.
Progress callbacks describe activity; they do not establish that a write
succeeded. If a request fails after the server may have committed it, blindly
repeating an append can duplicate rows. Current clients do not automatically
replay non-idempotent writes after `500`, `502`, `503`, or `504`, or after a
response is lost. A plain `503` does not prove that a write was rejected before
commit. Connection failures before sending, `409` transaction conflicts, and
explicit `429` rate-limit rejections may be retried within `RetryConfig` limits. Idempotent multipart
parts have a separate retry policy described below.

If multipart completion times out, loses its response, or receives a `5xx`, the
error reports **commit outcome unknown**. The client neither aborts the upload
nor resends completion or the input under
a new upload ID: aborting at this point could delete already committed data.
Check the table using your record IDs before retrying the operation.

Configure connection-wide limits with `client_config.retry_config` in Python
(`lancedb.remote.RetryConfig`) or `clientConfig.retryConfig` in TypeScript.
Override the budget for one `add()` with Python `table.add(data, retries=0)`
or TypeScript `table.add(data, { retries: 0 })`. The value must be an integer
between 0 and 255. Rust write builders also provide a per-call override, for
example `table.add(data).retries(0).execute().await?`,
`table.update().retries(0)`, and `table.merge_insert(&["id"]).retries(0)`.
This disables transport retries for that operation, including multipart part
retries; it does not change local transaction conflict handling. Increasing a
retry budget never makes ambiguous commits safe to replay.

## Configure large uploads {#bulk-ingests-into-a-remote-table}

<Badge color="red">Enterprise</Badge>

This section applies only to LanceDB Enterprise tables (`db://` connections). LanceDB Embedded writes are unaffected.

Large measured inputs, unknown-length streams, and explicit parallel uploads use
multipart writes when the server supports them. The client slices large batches
before IPC encoding and buffers each part for replay. A disconnected request,
read timeout, or configured retryable staging status retries exactly the same
upload ID, part ID, and IPC bytes. `RetryConfig.statuses` controls status retries;
an empty list disables them. The default includes `500`, `502`, `503`, and `504`.
The server replaces that part's staged transaction, so losing
its response does not append the rows twice. Exhausting the part retry budget
aborts the uncommitted session; it does not restart the entire upload.
A complete request rejected with a transaction conflict (`409`) or rate limit
(`429`) also triggers best-effort cleanup. Cleanup uses the call's retry budget;
its failure does not replace the original error. Ambiguous completion outcomes
never trigger cleanup or replay.

Part size still affects memory and HTTP timeouts. A single blob row cannot be
split across Arrow batches; a very large row may require a larger timeout and
more memory. No connection-pool or keepalive setting guarantees that a write
response will arrive.

Two environment variables control how a multipart write's parts are cut. Both are picked up automatically by the Python and TypeScript clients, for clients that support these settings:

| Variable | Default | What it controls |
|----------|---------|------------------|
| `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` | 64 MiB | Target LZ4-compressed IPC bytes per replayable part. A single indivisible row and its IPC schema may exceed the target. `0` disables byte-based cuts and can require buffering a whole partition. |
| `LANCE_CLIENT_MAX_REQUEST_DURATION` | Half the read timeout | Time spent filling a replayable payload, in whole seconds. `0` disables time-based cuts. HTTP deadlines are configured separately in `timeout_config`. |

A part is cut after an encoding slice reaches either limit. Input batches are
sliced to approximately 8 MiB before encoding, preserving row order. Peak
additional memory scales with active writers and the part budget. Lower
`write_parallelism` or the byte budget to reduce memory use.

<CodeGroup>
```bash Bash icon="terminal"
export LANCE_CLIENT_MAX_BYTES_PER_REQUEST=33554432  # 32 MiB
export LANCE_CLIENT_MAX_REQUEST_DURATION=30        # payload preparation budget
```
</CodeGroup>

Splitting into more parts does not change the final table. The server stages the parts under their shared upload ID before committing
the write. This upload protocol is separate from repeating an application-level
`add()` after a timeout.

Remote `create_table(name, data)` remains one atomic request, including overwrite
and `exist_ok` modes. The client encodes into an automatically removed temporary
file before sending, rejects payloads larger than 256 MiB of compressed IPC, and
never silently creates an empty table as a fallback. Encoding and input failures
therefore issue no create request. Temporary disk usage is bounded by this budget;
individual oversized rows can still need substantial encoding memory.

For larger imports, create the schema first and upload with progress reporting.
The empty table becomes visible immediately; `add()` commits its rows atomically.
The following examples use an existing connection `db`, an Arrow `schema`, and
input `data` matching that schema (an Arrow reader in Rust):

<CodeGroup>
```python Python icon="python"
table = db.create_table("items", schema=schema)
table.add(data, progress=True)
```

```typescript TypeScript icon="square-js"
const table = await db.createEmptyTable("items", schema);
await table.add(data, {
  progress: (p) => console.log(`Rows written: ${p.outputRows}`),
});
```

```rust Rust icon="rust"
let table = db.create_empty_table("items", schema).execute().await?;
table
    .add(data)
    .progress(|p| println!("Rows written: {}", p.output_rows()))
    .execute()
    .await?;
```
</CodeGroup>

## Complete write workflow {#complete-example}

<Accordion title="Complete examples: create, append, update, upsert and delete">

Each program creates a temporary local database, performs the five operations,
prints the result, and removes its temporary data when finished.

- **Python:** install `lancedb`, save the code as `ingestion.py`, and run
  `python ingestion.py`.
- **TypeScript:** install `@lancedb/lancedb`, save the code as `ingestion.mts`,
  and run `node ingestion.mts` with Node.js 22.18 or later.
- **Rust:** download the [complete Rust project](https://raw.githubusercontent.com/lancedb/lancedb/2af85bea8999c8cd23ec01ff62ba110c54652c10/docs/web/static/code/ingestion-rust.zip),
  extract it, and run `cargo run --locked`. It includes `src/main.rs`, the Cargo
  manifest and a lockfile for the example's dependencies. The code is shown below.

<CodeGroup>
<CodeBlock filename="ingestion.py" language="python" icon="python">
{PyWriteWorkflow}
</CodeBlock>
<CodeBlock filename="ingestion.mts" language="typescript" icon="square-js">
{TsWriteWorkflowBody}
{TsWriteWorkflowTail}
</CodeBlock>
<CodeBlock filename="src/main.rs" language="rust" icon="rust">
{RsWriteWorkflow}
</CodeBlock>
</CodeGroup>

After the operations, Alice has 10 logins, Bobby has 21, and Charlie has 6.
Dana is selected for deletion and is absent from the final result.

</Accordion>