mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
Remote multipart progress currently counts every scanned row before the corresponding IPC part is acknowledged. A rejected first part can therefore report uploaded rows, even though its acknowledged byte count remains zero. Count both rows and IPC bytes only after each part succeeds, including parts assembled from multiple batches or split from one batch. Retried parts count once. Native and single-request writes retain their existing accounting; acknowledged multipart progress describes staged data, not a successful commit. Validation follows the actual TCP upload path: one scanned batch becomes six decoded IPC parts, the sixth response is lost, and each request/retry observes only previously acknowledged rows and bytes. Rejection and complete-outcome cases preserve the same accounting boundary. This validation covers local HTTP behavior; deployed service throughput and the original disconnect investigation remain outside its scope. Frozen candidate validation: 839 remote tests and 7 progress tests passed. Scoped Clippy (`lancedb`, remote feature, lib/tests/examples, warnings denied) passed with unchanged source. Python/Node bindings were unchanged and were not rebuilt for this patch.
311 lines
14 KiB
Plaintext
311 lines
14 KiB
Plaintext
---
|
|
title: "Insert data"
|
|
sidebarTitle: "Insert"
|
|
description: "Append records, load file-backed data or batches, and verify ingestion."
|
|
icon: plus
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
import {
|
|
TsCreateTableFromIterator as TsCreateTableFromIterator,
|
|
RsCreateTableFromIterator as RsCreateTableFromIterator,
|
|
} from '/snippets/tables.mdx';
|
|
import {
|
|
PyAddFromDataset as AddFromDataset,
|
|
PyCreateTableFromIterator as CreateTableFromIterator,
|
|
} from '/snippets/ingestion_examples.mdx';
|
|
|
|
import { RsExample as RsAddProgress } from '/snippets/ingestion_progress.mdx';
|
|
import { PyAddProgress, TsAddProgressBody, TsAddProgressTail } from '/snippets/ingestion_examples.mdx';
|
|
import { PyAppendRows } from '/snippets/ingestion_examples.mdx';
|
|
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';
|
|
|
|
Use `add()` to append records to an existing table. It does not compare incoming
|
|
keys with stored keys. Choose [merge insert](/ingestion/merge-insert) when the
|
|
same record may arrive again and should update an existing row.
|
|
|
|
## Append a first batch {#append}
|
|
|
|
Install the Python SDK with `pip install lancedb`. This complete example starts
|
|
with Alice, appends Bob and Charlie, and prints all three records.
|
|
|
|
<CodeBlock filename="append_rows.py" language="python" icon="python">
|
|
{PyAppendRows}
|
|
</CodeBlock>
|
|
|
|
The program uses temporary local storage. For a complete TypeScript or Rust
|
|
example, see [Complete write workflow](#complete-example).
|
|
Enterprise SQL offers [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT`.
|
|
|
|
## Prepare the input {#prepare}
|
|
|
|
Match the table's schema, including required fields and vector dimensions.
|
|
Supported input formats differ by SDK; see
|
|
[Create a table](/ingestion/create-table). Inspect conversion errors before
|
|
retrying, and verify the final row count and representative values.
|
|
|
|
## Choose input for a large load {#large-datasets}
|
|
|
|
Avoid building one large in-memory list when records already live in files
|
|
or arrive as batches. Create a table with the intended schema, then append
|
|
through the appropriate input path.
|
|
|
|
<span id="bulk-ingestion-for-data-you-already-have"></span>
|
|
### Read from files {#from-files-parquet-csv-etc}
|
|
|
|
Python accepts a `pyarrow.dataset.Dataset`. This example writes a small Parquet
|
|
file locally, creates an empty table with its schema, and appends the two records.
|
|
For your own data, point the dataset reader at the input files instead.
|
|
|
|
<CodeBlock filename="load_parquet.py" language="python" icon="python">
|
|
{AddFromDataset}
|
|
</CodeBlock>
|
|
|
|
A file-backed source can be scanned without first loading the whole dataset
|
|
into application memory. This Python input API is distinct from converting
|
|
files to Arrow batches yourself in another language.
|
|
|
|
<span id="iterator-ingestion-for-data-you-transform-on-the-fly"></span>
|
|
### Produce batches during ingestion {#from-iterators-custom-batch-generation}
|
|
|
|
Use batches when processing the input as it arrives. The Python example creates
|
|
an empty table and appends ten records produced by a batch iterator.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="python" icon="python">
|
|
{CreateTableFromIterator}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsCreateTableFromIterator}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="rust" icon="rust">
|
|
{RsCreateTableFromIterator}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The language APIs differ: the TypeScript example appends each constructed Arrow
|
|
batch, while the Rust example supplies an Arrow reader when creating the table.
|
|
These are input-format examples, not equivalent parallel-write benchmarks.
|
|
|
|
### Control write parallelism {#write-parallelism}
|
|
|
|
Python's `add()` can estimate parallelism for materialized inputs. A streaming
|
|
reader may not expose its total size or support replay. Use `write_parallelism`
|
|
when controlling the number of concurrent partitions; increasing it can also
|
|
increase memory use. TypeScript does not expose that manual option.
|
|
|
|
Measure with realistic row sizes and storage. Avoid one write per row when a
|
|
batch will do, and avoid a large worker count for a small input.
|
|
|
|
## Track completion and progress {#tracking-ingestion-progress}
|
|
|
|
Python, TypeScript and Rust support progress callbacks when adding data.
|
|
These examples start with Alice, append Bob and Charlie, and print the number
|
|
of rows written as progress updates arrive. After the write succeeds, they
|
|
query the table to show all three records.
|
|
|
|
Each program uses a temporary local database and removes it when finished.
|
|
|
|
- **Python:** install `lancedb`, save the code as `insert_progress.py`, and run
|
|
`python insert_progress.py`.
|
|
- **TypeScript:** install `@lancedb/lancedb`, save the code as `insert_progress.mts`,
|
|
and run `node insert_progress.mts` with Node.js 22.18 or later.
|
|
- **Rust:** download the [Rust examples](https://raw.githubusercontent.com/lancedb/lancedb/2af85bea8999c8cd23ec01ff62ba110c54652c10/docs/web/static/code/ingestion-feedback-rust.zip),
|
|
extract the project, and run `cargo run --locked --bin progress`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="python" icon="python">
|
|
{PyAddProgress}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsAddProgressBody}
|
|
{TsAddProgressTail}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="rust" icon="rust">
|
|
{RsAddProgress}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
For a progress bar in Python, install `tqdm` and pass `progress=True` instead
|
|
of a callback.
|
|
|
|
### Read progress updates
|
|
|
|
Updates include row and byte counts, elapsed time, and parallel write activity.
|
|
The field names differ by language:
|
|
|
|
| Meaning | Python dictionary key | TypeScript property | Rust method |
|
|
|:--------|:----------------------|:--------------------|:------------|
|
|
| Rows written so far | `output_rows` | `outputRows` | `output_rows()` |
|
|
| Bytes written so far | `output_bytes` | `outputBytes` | `output_bytes()` |
|
|
| Expected row count, when known | `total_rows` | `totalRows` | `total_rows()` |
|
|
| Elapsed time | `elapsed_seconds` | `elapsedSeconds` | `elapsed()` |
|
|
| Active write tasks | `active_tasks` | `activeTasks` | `active_tasks()` |
|
|
| Total write tasks | `total_tasks` | `totalTasks` | `total_tasks()` |
|
|
| Final update | `done` | `done` | `done()` |
|
|
|
|
Python and TypeScript report elapsed time in seconds; Rust returns a `Duration`.
|
|
The expected row count may be unknown for streaming input. On the final update,
|
|
it falls back to the number of rows written.
|
|
|
|
For remote multipart writes, both rows and IPC bytes count acknowledged parts.
|
|
Both start at zero; a retried part is counted once, after acknowledgement.
|
|
Scanning input or sending a rejected part does not advance these counts.
|
|
Acknowledged parts are staged and do not imply a successful commit.
|
|
|
|
Keep callbacks short and use the write operation's result to determine success.
|
|
The final `done` update means progress reporting has ended; it can also occur
|
|
after a write fails or is cancelled. Input errors before writing starts may
|
|
produce no updates.
|
|
|
|
TypeScript delivers callbacks asynchronously. Slow callbacks can queue up,
|
|
and the `add()` promise may resolve before every callback has run. Exceptions
|
|
from a TypeScript callback are logged and do not abort the write.
|
|
|
|
## Verify writes and handle retries {#performance}
|
|
|
|
Wait for the write operation's result, then check the expected records.
|
|
Progress callbacks describe activity; they do not establish that a write
|
|
succeeded. If a request fails after the server may have committed it, blindly
|
|
repeating an append can duplicate rows. Current clients do not automatically
|
|
replay non-idempotent writes after `500`, `502`, `503`, or `504`, or after a
|
|
response is lost. A plain `503` does not prove that a write was rejected before
|
|
commit. Connection failures before sending, `409` transaction conflicts, and
|
|
explicit `429` rate-limit rejections may be retried within `RetryConfig` limits. Idempotent multipart
|
|
parts have a separate retry policy described below.
|
|
|
|
If multipart completion times out, loses its response, or receives a `5xx`, the
|
|
error reports **commit outcome unknown**. The client neither aborts the upload
|
|
nor resends completion or the input under
|
|
a new upload ID: aborting at this point could delete already committed data.
|
|
Check the table using your record IDs before retrying the operation.
|
|
|
|
Configure connection-wide limits with `client_config.retry_config` in Python
|
|
(`lancedb.remote.RetryConfig`) or `clientConfig.retryConfig` in TypeScript.
|
|
Override the budget for one `add()` with Python `table.add(data, retries=0)`
|
|
or TypeScript `table.add(data, { retries: 0 })`. The value must be an integer
|
|
between 0 and 255. Rust write builders also provide a per-call override, for
|
|
example `table.add(data).retries(0).execute().await?`,
|
|
`table.update().retries(0)`, and `table.merge_insert(&["id"]).retries(0)`.
|
|
This disables transport retries for that operation, including multipart part
|
|
retries; it does not change local transaction conflict handling. Increasing a
|
|
retry budget never makes ambiguous commits safe to replay.
|
|
|
|
## Configure large uploads {#bulk-ingests-into-a-remote-table}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
This section applies only to LanceDB Enterprise tables (`db://` connections). LanceDB Embedded writes are unaffected.
|
|
|
|
Large measured inputs, unknown-length streams, and explicit parallel uploads use
|
|
multipart writes when the server supports them. The client slices large batches
|
|
before IPC encoding and buffers each part for replay. A disconnected request,
|
|
read timeout, or configured retryable staging status retries exactly the same
|
|
upload ID, part ID, and IPC bytes. `RetryConfig.statuses` controls status retries;
|
|
an empty list disables them. The default includes `500`, `502`, `503`, and `504`.
|
|
The server replaces that part's staged transaction, so losing
|
|
its response does not append the rows twice. Exhausting the part retry budget
|
|
aborts the uncommitted session; it does not restart the entire upload.
|
|
A complete request rejected with a transaction conflict (`409`) or rate limit
|
|
(`429`) also triggers best-effort cleanup. Cleanup uses the call's retry budget;
|
|
its failure does not replace the original error. Ambiguous completion outcomes
|
|
never trigger cleanup or replay.
|
|
|
|
Part size still affects memory and HTTP timeouts. A single blob row cannot be
|
|
split across Arrow batches; a very large row may require a larger timeout and
|
|
more memory. No connection-pool or keepalive setting guarantees that a write
|
|
response will arrive.
|
|
|
|
Two environment variables control how a multipart write's parts are cut. Both are picked up automatically by the Python and TypeScript clients, for clients that support these settings:
|
|
|
|
| Variable | Default | What it controls |
|
|
|----------|---------|------------------|
|
|
| `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` | 64 MiB | Target LZ4-compressed IPC bytes per replayable part. A single indivisible row and its IPC schema may exceed the target. `0` disables byte-based cuts and can require buffering a whole partition. |
|
|
| `LANCE_CLIENT_MAX_REQUEST_DURATION` | Half the read timeout | Time spent filling a replayable payload, in whole seconds. `0` disables time-based cuts. HTTP deadlines are configured separately in `timeout_config`. |
|
|
|
|
A part is cut after an encoding slice reaches either limit. Input batches are
|
|
sliced to approximately 8 MiB before encoding, preserving row order. Peak
|
|
additional memory scales with active writers and the part budget. Lower
|
|
`write_parallelism` or the byte budget to reduce memory use.
|
|
|
|
<CodeGroup>
|
|
```bash Bash icon="terminal"
|
|
export LANCE_CLIENT_MAX_BYTES_PER_REQUEST=33554432 # 32 MiB
|
|
export LANCE_CLIENT_MAX_REQUEST_DURATION=30 # payload preparation budget
|
|
```
|
|
</CodeGroup>
|
|
|
|
Splitting into more parts does not change the final table. The server stages the parts under their shared upload ID before committing
|
|
the write. This upload protocol is separate from repeating an application-level
|
|
`add()` after a timeout.
|
|
|
|
Remote `create_table(name, data)` remains one atomic request, including overwrite
|
|
and `exist_ok` modes. The client encodes into an automatically removed temporary
|
|
file before sending, rejects payloads larger than 256 MiB of compressed IPC, and
|
|
never silently creates an empty table as a fallback. Encoding and input failures
|
|
therefore issue no create request. Temporary disk usage is bounded by this budget;
|
|
individual oversized rows can still need substantial encoding memory.
|
|
|
|
For larger imports, create the schema first and upload with progress reporting.
|
|
The empty table becomes visible immediately; `add()` commits its rows atomically.
|
|
The following examples use an existing connection `db`, an Arrow `schema`, and
|
|
input `data` matching that schema (an Arrow reader in Rust):
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table = db.create_table("items", schema=schema)
|
|
table.add(data, progress=True)
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
const table = await db.createEmptyTable("items", schema);
|
|
await table.add(data, {
|
|
progress: (p) => console.log(`Rows written: ${p.outputRows}`),
|
|
});
|
|
```
|
|
|
|
```rust Rust icon="rust"
|
|
let table = db.create_empty_table("items", schema).execute().await?;
|
|
table
|
|
.add(data)
|
|
.progress(|p| println!("Rows written: {}", p.output_rows()))
|
|
.execute()
|
|
.await?;
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Complete write workflow {#complete-example}
|
|
|
|
<Accordion title="Complete examples: create, append, update, upsert and delete">
|
|
|
|
Each program creates a temporary local database, performs the five operations,
|
|
prints the result, and removes its temporary data when finished.
|
|
|
|
- **Python:** install `lancedb`, save the code as `ingestion.py`, and run
|
|
`python ingestion.py`.
|
|
- **TypeScript:** install `@lancedb/lancedb`, save the code as `ingestion.mts`,
|
|
and run `node ingestion.mts` with Node.js 22.18 or later.
|
|
- **Rust:** download the [complete Rust project](https://raw.githubusercontent.com/lancedb/lancedb/2af85bea8999c8cd23ec01ff62ba110c54652c10/docs/web/static/code/ingestion-rust.zip),
|
|
extract it, and run `cargo run --locked`. It includes `src/main.rs`, the Cargo
|
|
manifest and a lockfile for the example's dependencies. The code is shown below.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="ingestion.py" language="python" icon="python">
|
|
{PyWriteWorkflow}
|
|
</CodeBlock>
|
|
<CodeBlock filename="ingestion.mts" language="typescript" icon="square-js">
|
|
{TsWriteWorkflowBody}
|
|
{TsWriteWorkflowTail}
|
|
</CodeBlock>
|
|
<CodeBlock filename="src/main.rs" language="rust" icon="rust">
|
|
{RsWriteWorkflow}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
After the operations, Alice has 10 logins, Bobby has 21, and Charlie has 6.
|
|
Dana is selected for deletion and is absent from the final result.
|
|
|
|
</Accordion>
|