mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-24 20:52:29 +00:00
Brendan Clement
758f738b8c
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. Ingestion, search and integration examples are incomplete or inconsistent across languages, and some guidance misstates SDK behavior. This brings the guides and their example sources together. It includes complete Python, TypeScript and Rust workflows, isolated provider and reranker examples, the Superlinked corrections, and guidance on retries that can duplicate writes. Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after #4491 is squash-merged. Local assembly, Mint validation and link checks pass. Example code is unchanged from the previously validated package. Publication stays held through integration.
98 lines
4.4 KiB
Plaintext
98 lines
4.4 KiB
Plaintext
---
|
|
title: "Ingesting data"
|
|
sidebarTitle: "Overview"
|
|
description: "Choose how to create a table, append records or change existing data."
|
|
icon: inbox
|
|
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Create tables and write records through the SDKs in OSS and Enterprise. Enterprise also provides SQL write statements.
|
|
---
|
|
|
|
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';
|
|
|
|
Ingestion brings data into a LanceDB table. Start by choosing a schema and
|
|
creating the table, then choose a write operation according to how incoming
|
|
records relate to data already stored.
|
|
|
|
## Choose a write operation {#choose-a-write-method}
|
|
|
|
| Task | Operation | What happens |
|
|
| --- | --- | --- |
|
|
| [Create a table](/ingestion/create-table) | `create_table` / `createTable` | Create a table from data or an explicit schema. |
|
|
| [Append records](/ingestion/insert) | `add` | Add rows without matching existing keys. |
|
|
| [Upsert by key](/ingestion/merge-insert) | `merge_insert` / `mergeInsert` | Choose whether matching rows are updated and new keys inserted. |
|
|
| [Update selected rows](/ingestion/update) | `update` | Change values in rows selected by a predicate. |
|
|
| [Delete selected rows](/ingestion/delete) | `delete` | Remove matching rows from the current table contents. |
|
|
|
|
Appending the same input twice can create duplicate records. Use merge insert
|
|
with a stable key when the incoming data should update existing records.
|
|
|
|
## Connect and prepare the data {#connect-to-lancedb}
|
|
|
|
Use the [connection guide](/basics/connect) to connect to OSS or Enterprise.
|
|
The SDK write operations in this section work with both. Enterprise also
|
|
supports [SQL write statements](/sql#dml); an SDK filter expression does not
|
|
require a separate SQL connection.
|
|
|
|
Choose the column types, required fields and vector dimensions before a large
|
|
load. SDKs support different input representations, including row objects and
|
|
Arrow batches. The [table creation guide](/ingestion/create-table) covers
|
|
those choices.
|
|
|
|
## Try a complete write workflow {#create-the-example-table}
|
|
|
|
This example starts with Alice and Bob, appends Charlie, updates Bob's name
|
|
and login count, then merges a batch containing Charlie and Dana. Finally, it
|
|
selects Dana's row for inspection and deletes it.
|
|
|
|
<Accordion title="Complete examples: create, append, update, upsert and delete">
|
|
|
|
Each program creates a temporary local database, performs the five operations,
|
|
prints the result, and removes its temporary data when finished.
|
|
|
|
- **Python:** install `lancedb`, save the code as `ingestion.py`, and run
|
|
`python ingestion.py`.
|
|
- **TypeScript:** install `@lancedb/lancedb`, save the code as `ingestion.mts`,
|
|
and run `node ingestion.mts` with Node.js 22.18 or later.
|
|
- **Rust:** download the [complete Rust project](/static/code/ingestion-rust.zip),
|
|
extract it, and run `cargo run --locked`. It includes `src/main.rs`, the Cargo
|
|
manifest and a lockfile for the example's dependencies. The code is shown below.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="ingestion.py" language="python" icon="python">
|
|
{PyWriteWorkflow}
|
|
</CodeBlock>
|
|
<CodeBlock filename="ingestion.mts" language="typescript" icon="square-js">
|
|
{TsWriteWorkflowBody}
|
|
{TsWriteWorkflowTail}
|
|
</CodeBlock>
|
|
<CodeBlock filename="src/main.rs" language="rust" icon="rust">
|
|
{RsWriteWorkflow}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
After the operations, Alice has 10 logins, Bobby has 21, and Charlie has 6.
|
|
Dana is selected for deletion and is absent from the final result.
|
|
|
|
</Accordion>
|
|
|
|
## Verify the result {#verify}
|
|
|
|
After a write completes, inspect the affected keys and fields and check the row
|
|
count. SDK return values differ by operation and language; use the relevant
|
|
API reference when relying on returned counts or version metadata.
|
|
|
|
Readers with older snapshots may not see another handle's writes immediately.
|
|
See [Consistency](/tables/consistency). A table checked out at a historical
|
|
version must return to the latest version before it can be modified.
|
|
|
|
## Load larger datasets {#large-loads}
|
|
|
|
Use batches or file-backed input when the dataset is too large to hold in
|
|
memory. The [insert guide](/ingestion/insert#large-datasets) covers input
|
|
formats, progress reporting and write parallelism. Keep retry behavior in mind:
|
|
repeating an append is not the same as retrying a key-based upsert.
|