mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
Brendan Clement
8cdff48859
Found by `mint broken-links --check-anchors` on the assembled site. Three point at sections that were renamed or moved: the tag-versioning heading, the branches comparison table, and the table-creation patterns, which now live on the Create a table page. Three named sections that no longer exist and now link to the page they were on.
126 lines
4.8 KiB
Plaintext
126 lines
4.8 KiB
Plaintext
---
|
|
title: "Ingesting data"
|
|
sidebarTitle: "Overview"
|
|
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
|
|
icon: inbox
|
|
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
|
---
|
|
|
|
import {
|
|
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
|
|
PyUpdateConnectLocal as UpdateConnectLocal,
|
|
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
|
|
TsUpdateConnectEnterprise,
|
|
TsUpdateConnectLocal,
|
|
TsUpdateExampleTableSetup,
|
|
RsUpdateConnectEnterprise,
|
|
RsUpdateConnectLocal,
|
|
RsUpdateExampleTableSetup,
|
|
RsUpdateMakeUsersReader,
|
|
} from '/snippets/tables.mdx';
|
|
|
|
In LanceDB, tables store records with a defined schema that specifies column
|
|
names and types. Across the SDKs, you can create tables from row-oriented data
|
|
and Apache Arrow data structures. The Python SDK additionally supports PyArrow
|
|
schemas for explicit schema control, and `LanceModel` for Pydantic-based
|
|
validation.
|
|
|
|
Getting data in has two halves: creating a table from data you already have, and
|
|
writing to a table that exists. This page covers the first and introduces the
|
|
second.
|
|
|
|
<Note>
|
|
Every operation here has a SQL form -- see [DML](/sql#dml).
|
|
</Note>
|
|
|
|
Start by [creating a table](/ingestion/create-table); the rest of this
|
|
section writes to one that exists.
|
|
|
|
## Choose a write method {#choose-a-write-method}
|
|
|
|
| Family | Method | Use this when... |
|
|
| --------------- | --------------- | ---------------- |
|
|
| [`add`](/ingestion/insert) | `add(data)` | You are appending new rows and do not need to match against existing ones. |
|
|
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
|
|
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
|
|
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
|
|
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
|
|
| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. |
|
|
|
|
Write operations return status metadata. For example, `update` reports updated row count and
|
|
committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports
|
|
inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table
|
|
handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest`
|
|
before modifying data.
|
|
The committed `version` advances even for writes that affect zero rows, such as a delete predicate
|
|
that matches nothing.
|
|
|
|
## Connect to LanceDB {#connect-to-lancedb}
|
|
|
|
Connect to your local LanceDB instance:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Or, connect to LanceDB Enterprise:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateMakeUsersReader}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Create the example table {#create-the-example-table}
|
|
|
|
We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table.
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Expected table contents:
|
|
|
|
| id | name | login_count |
|
|
| --- | --- | --- |
|
|
| 1 | Alice | 10 |
|
|
| 2 | Bob | 20 |
|
|
|
|
The example above shows a PyArrow schema. You can just as well create the table using other
|
|
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Create a table](/ingestion/create-table#create-a-table-with-data).
|