mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
Jack Ye
52684d50b9
OSS docs now own the full usage content for features that only run against an Enterprise deployment, since the client interface itself lives here; Enterprise docs keep only the parts specific to its own behavior. Sophon's overlay for these five pages is dropped since the OSS versions are now the complete reference. SQL Reference is restructured to match an object-type SQL manual (DDL by object, then DML, then DQL, then Expressions) instead of the old flat, loosely-ordered page list, absorbing the SQL client-connection guide and FTS-via-SQL examples into it.
126 lines
4.7 KiB
Plaintext
126 lines
4.7 KiB
Plaintext
---
|
|
title: "Ingesting data"
|
|
sidebarTitle: "Overview"
|
|
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
|
|
icon: inbox
|
|
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
|
---
|
|
|
|
import {
|
|
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
|
|
PyUpdateConnectLocal as UpdateConnectLocal,
|
|
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
|
|
TsUpdateConnectEnterprise,
|
|
TsUpdateConnectLocal,
|
|
TsUpdateExampleTableSetup,
|
|
RsUpdateConnectEnterprise,
|
|
RsUpdateConnectLocal,
|
|
RsUpdateExampleTableSetup,
|
|
RsUpdateMakeUsersReader,
|
|
} from '/snippets/tables.mdx';
|
|
|
|
In LanceDB, tables store records with a defined schema that specifies column
|
|
names and types. Across the SDKs, you can create tables from row-oriented data
|
|
and Apache Arrow data structures. The Python SDK additionally supports PyArrow
|
|
schemas for explicit schema control, and `LanceModel` for Pydantic-based
|
|
validation.
|
|
|
|
Getting data in has two halves: creating a table from data you already have, and
|
|
writing to a table that exists. This page covers the first and introduces the
|
|
second.
|
|
|
|
<Note>
|
|
Every operation here has a SQL form -- see [DML](/sql#dml).
|
|
</Note>
|
|
|
|
Start by [creating a table](/ingestion/create-table); the rest of this
|
|
section writes to one that exists.
|
|
|
|
## Choose a write method {#choose-a-write-method}
|
|
|
|
| Family | Method | Use this when... |
|
|
| --------------- | --------------- | ---------------- |
|
|
| [`add`](/ingestion/insert) | `add(data)` | You are appending new rows and do not need to match against existing ones. |
|
|
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
|
|
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
|
|
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
|
|
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
|
|
| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. |
|
|
|
|
Write operations return status metadata. For example, `update` reports updated row count and
|
|
committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports
|
|
inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table
|
|
handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest`
|
|
before modifying data.
|
|
The committed `version` advances even for writes that affect zero rows, such as a delete predicate
|
|
that matches nothing.
|
|
|
|
## Connect to LanceDB {#connect-to-lancedb}
|
|
|
|
Connect to your local LanceDB instance:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Or, connect to LanceDB Enterprise:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateMakeUsersReader}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Create the example table {#create-the-example-table}
|
|
|
|
We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table.
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Expected table contents:
|
|
|
|
| id | name | login_count |
|
|
| --- | --- | --- |
|
|
| 1 | Alice | 10 |
|
|
| 2 | Bob | 20 |
|
|
|
|
The example above shows a PyArrow schema. You can just as well create the table using other
|
|
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [above](#create-a-table-with-data).
|