mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Brendan Clement
6a1645116e
Creating, writing, updating, merging and deleting rows, managing tables, branches and read consistency work in the embedded library and on a deployment, and those pages now say so. The SQL forms they mention need a deployment. Where the offerings differ, the section says how: a deployment reuses an existing table without comparing schemas, a merge insert can be refused past a configured number of unindexed rows, space is reclaimed by optimize() locally and by the deployment's compaction and configured cleanup otherwise, renaming depends on the catalog, a deployment clones shallowly only, drops all tables only locally, keeps freshness state per table handle and defaults freshness to its own setting, and refuses to delete a branch main depends on. table_exists works only on a namespace connection. Guarantees and limits span both offerings, so they carry section notes rather than a label: reads, retention, index freshness and compaction differ on a deployment, and the name and job limits are a deployment's.
136 lines
5.1 KiB
Plaintext
136 lines
5.1 KiB
Plaintext
---
|
|
title: "Ingesting data"
|
|
sidebarTitle: "Overview"
|
|
description: "Getting data into a table: creating one from your data, and the four write operations that follow."
|
|
icon: inbox
|
|
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Tables are created and written to in the embedded library and on a
|
|
deployment. The SQL forms of these writes need a deployment.
|
|
---
|
|
|
|
import {
|
|
PyUpdateConnectEnterprise as UpdateConnectEnterprise,
|
|
PyUpdateConnectLocal as UpdateConnectLocal,
|
|
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
|
|
TsUpdateConnectEnterprise,
|
|
TsUpdateConnectLocal,
|
|
TsUpdateExampleTableSetup,
|
|
RsUpdateConnectEnterprise,
|
|
RsUpdateConnectLocal,
|
|
RsUpdateExampleTableSetup,
|
|
RsUpdateMakeUsersReader,
|
|
} from '/snippets/tables.mdx';
|
|
|
|
In LanceDB, tables store records with a defined schema that specifies column
|
|
names and types. Across the SDKs, you can create tables from row-oriented data
|
|
and Apache Arrow data structures. The Python SDK additionally supports PyArrow
|
|
schemas for explicit schema control, and `LanceModel` for Pydantic-based
|
|
validation.
|
|
|
|
Getting data in has two halves: creating a table from data you already have, and
|
|
writing to a table that exists. This page covers the first and introduces the
|
|
second.
|
|
|
|
<Note>
|
|
On a deployment, every operation here except merge insert also has a SQL form --
|
|
see [DML](/sql#dml).
|
|
</Note>
|
|
|
|
Start by [creating a table](/ingestion/create-table); the rest of this
|
|
section writes to one that exists.
|
|
|
|
## Choose a write method {#choose-a-write-method}
|
|
|
|
| Family | Method | Use this when... |
|
|
| --------------- | --------------- | ---------------- |
|
|
| [`add`](/ingestion/insert) | `add(data)` | You are appending new rows and do not need to match against existing ones. |
|
|
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
|
|
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
|
|
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
|
|
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
|
|
| `merge_insert` | `.when_not_matched_by_source_delete(...)` | You want to remove target rows that are missing from the incoming source set. |
|
|
|
|
Write operations return status metadata. For example, `update` reports updated row count and
|
|
committed `version`, `delete` reports deleted row count and `version`, and `merge_insert` reports
|
|
inserted, updated, deleted, retry-attempt, and `version` fields. Writes require a mutable table
|
|
handle; after checking out an older version for reads, call `checkout_latest` / `checkoutLatest`
|
|
before modifying data.
|
|
The committed `version` advances even for writes that affect zero rows, such as a delete predicate
|
|
that matches nothing.
|
|
|
|
## Connect to LanceDB {#connect-to-lancedb}
|
|
|
|
Connect to your local LanceDB instance:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectLocal}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Or, connect to LanceDB Enterprise:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateConnectEnterprise}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
For a deployment with its own endpoint, also pass that endpoint as
|
|
`host_override` -- see [Connect to LanceDB](/basics/connect#enterprise).
|
|
|
|
In the Rust snippets, a `make_users_reader` helper is used to build Arrow input data.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateMakeUsersReader}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Create the example table {#create-the-example-table}
|
|
|
|
We'll start by creating a simple table with `id`, `name`, and `login_count` columns. All examples below use the same table.
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{UpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsUpdateExampleTableSetup}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Expected table contents:
|
|
|
|
| id | name | login_count |
|
|
| --- | --- | --- |
|
|
| 1 | Alice | 10 |
|
|
| 2 | Bob | 20 |
|
|
|
|
The example above shows a PyArrow schema. You can just as well create the table using other
|
|
table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see [Create a table](/ingestion/create-table#create-a-table-with-data).
|