mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
docs: one page per object in Manage
Namespace and Table were the only objects behind a nested group. Both are now a single page like Database, Materialized view, View and Function, so Manage is a flat list of the things a catalog holds. Namespace merges its two pages, which were a concept page and a usage page for the same object. Table is a new page. The old tables/index was a 672-line basic-usage tutorial whose every section had acquired a better home during the restructure -- connecting, ingesting, searching, filtering, adding and dropping columns -- to the point that its own Next steps cards pointed at the pages that superseded it. It is replaced by a table object page: what a table carries, create, open, list, rename, clone, drop, and links out. Schema evolution, versioning and branching stay as siblings rather than folding in. Merging them would have made one 1,800-line page, which is not what the other object pages look like.
This commit is contained in:
Jack Ye
@@ -19,8 +19,8 @@ feature store. The catalog part is what this page is about.
|
||||
| Object | What it is |
|
||||
| --- | --- |
|
||||
| [Database](/management/database) | The root. A connection opens exactly one. |
|
||||
| [Namespace](/namespaces) | A named grouping inside a database. Can nest. |
|
||||
| [Table](/tables) | Rows and columns, versioned. What most operations act on. |
|
||||
| [Namespace](/management/namespace) | A named grouping inside a database. Can nest. |
|
||||
| [Table](/management/table) | Rows and columns, versioned. What most operations act on. |
|
||||
| [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. |
|
||||
| [View](/management/view) | A saved query. Computed when read, not stored. |
|
||||
| [Function](/management/function) | Registered code, applied to produce column values. |
|
||||
|
||||
+17
-16
@@ -90,6 +90,18 @@
|
||||
{
|
||||
"source": "/tutorials/feature-engineering",
|
||||
"destination": "/tutorials/feature-engineering/feature-engineering-101"
|
||||
},
|
||||
{
|
||||
"source": "/tables",
|
||||
"destination": "/management/table"
|
||||
},
|
||||
{
|
||||
"source": "/namespaces",
|
||||
"destination": "/management/namespace"
|
||||
},
|
||||
{
|
||||
"source": "/namespaces/usage",
|
||||
"destination": "/management/namespace"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -124,22 +136,11 @@
|
||||
"pages": [
|
||||
"management/index",
|
||||
"management/database",
|
||||
{
|
||||
"group": "Namespace",
|
||||
"pages": [
|
||||
"namespaces/index",
|
||||
"namespaces/usage"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Table",
|
||||
"pages": [
|
||||
"tables/index",
|
||||
"tables/schema",
|
||||
"tables/versioning",
|
||||
"tables/branching"
|
||||
]
|
||||
},
|
||||
"management/namespace",
|
||||
"management/table",
|
||||
"tables/schema",
|
||||
"tables/versioning",
|
||||
"tables/branching",
|
||||
"management/materialized-view",
|
||||
"management/function"
|
||||
]
|
||||
|
||||
+1
-1
@@ -94,7 +94,7 @@ for agents. Start here:
|
||||
<Card
|
||||
title="Basic Table Operations"
|
||||
icon="table"
|
||||
href="/tables/"
|
||||
href="/management/table"
|
||||
>
|
||||
Create tables, evolve schemas, version data, and modify rows in LanceDB.
|
||||
</Card>
|
||||
|
||||
@@ -110,7 +110,7 @@ SELECT animal, noise, vector, _hybrid_score, _distance, _score
|
||||
|
||||
A directory namespace maps a LanceDB catalog root to namespace-qualified table identifiers in DuckDB. This keeps table discovery and table naming stable as your project grows.
|
||||
|
||||
To learn more about the catalog and namespace model, see [Namespaces and the Catalog Model](/namespaces).
|
||||
To learn more about the catalog and namespace model, see [Namespaces and the Catalog Model](/management/namespace).
|
||||
|
||||
|
||||
## Advanced usage {#advanced-usage}
|
||||
|
||||
@@ -169,7 +169,7 @@ DROP TABLE documents;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Each takes a `namespace_path` to act inside a [namespace](/namespaces) rather
|
||||
Each takes a `namespace_path` to act inside a [namespace](/management/namespace) rather
|
||||
than at the root.
|
||||
|
||||
<Warning>
|
||||
|
||||
@@ -30,7 +30,7 @@ See [Python, TypeScript, and SQL](/basics/apis).
|
||||
| exists | `namespace_exists` | -- | -- |
|
||||
| drop | `drop_namespace` | `dropNamespace` | -- |
|
||||
|
||||
See [Namespaces](/namespaces).
|
||||
See [Namespaces](/management/namespace).
|
||||
|
||||
### Tables {#tables}
|
||||
|
||||
@@ -46,7 +46,7 @@ See [Namespaces](/namespaces).
|
||||
| drop | `drop_table`, `drop_table_async` | `dropTable`, `dropTableAsync` | `DROP TABLE` |
|
||||
| drop all | `drop_all_tables` | `dropAllTables` | -- |
|
||||
|
||||
See [Tables](/tables) for creating one and shaping its schema.
|
||||
See [Tables](/management/table) for creating one and shaping its schema.
|
||||
|
||||
### Materialized views {#materialized-views}
|
||||
|
||||
|
||||
@@ -1,11 +1,20 @@
|
||||
---
|
||||
title: "Namespaces and the Catalog Model"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Understand LanceDB as a catalog-level abstraction over Lance's table format, and learn how namespaces help organize Lance tables."
|
||||
icon: "sitemap"
|
||||
keywords: ["namespace", "catalog", "lance format", "table format", "lancedb"]
|
||||
title: "Namespaces"
|
||||
sidebarTitle: "Namespace"
|
||||
description: "Grouping tables into a hierarchy, and addressing them through it."
|
||||
icon: folder-tree
|
||||
keywords: ["namespace", "catalog", "hierarchy", "directory namespace", "rest namespace"]
|
||||
---
|
||||
|
||||
import {
|
||||
PyNamespaceTableOps,
|
||||
PyNamespaceAdminOps,
|
||||
TsNamespaceTableOps,
|
||||
TsNamespaceAdminOps,
|
||||
RsNamespaceAdminOps,
|
||||
RsNamespaceTableOps,
|
||||
} from '/snippets/connection.mdx';
|
||||
|
||||
Despite its name, LanceDB is not a "database" in the traditional sense -- it is a **Multimodal Lakehouse** that builds on the table abstraction,
|
||||
similar to many other lakehouse projects. LanceDB exposes a catalog-level abstraction over the Lance table format, via a *namespace spec*.
|
||||
If you're coming from traditional databases or lakehouses, you can think of a namespace as the catalog path that says where a table name lives.
|
||||
@@ -215,7 +224,7 @@ let db = lancedb::connect_namespace("rest", properties)
|
||||
</CodeGroup>
|
||||
[LanceDB Enterprise](/enterprise) operates a REST namespace server on top of the Lance format, so any REST client that can speak the REST namespace API
|
||||
contract can be used to interact with it. For authentication examples in LanceDB Enterprise, visit
|
||||
the [Namespaces in SDKs](/namespaces/usage#namespaces-in-lancedb-enterprise) page.
|
||||
the [Namespaces in SDKs](#namespaces-in-lancedb-enterprise) page.
|
||||
|
||||
## Best practices {#best-practices}
|
||||
|
||||
@@ -224,3 +233,127 @@ Below, we list some best practices for working with namespaces:
|
||||
- For remote storage locations, introduce explicit namespaces when multiple teams, environments, or domains share the same catalog.
|
||||
- Treat namespace paths as stable identifiers (for example `"prod/search"`, `"staging/recs"`).
|
||||
- For maintainability reasons, avoid hard-coding object-store table paths in application code -- instead, prefer catalog identifiers + namespaces.
|
||||
|
||||
|
||||
## Table operations with namespace paths {#table-operations-with-namespace-paths}
|
||||
|
||||
Let's imagine a scenario where your table management needs have evolved, and you now have the following multi-level structure to organize your tables outside the root namespace.
|
||||
```
|
||||
./local_lancedb (root)
|
||||
└── prod
|
||||
└── search
|
||||
└── user (table)
|
||||
└── data (table)
|
||||
└── recommendations
|
||||
└── user (table)
|
||||
└── data (data)
|
||||
```
|
||||
|
||||
Below, we show how you would express table operations within that namespace. Each item in the namespace
|
||||
list (`["prod", "search"]`) represents a level in the namespace hierarchy, and the table name is
|
||||
specified when you create, open, list, or drop it.
|
||||
|
||||
The SDK methods expose the namespace path in the idiom of each language:
|
||||
|
||||
- Python: pass `namespace_path=["prod", "search"]` to table operations.
|
||||
- Rust: call builder methods such as `.namespace(vec!["prod".to_string(), "search".to_string()])`.
|
||||
- TypeScript: pass a `namespacePath` array, for example `await db.openTable("user", ["prod", "search"])`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Using namespaces is **optional** in LanceDB, and most basic use cases do not require to work with them.
|
||||
An empty namespace (`[]`), which is the default, means "root namespace", and the data will be stored in
|
||||
the `data/` directory under the specified root path.
|
||||
</Note>
|
||||
|
||||
## Namespace management APIs {#namespace-management-apis}
|
||||
|
||||
You can open/create/drop tables inside a namespace path (like `["prod", "search"]`).
|
||||
All three SDKs expose namespace lifecycle operations directly.
|
||||
In Python, use `lancedb.connect_namespace(...)` when calling namespace lifecycle methods such as
|
||||
`create_namespace`, `list_namespaces`, `describe_namespace`, and `drop_namespace`.
|
||||
In TypeScript, use `lancedb.connectNamespace(...)` and call `createNamespace`, `listNamespaces`,
|
||||
`describeNamespace`, and `dropNamespace` on the returned `Connection`.
|
||||
In Rust, use `lancedb::connect_namespace(...)` and call `create_namespace`, `list_namespaces`,
|
||||
and `drop_namespace`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Namespace creation and deletion have modes that control what happens when the target already exists,
|
||||
doesn't exist, or contains data:
|
||||
|
||||
- Create mode: `create` fails if the namespace already exists, `exist_ok` keeps the existing namespace, and `overwrite` replaces it.
|
||||
- Drop mode: `fail` reports an error when the namespace doesn't exist, and `skip` treats a missing namespace as a successful no-op.
|
||||
- Drop behavior: `restrict` keeps non-empty namespaces from being dropped, and `cascade` drops child namespaces and tables first.
|
||||
|
||||
Namespace path components can't be empty. Each component can contain only letters, numbers,
|
||||
underscores, hyphens, and periods.
|
||||
|
||||
Listing APIs return the immediate children of the requested namespace path. Use `limit` with the
|
||||
returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you
|
||||
want to list from the root namespace.
|
||||
|
||||
## Namespaces in LanceDB Enterprise {#namespaces-in-lancedb-enterprise}
|
||||
|
||||
In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory.
|
||||
LanceDB Enterprise supports both directory-based (`ns_impl = "dir"`) and REST-based (`ns_impl = "rest"`) namespace implementations.
|
||||
The example below shows how to configure a directory-based namespace implementation in LanceDB Enterprise.
|
||||
```toml
|
||||
# Federated database configuration for DirectoryNamespace
|
||||
# This example uses minio storage
|
||||
[federated_dbs.federated_dir_test]
|
||||
ns_impl = "dir"
|
||||
root = "s3://<your-deployment>/<your-dir>"
|
||||
"storage.region" = "us-east-1"
|
||||
"storage.endpoint" = "http://localhost:9000"
|
||||
"storage.access_key_id" = "minioadmin"
|
||||
"storage.secret_access_key" = "minioadmin"
|
||||
"storage.allow_http" = "true"
|
||||
# Far future expiration (year 2100)
|
||||
"storage.expires_at_millis" = "4102444800000"
|
||||
```
|
||||
|
||||
The example above uses MinIO, but the same approach applies to other cloud object storage platforms based on your deployment.
|
||||
|
||||
For REST-based namespace servers, you can specify the namespace implementation as `"rest"` with forwarding prefixed headers
|
||||
for authentication and context propagation.
|
||||
|
||||
```toml
|
||||
[federated_dbs.federated_rest_test]
|
||||
ns_impl = "rest"
|
||||
uri = "http://<your_org>.internal.catalog.com"
|
||||
forward_header_prefixes = ["X-forward"]
|
||||
```
|
||||
|
||||
With `forward_header_prefixes = ["X-forward"]`, any incoming header starting with `X-forward` is forwarded to
|
||||
`http://<your_org>.internal.catalog.com`. This is useful for auth propagation, for example sending
|
||||
`X-forward-authorization: Bearer xxxx`.
|
||||
|
||||
For the LanceDB REST API itself, requests use `x-api-key` for API-key authentication. If your endpoint
|
||||
serves more than one database, LanceDB can also use headers such as `x-lancedb-database` or
|
||||
`x-lancedb-database-prefix` to route the request to the right database context.
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: "Tables"
|
||||
sidebarTitle: "Table"
|
||||
description: "The object most operations act on: creating, opening, renaming, cloning and dropping one."
|
||||
icon: table
|
||||
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
|
||||
---
|
||||
|
||||
A table is rows and columns, versioned. It is what searches read, what writes
|
||||
commit to, and what indexes and branches belong to.
|
||||
|
||||
This page covers the table as a catalog object. Getting data into one is
|
||||
[Ingest](/ingestion); changing its shape is
|
||||
[Schema evolution](/tables/schema).
|
||||
|
||||
## What a table carries {#carries}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Columns** | including vectors, blobs, and [computed](/feature-engineering/add-computed-column) columns |
|
||||
| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) |
|
||||
| **Indexes** | vector, full-text and scalar, built over its columns |
|
||||
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
|
||||
|
||||
## Creating {#create}
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl = db.create_table("documents", data)
|
||||
tbl = db.create_table("documents", schema=schema) # empty, typed
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
const tbl = await db.createTable("documents", data);
|
||||
const tbl = await db.createEmptyTable("documents", schema);
|
||||
```
|
||||
|
||||
```sql SQL
|
||||
CREATE TABLE documents (id BIGINT, title VARCHAR)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`data` can be a list of objects, an Arrow table or reader, a pandas or polars
|
||||
frame, or Pydantic models. [Ingesting data](/ingestion) covers each, plus
|
||||
`exist_ok` and `mode="overwrite"` for re-runnable scripts.
|
||||
|
||||
<Warning>
|
||||
An explicit schema is where nullability is decided, and where you opt into
|
||||
**stable row ids** -- which cannot be enabled later and are required for a
|
||||
[materialized view](/management/materialized-view) source. See
|
||||
[Limits](/basics/limits#schema).
|
||||
</Warning>
|
||||
|
||||
## Opening and listing {#open}
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
tbl = db.open_table("documents")
|
||||
db.table_names()
|
||||
db.table_exists("documents")
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
const tbl = await db.openTable("documents");
|
||||
await db.tableNames();
|
||||
```
|
||||
|
||||
```sql SQL
|
||||
SHOW TABLES
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Each accepts `namespace_path` to act inside a [namespace](/management/namespace)
|
||||
rather than at the database root. TypeScript has no `tableExists`.
|
||||
|
||||
## Renaming and cloning {#rename-clone}
|
||||
|
||||
```python
|
||||
db.rename_table("documents", "docs")
|
||||
|
||||
db.clone_table(
|
||||
"docs_snapshot",
|
||||
source_uri="s3://bucket/db/documents.lance",
|
||||
source_version=42, # or source_tag="release-1"
|
||||
is_shallow=True,
|
||||
)
|
||||
```
|
||||
|
||||
A **shallow** clone -- the default -- references the source's data files rather
|
||||
than copying them, so it is fast and cheap regardless of table size. It is the
|
||||
practical way to branch an experiment off a known version, or to pin a training
|
||||
set to the exact version a model saw.
|
||||
|
||||
<Warning>
|
||||
A shallow clone depends on the source's files continuing to exist. If the source
|
||||
is [compacted](/maintenance/compaction) past the retention window, the version
|
||||
the clone points at can be pruned. Use a deep clone (`is_shallow=False`) when
|
||||
the copy has to outlive the source's history.
|
||||
</Warning>
|
||||
|
||||
## Dropping {#drop}
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
db.drop_table("documents")
|
||||
job = db.drop_table_async("documents")
|
||||
db.drop_all_tables()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await db.dropTable("documents");
|
||||
const job = await db.dropTableAsync("documents");
|
||||
await db.dropAllTables();
|
||||
```
|
||||
|
||||
```sql SQL
|
||||
DROP TABLE documents
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Warning>
|
||||
Dropping removes the table and **every version of it**.
|
||||
[Versioning](/tables/versioning) does not survive a drop, and neither
|
||||
`drop_all_tables` nor `drop_database` asks for confirmation.
|
||||
</Warning>
|
||||
|
||||
On a large table the drop can take a while, so `drop_table_async` returns a
|
||||
[job](/basics/jobs) rather than blocking.
|
||||
|
||||
## The full surface {#surface}
|
||||
|
||||
| | Python | TypeScript | SQL |
|
||||
| --- | --- | --- | --- |
|
||||
| create | `create_table` | `createTable` | `CREATE TABLE` |
|
||||
| create empty | `create_table(schema=)` | `createEmptyTable` | `CREATE TABLE` |
|
||||
| open | `open_table` | `openTable` | -- |
|
||||
| list | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` |
|
||||
| exists | `table_exists` | -- | -- |
|
||||
| rename | `rename_table` | `renameTable` | -- |
|
||||
| clone | `clone_table` | `cloneTable` | -- |
|
||||
| drop | `drop_table`, `drop_table_async` | `dropTable`, `dropTableAsync` | `DROP TABLE` |
|
||||
| drop all | `drop_all_tables` | `dropAllTables` | -- |
|
||||
|
||||
## Where to go next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Ingesting data" icon="inbox" href="/ingestion">
|
||||
Creating a table from data, and writing to one that exists.
|
||||
</Card>
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/tables/schema">
|
||||
Adding, altering and dropping columns.
|
||||
</Card>
|
||||
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
|
||||
Every write commits a version; reads can go back.
|
||||
</Card>
|
||||
<Card title="Branching" icon="code-branch" href="/tables/branching">
|
||||
Isolated lines of change, merged or discarded.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -1,149 +0,0 @@
|
||||
---
|
||||
title: "Using Namespaces"
|
||||
sidebarTitle: "Using namespaces"
|
||||
description: "Use LanceDB's namespace-aware table and catalog APIs in Python, TypeScript, and Rust."
|
||||
icon: "folder-tree"
|
||||
keywords: ["namespace", "create_table", "open_table", "list_tables", "catalog"]
|
||||
---
|
||||
|
||||
import {
|
||||
PyNamespaceTableOps,
|
||||
PyNamespaceAdminOps,
|
||||
TsNamespaceTableOps,
|
||||
TsNamespaceAdminOps,
|
||||
RsNamespaceAdminOps,
|
||||
RsNamespaceTableOps,
|
||||
} from '/snippets/connection.mdx';
|
||||
|
||||
As your table organization needs grow over time and your projects become more complex, you can use namespaces to organize your tables in a way that reflects your business domains, teams, or environments.
|
||||
|
||||
As described in the [Namespaces and Catalog Model](/namespaces) section, namespaces are LanceDB's way of generalizing catalog specs, providing developers a clean way to manage hierarchical organization of tables in the catalog. The SDKs treat a namespace as a path and can use it for table resolution when you use LanceDB outside the root namespace.
|
||||
|
||||
## Table operations with namespace paths {#table-operations-with-namespace-paths}
|
||||
|
||||
Let's imagine a scenario where your table management needs have evolved, and you now have the following multi-level structure to organize your tables outside the root namespace.
|
||||
```
|
||||
./local_lancedb (root)
|
||||
└── prod
|
||||
└── search
|
||||
└── user (table)
|
||||
└── data (table)
|
||||
└── recommendations
|
||||
└── user (table)
|
||||
└── data (data)
|
||||
```
|
||||
|
||||
Below, we show how you would express table operations within that namespace. Each item in the namespace
|
||||
list (`["prod", "search"]`) represents a level in the namespace hierarchy, and the table name is
|
||||
specified when you create, open, list, or drop it.
|
||||
|
||||
The SDK methods expose the namespace path in the idiom of each language:
|
||||
|
||||
- Python: pass `namespace_path=["prod", "search"]` to table operations.
|
||||
- Rust: call builder methods such as `.namespace(vec!["prod".to_string(), "search".to_string()])`.
|
||||
- TypeScript: pass a `namespacePath` array, for example `await db.openTable("user", ["prod", "search"])`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsNamespaceTableOps}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
Using namespaces is **optional** in LanceDB, and most basic use cases do not require to work with them.
|
||||
An empty namespace (`[]`), which is the default, means "root namespace", and the data will be stored in
|
||||
the `data/` directory under the specified root path.
|
||||
</Note>
|
||||
|
||||
## Namespace management APIs {#namespace-management-apis}
|
||||
|
||||
You can open/create/drop tables inside a namespace path (like `["prod", "search"]`).
|
||||
All three SDKs expose namespace lifecycle operations directly.
|
||||
In Python, use `lancedb.connect_namespace(...)` when calling namespace lifecycle methods such as
|
||||
`create_namespace`, `list_namespaces`, `describe_namespace`, and `drop_namespace`.
|
||||
In TypeScript, use `lancedb.connectNamespace(...)` and call `createNamespace`, `listNamespaces`,
|
||||
`describeNamespace`, and `dropNamespace` on the returned `Connection`.
|
||||
In Rust, use `lancedb::connect_namespace(...)` and call `create_namespace`, `list_namespaces`,
|
||||
and `drop_namespace`.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PyNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsNamespaceAdminOps}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Namespace creation and deletion have modes that control what happens when the target already exists,
|
||||
doesn't exist, or contains data:
|
||||
|
||||
- Create mode: `create` fails if the namespace already exists, `exist_ok` keeps the existing namespace, and `overwrite` replaces it.
|
||||
- Drop mode: `fail` reports an error when the namespace doesn't exist, and `skip` treats a missing namespace as a successful no-op.
|
||||
- Drop behavior: `restrict` keeps non-empty namespaces from being dropped, and `cascade` drops child namespaces and tables first.
|
||||
|
||||
Namespace path components can't be empty. Each component can contain only letters, numbers,
|
||||
underscores, hyphens, and periods.
|
||||
|
||||
Listing APIs return the immediate children of the requested namespace path. Use `limit` with the
|
||||
returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you
|
||||
want to list from the root namespace.
|
||||
|
||||
## Namespaces in LanceDB Enterprise {#namespaces-in-lancedb-enterprise}
|
||||
|
||||
In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory.
|
||||
LanceDB Enterprise supports both directory-based (`ns_impl = "dir"`) and REST-based (`ns_impl = "rest"`) namespace implementations.
|
||||
The example below shows how to configure a directory-based namespace implementation in LanceDB Enterprise.
|
||||
```toml
|
||||
# Federated database configuration for DirectoryNamespace
|
||||
# This example uses minio storage
|
||||
[federated_dbs.federated_dir_test]
|
||||
ns_impl = "dir"
|
||||
root = "s3://<your-deployment>/<your-dir>"
|
||||
"storage.region" = "us-east-1"
|
||||
"storage.endpoint" = "http://localhost:9000"
|
||||
"storage.access_key_id" = "minioadmin"
|
||||
"storage.secret_access_key" = "minioadmin"
|
||||
"storage.allow_http" = "true"
|
||||
# Far future expiration (year 2100)
|
||||
"storage.expires_at_millis" = "4102444800000"
|
||||
```
|
||||
|
||||
The example above uses MinIO, but the same approach applies to other cloud object storage platforms based on your deployment.
|
||||
|
||||
For REST-based namespace servers, you can specify the namespace implementation as `"rest"` with forwarding prefixed headers
|
||||
for authentication and context propagation.
|
||||
|
||||
```toml
|
||||
[federated_dbs.federated_rest_test]
|
||||
ns_impl = "rest"
|
||||
uri = "http://<your_org>.internal.catalog.com"
|
||||
forward_header_prefixes = ["X-forward"]
|
||||
```
|
||||
|
||||
With `forward_header_prefixes = ["X-forward"]`, any incoming header starting with `X-forward` is forwarded to
|
||||
`http://<your_org>.internal.catalog.com`. This is useful for auth propagation, for example sending
|
||||
`X-forward-authorization: Bearer xxxx`.
|
||||
|
||||
For the LanceDB REST API itself, requests use `x-api-key` for API-key authentication. If your endpoint
|
||||
serves more than one database, LanceDB can also use headers such as `x-lancedb-database` or
|
||||
`x-lancedb-database-prefix` to route the request to the right database context.
|
||||
|
||||
## Related references {#related-references}
|
||||
|
||||
- [Client SDK API references](/basics/apis#reference)
|
||||
- [REST API Reference](/api-reference/rest)
|
||||
- [Namespaces and the Catalog Model](/namespaces)
|
||||
@@ -514,7 +514,7 @@ full-text and hybrid search, reranking, and server-side feature engineering.
|
||||
<Card
|
||||
title="Basic table operations"
|
||||
icon="table"
|
||||
href="/tables"
|
||||
href="/management/table"
|
||||
>
|
||||
Build on this quickstart with table creation, updates, and schema tips.
|
||||
</Card>
|
||||
|
||||
@@ -20,7 +20,7 @@ DROP TABLE documents
|
||||
```
|
||||
|
||||
Column types are Arrow types under SQL names -- see
|
||||
[Data types](/sql/data-types). A table in a [namespace](/namespaces) is written
|
||||
[Data types](/sql/data-types). A table in a [namespace](/management/namespace) is written
|
||||
as a dotted path: `prod.events.clicks`.
|
||||
|
||||
## Views {#views}
|
||||
|
||||
@@ -32,7 +32,7 @@ avoids repeating it in the select list.
|
||||
|
||||
## Across namespaces {#namespaces}
|
||||
|
||||
Tables in different [namespaces](/namespaces) join by their dotted paths -- the
|
||||
Tables in different [namespaces](/management/namespace) join by their dotted paths -- the
|
||||
catalog is one catalog, so there is nothing to federate:
|
||||
|
||||
```sql
|
||||
|
||||
@@ -117,7 +117,7 @@ WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
|
||||
## Namespaced tables {#namespaces}
|
||||
|
||||
A table inside a [namespace](/namespaces) is written as a dotted path:
|
||||
A table inside a [namespace](/management/namespace) is written as a dotted path:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks
|
||||
|
||||
@@ -1,672 +0,0 @@
|
||||
---
|
||||
title: "Basic Table Operations"
|
||||
sidebarTitle: "Basic usage"
|
||||
description: "Create tables, search vectors, and append data in LanceDB."
|
||||
icon: "table"
|
||||
keywords: ["create table", "polars", "pandas", "pyarrow", "dataframe", "nested data"]
|
||||
---
|
||||
|
||||
import { PyConnect, PyConnectEnterpriseQuickstart, TsConnect, TsConnectEnterpriseQuickstart, RsConnect, RsConnectEnterpriseQuickstart } from '/snippets/connection.mdx';
|
||||
import {
|
||||
PyBasicImports,
|
||||
PyDataLoad,
|
||||
PyBasicCreateTable,
|
||||
PyBasicOpenTable,
|
||||
PyBasicCreateEmptyTable,
|
||||
PyBasicCreateTablePandas,
|
||||
PyBasicCreateTablePolars,
|
||||
PyBasicAsyncApi,
|
||||
PyBasicVectorSearch,
|
||||
PyBasicVectorSearchQ1,
|
||||
PyBasicVectorSearchQ2,
|
||||
PyBasicVectorSearchQ3,
|
||||
PyBasicVectorSearchQ4,
|
||||
PyBasicSortPolars,
|
||||
PyBasicDeleteRows,
|
||||
PyBasicAddData,
|
||||
PyBasicAddColumns,
|
||||
PyBasicDropColumns,
|
||||
PyBasicDropTable,
|
||||
TsBasicImports,
|
||||
TsDataLoad,
|
||||
TsBasicCreateTable,
|
||||
TsBasicOpenTable,
|
||||
TsBasicCreateEmptyTable,
|
||||
TsBasicVectorSearchQ1,
|
||||
TsBasicVectorSearchQ2,
|
||||
TsBasicVectorSearchQ3,
|
||||
TsBasicVectorSearchQ4,
|
||||
TsBasicDeleteRows,
|
||||
TsBasicAddData,
|
||||
TsBasicAddColumns,
|
||||
TsBasicDropColumns,
|
||||
TsBasicDropTable,
|
||||
RsBasicImports,
|
||||
RsDataLoad,
|
||||
RsBasicCreateTable,
|
||||
RsBasicOpenTable,
|
||||
RsBasicCreateEmptyTable,
|
||||
RsBasicAddData,
|
||||
RsBasicVectorSearchQ1,
|
||||
RsBasicVectorSearchQ2,
|
||||
RsBasicVectorSearchQ3,
|
||||
RsBasicVectorSearchQ4,
|
||||
RsBasicDeleteRows,
|
||||
RsBasicAddColumns,
|
||||
RsBasicDropColumns,
|
||||
RsBasicDropTable,
|
||||
} from '/snippets/basic_usage.mdx';
|
||||
|
||||
Now that you've completed the [LanceDB quickstart](/quickstart), you're ready to
|
||||
explore some more table operations you'll typically need when working with LanceDB.
|
||||
|
||||
- **Ingest data into tables** from JSON data (and in Python, Pandas or Polars DataFrames)
|
||||
- **Create empty tables** by defining explicit Arrow schemas
|
||||
- **Vector similarity search** with filtering and projections
|
||||
- **Filtered queries** that can operate on nested structs
|
||||
- **Query Lance tables in DuckDB** via the Lance extension for SQL analytics (including joins)
|
||||
|
||||
<Note title="Python async users">
|
||||
This page uses **synchronous** Python snippets. If your Python app uses `asyncio`,
|
||||
the same flow works with `connect_async(...)` and `await`-based table/query calls.
|
||||
Use the example below as a template, and see [Quickstart](/quickstart#python-sync-and-async-apis)
|
||||
for example snippets on both sync and async Python usage.
|
||||
</Note>
|
||||
|
||||
## Dataset {#dataset}
|
||||
|
||||
We'll work with this small dataset based on characters from the legends of Camelot. Note that
|
||||
the `vector` column holds 4-dimensional embeddings, and the `stats` column is a nested struct
|
||||
with several integer fields, indicating each character's attributes.
|
||||
|
||||
```json camelot.json icon="brackets-curly" expandable=true
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"name": "King Arthur",
|
||||
"role": "King of Camelot",
|
||||
"description": "The legendary ruler of Camelot, wielder of Excalibur, and leader of the Knights of the Round Table.",
|
||||
"vector": [0.72, -0.28, 0.60, 0.86],
|
||||
"stats": { "strength": 2, "courage": 5, "magic": 1, "wisdom": 4 }
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"name": "Merlin",
|
||||
"role": "Wizard and Advisor",
|
||||
"description": "A powerful wizard and prophet who mentors Arthur and shapes the destiny of Camelot through magic and foresight.",
|
||||
"vector": [0.05, 0.88, 0.62, 0.85],
|
||||
"stats": { "strength": 2, "courage": 4, "magic": 5, "wisdom": 5 }
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"name": "Queen Guinevere",
|
||||
"role": "Queen of Camelot",
|
||||
"description": "Arthur's queen, admired for her grace and diplomacy, whose romances and loyalties influence Camelot's fate.",
|
||||
"vector": [0.22, -0.22, 0.42, 0.82],
|
||||
"stats": { "strength": 1, "courage": 3, "magic": 1, "wisdom": 4 }
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"name": "Sir Lancelot",
|
||||
"role": "Knight of the Round Table",
|
||||
"description": "Arthur's most skilled knight, famed for unmatched combat prowess and his tragic love for Queen Guinevere.",
|
||||
"vector": [0.86, -0.35, 0.38, 0.55],
|
||||
"stats": { "strength": 5, "courage": 5, "magic": 1, "wisdom": 3 }
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"name": "Sir Gawain",
|
||||
"role": "Knight of the Round Table",
|
||||
"description": "A noble and honorable knight known for his courtesy and his encounter with the Green Knight.",
|
||||
"vector": [0.82, -0.32, 0.52, 0.60],
|
||||
"stats": { "strength": 4, "courage": 5, "magic": 1, "wisdom": 4 }
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"name": "Sir Galahad",
|
||||
"role": "Knight of the Round Table",
|
||||
"description": "The purest and most virtuous knight, chosen to achieve the Holy Grail due to his unwavering spiritual purity.",
|
||||
"vector": [0.80, -0.20, 0.70, 0.78],
|
||||
"stats": { "strength": 4, "courage": 5, "magic": 2, "wisdom": 5 }
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"name": "Sir Percival",
|
||||
"role": "Knight of the Round Table",
|
||||
"description": "A loyal and innocent knight whose bravery and sincerity make him one of the key seekers of the Holy Grail.",
|
||||
"vector": [0.78, -0.36, 0.48, 0.52],
|
||||
"stats": { "strength": 4, "courage": 4, "magic": 1, "wisdom": 3 }
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"name": "Mordred",
|
||||
"role": "Traitor Knight",
|
||||
"description": "Arthur's treacherous son or nephew who ultimately rebels against him, leading to Camelot's downfall.",
|
||||
"vector": [0.68, -0.30, -0.65, 0.20],
|
||||
"stats": { "strength": 4, "courage": 2, "magic": 1, "wisdom": 2 }
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
<Note>
|
||||
The `vector` arrays here are synthetic and for demonstration purposes only. In your real-world
|
||||
applications, you'd generate these vectors from the raw text fields using a suitable embedding model.
|
||||
</Note>
|
||||
|
||||
## Connect to a database {#connect-to-a-database}
|
||||
|
||||
### Option 1: Direct table access {#option-1-direct-table-access}
|
||||
|
||||
We start by connecting to a LanceDB database path. The example below uses a local path in LanceDB OSS.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsConnect}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsConnect}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
You can also connect LanceDB OSS directly to object storage. For credentials, endpoints, and provider-specific options, see
|
||||
[Configuring storage](/storage/configuration).
|
||||
|
||||
### Option 2: Remote tables {#option-2-remote-tables}
|
||||
|
||||
If you're using LanceDB [Enterprise](/enterprise), you can connect using a `db://` URI,
|
||||
along with any necessary credentials. Simply replace the local path with a remote `uri`
|
||||
that points to where your data is stored, and you're ready to go.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{ "import lancedb\n\n" }
|
||||
{PyConnectEnterpriseQuickstart}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
|
||||
{TsConnectEnterpriseQuickstart}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{ "use lancedb::connect;\n\n" }
|
||||
{RsConnectEnterpriseQuickstart}
|
||||
</CodeBlock>
|
||||
</CodeGroup >
|
||||
|
||||
<Note>
|
||||
- When you connect to a remote URI (Enterprise), `open_table(...)` returns a *remote* table.
|
||||
Remote tables support core operations (ingest, search, update, delete), but some convenience
|
||||
methods for bulk data export are not available.
|
||||
- In the Python SDK, `table.to_arrow()` and `table.to_pandas()` are not implemented for remote tables.
|
||||
To retrieve data, use search queries instead: `table.search(query).limit(n).to_arrow()`.
|
||||
</Note>
|
||||
|
||||
## Create a table and ingest data {#create-a-table-and-ingest-data}
|
||||
|
||||
### From JSON {#from-json}
|
||||
LanceDB stores records in Lance tables. Each row is a record and each column
|
||||
holds a field or related metadata. The simplest way to start is to obtain the source
|
||||
data as a list of JSON records that includes a vector column and any metadata
|
||||
fields you care about.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicImports}
|
||||
</CodeBlock>
|
||||
</CodeGroup >
|
||||
|
||||
Load the data from the JSON file:
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyDataLoad}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDataLoad}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDataLoad}
|
||||
</CodeBlock>
|
||||
</CodeGroup >
|
||||
|
||||
You can now create a LanceDB table from the loaded data. By default, creating a table with a name
|
||||
that already exists raises an error. Use `mode="overwrite"` only when you intentionally want to
|
||||
replace the existing table and its data, or use `exist_ok` / `existOk` when repeatable setup should
|
||||
reuse the existing table instead of writing the supplied rows again.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicCreateTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicCreateTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicCreateTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup >
|
||||
|
||||
<Info>
|
||||
If you want to avoid overwriting an existing table, omit the overwrite mode. For append-only
|
||||
ingestion into a table that already exists, open the table and call `add(...)` instead of
|
||||
`create_table(...)`. For repeatable setup with `exist_ok` / `existOk`, see
|
||||
[Handle existing tables](/ingestion#handle-existing-tables).
|
||||
</Info>
|
||||
|
||||
For more ingestion patterns, including PyArrow tables, Python `pyarrow.dataset.Dataset` inputs,
|
||||
empty tables, and Python `LanceModel` schemas with nested fields, see
|
||||
[Ingesting data](/ingestion).
|
||||
|
||||
### From Pandas DataFrames {#from-pandas-dataframes}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
You can create LanceDB tables directly from [Pandas](https://pandas.pydata.org/) DataFrames. Simply
|
||||
obtain the source data as a Pandas DataFrame, then create the table
|
||||
and directly ingest to it.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicCreateTablePandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### From Polars DataFrames {#from-polars-dataframes}
|
||||
<Badge color="green">Python Only</Badge>
|
||||
|
||||
You can also create LanceDB tables directly from [Polars](https://www.pola.rs/) DataFrames. Simply
|
||||
obtain the source data as a Polars DataFrame, then create the table
|
||||
and directly ingest to it.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicCreateTablePolars}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### From an Arrow schema {#from-an-arrow-schema}
|
||||
|
||||
If you want to create an _empty_ table without any data -- say you want to
|
||||
define the schema first and then incrementally add data later -- you can
|
||||
do so by defining an Arrow schema explicitly.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicCreateEmptyTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Once the empty table is defined, LanceDB is ready to accept new data via
|
||||
the `add` method, as shown in the next section.
|
||||
|
||||
<Card title="Display table schema" icon="book">
|
||||
LanceDB tables are type-aware, leveraging Apache Arrow under the hood.
|
||||
You can display a given table's schema using the `schema` property or
|
||||
method. For example, in Python, running `print(table.schema)` would show
|
||||
something like the following:
|
||||
|
||||
```txt expandable=true
|
||||
id: int64
|
||||
name: string
|
||||
role: string
|
||||
description: string
|
||||
vector: fixed_size_list<item: float>[4]
|
||||
child 0, item: float
|
||||
stats: struct<courage: int64, magic: int64, strength: int64, wisdom: int64>
|
||||
child 0, courage: int64
|
||||
child 1, magic: int64
|
||||
child 2, strength: int64
|
||||
child 3, wisdom: int64
|
||||
```
|
||||
</Card>
|
||||
|
||||
## Append data to a table {#append-data-to-a-table}
|
||||
|
||||
LanceDB tables are mutable, and you can append new records to existing tables.
|
||||
If you're starting with a fresh session, connect to the database and open the
|
||||
existing table named `camelot`.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicOpenTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicOpenTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicOpenTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Prepare the new records to add. Here, we add two new magical characters
|
||||
via the `add` method. For the Rust snippet, you can find the helper functions in the
|
||||
[code](https://github.com/lancedb/docs/blob/main/tests/rs/basic_usage.rs).
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicAddData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicAddData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
We now have two new records in the table. Let's begin to query our data!
|
||||
|
||||
## Vector search {#vector-search}
|
||||
|
||||
It's straightforward to run vector similarity search in LanceDB. Let's answer
|
||||
some questions about the data using vector search with projections (returning only
|
||||
the desired columns).
|
||||
|
||||
> Q1: _Who are the characters similar to "wizard"?_
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicVectorSearchQ1}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicVectorSearchQ1}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicVectorSearchQ1}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
| name | role | description |
|
||||
| --- | --- | --- |
|
||||
| Merlin | Wizard and Advisor | A powerful wizard and prophet |
|
||||
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… |
|
||||
| Morgan le Fay | Sorceress | A powerful enchantress, Arthur… |
|
||||
| Queen Guinevere | Queen of Camelot | Arthur's queen, admired for he… |
|
||||
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… |
|
||||
|
||||
We have Merlin, The Lady of the Lake, and Morgan le Fay in the top results, who
|
||||
all have magical abilities.
|
||||
|
||||
Next, let's try to answer a different question that involves vector search while
|
||||
filtering on a nested struct field. Filtering is done using the `where` method,
|
||||
into which you can pass SQL-like expressions.
|
||||
|
||||
> Q2: _Who are the characters similar to "wizard" with high magic stats?_
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python" icon="Python">
|
||||
{PyBasicVectorSearchQ2}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicVectorSearchQ2}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicVectorSearchQ2}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
| name | role | description |
|
||||
| --- | --- | --- |
|
||||
| Merlin | Wizard and Advisor | A powerful wizard and prophet |
|
||||
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… |
|
||||
| Morgan le Fay | Sorceress | A powerful enchantress, Arthur… |
|
||||
|
||||
Only three characters have magical abilities greater than 3. Merlin is
|
||||
clearly the most magical of them all!
|
||||
|
||||
## Filtered search {#filtered-search}
|
||||
|
||||
You can also run traditional analytics-style search queries that do not
|
||||
involve vectors. For example, let's find the strongest characters in
|
||||
the dataset. In the query below, we leave the `search` method empty to indicate
|
||||
that we don't want to use any vector for similarity search (in TypeScript/Rust,
|
||||
use `query()` instead), and use the `where` method to filter on the `strength` field.
|
||||
|
||||
> Q3: _Who are the strongest characters?_
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicVectorSearchQ3}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicVectorSearchQ3}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicVectorSearchQ3}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
| name | role | description |
|
||||
| --- | --- | --- |
|
||||
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… |
|
||||
| Sir Gawain | Knight of the Round Table | A noble and honorable knight k… |
|
||||
| Sir Percival | Knight of the Round Table | A loyal and innocent knight wh… |
|
||||
| Sir Lancelot | Knight of the Round Table | Arthur's most skilled knight, … |
|
||||
| Mordred | Traitor Knight | Arthur's treacherous son or ne… |
|
||||
|
||||
Clearly, the strongest characters are all Knights of the Round Table!
|
||||
|
||||
<Note>
|
||||
Need SQL analytics like filters, aggregations, or joins on Lance tables? Use the DuckDB
|
||||
Lance extension to query Lance tables directly with SQL. See the
|
||||
[DuckDB integration guide](/integrations/data/duckdb).
|
||||
</Note>
|
||||
|
||||
## Add column {#add-column}
|
||||
|
||||
We can also add new columns to an existing LanceDB table using the `add_columns` method.
|
||||
For this example, let's add a new float column named `power` that shows the average
|
||||
of each character's strength, courage, magic, and wisdom stats.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicAddColumns}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicAddColumns}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicAddColumns}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
The example above sums up the individual stats and divides by 4 to compute the average.
|
||||
The resulting average total stats is cast to an Arrow float type under the hood for the
|
||||
Lance table.
|
||||
|
||||
We can display the results of this column in descending order of power.
|
||||
|
||||
> Q4: _Who are the most powerful characters?_
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicVectorSearchQ4}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicVectorSearchQ4}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicVectorSearchQ4}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Note that LanceDB's `where` only filters rows, but doesn't sort them by applying an `ORDER BY`
|
||||
clause that you may be used to when working with SQL databases.
|
||||
|
||||
You can also sort the results after converting them to a Polars DataFrame.
|
||||
In TypeScript/Rust, you can sort the
|
||||
returned array in application code.
|
||||
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicSortPolars}
|
||||
</CodeBlock>
|
||||
|
||||
| name | role | description | power |
|
||||
| --- | --- | --- | --- |
|
||||
| Merlin | Wizard and Advisor | A powerful wizard and prophet … | 4.0 |
|
||||
| Sir Galahad | Knight of the Round Table | The purest and most virtuous k… | 4.0 |
|
||||
| The Lady of the Lake | Mystical Guardian | A mysterious supernatural figu… | 3.75 |
|
||||
| Sir Lancelot | Knight of the Round Table | Arthur's most skilled knight, … | 3.5 |
|
||||
| Sir Gawain | Knight of the Round Table | A noble and honorable knight k… | 3.5 |
|
||||
|
||||
Merlin and Sir Galahad are the most powerful characters when considering the average of
|
||||
all their abilities! Sir Lancelot and the Lady of the Lake follow closely behind.
|
||||
|
||||
For column renames, type changes, nullability changes, and grouping multiple schema changes into
|
||||
one operation, see [Schema and data evolution](/tables/schema/).
|
||||
|
||||
## Delete data {#delete-data}
|
||||
|
||||
You can delete rows from a LanceDB table using the `delete` method with
|
||||
a filtering expression.
|
||||
|
||||
Say we want to remove Mordred, the traitor knight, from our table.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicDeleteRows}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicDeleteRows}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicDeleteRows}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This will delete the row(s) where the `role` value matches "Traitor Knight".
|
||||
You can verify that the row has been deleted by running a search query again,
|
||||
and confirming that Mordred no longer appears in the results.
|
||||
|
||||
## Drop column {#drop-column}
|
||||
|
||||
If you want to remove or delete a column from an existing LanceDB table, you can use
|
||||
the `drop_columns` method.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicDropColumns}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicDropColumns}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicDropColumns}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This will remove the `power` column we added earlier from the table schema.
|
||||
|
||||
## Drop table {#drop-table}
|
||||
|
||||
If you want to delete an entire table from the database, you can use the
|
||||
`drop_table` method.
|
||||
|
||||
<CodeGroup >
|
||||
<CodeBlock filename="Python" language="Python">
|
||||
{PyBasicDropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBasicDropTable}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBasicDropTable}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
This will delete the `camelot` table from the connected LanceDB database.
|
||||
|
||||
<Info>
|
||||
See the full code for these examples (including helper functions) in the
|
||||
`basic_usage` file for the appropriate client language in the
|
||||
[docs repo](https://github.com/lancedb/docs/tree/main/tests).
|
||||
</Info>
|
||||
|
||||
## What about vector indexes? {#what-about-vector-indexes}
|
||||
|
||||
LanceDB supports vector indexes to speed up similarity search on large datasets.
|
||||
For datasets up to a few hundred thousand vectors, LanceDB's highly efficient kNN
|
||||
(brute-force) retrieval of nearest neighbors is often sufficient. As your dataset
|
||||
grows larger, you can create vector indexes on your vector columns to accelerate
|
||||
search. See the [indexing](/indexing/) documentation for details on how to create and use
|
||||
vector indexes in LanceDB.
|
||||
|
||||
## Next steps {#next-steps}
|
||||
Now that you've learned the basics of creating tables, adding data, running
|
||||
vector search, and modifying table schemas, you're ready to explore more
|
||||
advanced features of LanceDB. Below are some suggested next pages.
|
||||
|
||||
<Columns cols={2}>
|
||||
<Card
|
||||
title="Ingesting data"
|
||||
icon="cookie"
|
||||
href="/ingestion"
|
||||
>
|
||||
Learn the different approaches to creating and ingesting data into LanceDB tables from various sources.
|
||||
</Card>
|
||||
<Card
|
||||
title="Ingest and modify data"
|
||||
icon="clone"
|
||||
href="/ingestion"
|
||||
>
|
||||
Insert, merge, update and delete rows in an existing table.
|
||||
</Card>
|
||||
<Card
|
||||
title="Schema & data evolution"
|
||||
icon="boxes-stacked"
|
||||
href="/tables/schema/"
|
||||
>
|
||||
Understand how to evolve your table schemas and data over time with LanceDB.
|
||||
</Card>
|
||||
<Card
|
||||
title="Versioning and time travel"
|
||||
icon="clock"
|
||||
href="/tables/versioning/"
|
||||
>
|
||||
Explore LanceDB's built-in table versioning and time travel capabilities.
|
||||
</Card>
|
||||
</Columns>
|
||||
Reference in New Issue
Block a user