rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: reorder Basics and give Management a database page

Browse Source 
Basics now reads outward from the format: Lance Format, Lakehouse & Catalog,
Query & Job, SDKs & SQL, Agentic Skills.

Jobs becomes Query & Job. Async means two unrelated things here -- whether your
thread blocks, and whether the work outlives the call -- and nothing said so.

The API Reference tab is gone. Its SDK index is folded into SDKs & SQL, where
the surfaces are already compared; the generated REST pages move into Basics
beside it.

Connecting was described in passing on the catalog page. It is now a Management
page covering the URI forms and what each changes, deployment options, storage
options, and read consistency, since which one you pass decides what exists.

Quickstart ran out at multimodal storage. It now continues through a computed
column and its refresh into a PyTorch loader, which is the arc the rest of the
docs assume.
This commit is contained in:
Jack Ye
2026-09-09 23:01:02 -07:00
parent 4015560b83
commit c3f428acae
9 changed files with 497 additions and 221 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/api-reference/index.mdx
-48
View File
@@ -1,48 +0,0 @@
---
title: SDKs and REST API Reference
sidebarTitle: "SDKs"
description: "SDK and REST API reference for LanceDB Enterprise and OSS."
---
For detailed information of the available functions and methods in your preferred language's SDKs,
refer to the API documentation linked below.
If you're looking for a REST API reference, visit the [REST API](/api-reference/rest) page.
If you're looking for conceptual and practical namespace guidance before diving into method signatures, see
[Namespaces and Catalog Model](/namespaces) and [Using Namespaces in SDKs](/namespaces/usage).
## Supported SDKs {#supported-sdks}
Python, Typescript and Rust SDKs are officially supported by LanceDB. You can use these SDKs to interact with both LanceDB OSS and Enterprise deployments.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="python" /> [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration |
| <Icon icon="square-js" /> [Typescript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs`
| <Icon icon="rust" /> [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance |
## REST API SDKs {#rest-api-sdks}
<Badge color="red">Enterprise</Badge>
REST API-based SDKs provide a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="java" /> [Java](https://lancedb.github.io/lancedb/java/java/)| REST API Enterprise SDK in Java |
## Community-driven SDKs {#community-driven-sdks}
In addition to the officially supported SDKs, the LanceDB community may contribute SDKs in other languages.
These SDKs may not have the same level of support or feature parity as the official ones supported by LanceDB, but they can be an option
for users working in languages other than those listed above.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="golang" /> [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK for LanceDB |
| <Icon icon="gem" /> [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings for LanceDB |
| <Icon icon="swift" /> [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK for LanceDB |
| <Icon icon="R" /> [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package for LanceDB |
| <Icon icon="flutter" /> [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings for LanceDB |
docs/web/basics/apis.mdx
+73 -28
View File
@@ -1,13 +1,13 @@
---
title: "Python, TypeScript, and SQL"
sidebarTitle: "Python vs SQL"
description: "Three ways to reach the same data, what each one can do, and where each one runs."
title: "SDKs and SQL"
sidebarTitle: "SDKs & SQL"
description: "The ways to reach LanceDB — the language SDKs, the SQL endpoint, the REST API — and what each can do."
icon: code-compare
keywords: ["python", "typescript", "sql", "api", "flight sql", "client"]
keywords: ["python", "typescript", "rust", "java", "sql", "rest", "api reference", "sdk"]
---
The same table can be reached three ways: the Python SDK, the TypeScript SDK,
or SQL. They are views onto one engine, not three products -- a table written
The same table can be reached several ways: a language SDK, SQL, or the REST
API. They are views onto one engine, not separate products -- a table written
from Python is queried from SQL without an export step.
They are not interchangeable, though, and the differences are worth knowing
@@ -18,13 +18,14 @@ before you pick one.
| | Python | TypeScript | SQL |
| --- | :---: | :---: | :---: |
| Create and manage catalog objects | yes | yes | partly |
| Insert, merge, update, delete | yes | yes | -- |
| Insert, merge, update, delete | yes | yes | yes |
| Filter, project, aggregate, join | yes | yes | yes |
| Vector and full-text search | yes | yes | yes |
| Feature engineering, jobs | yes | yes | -- |
| Vector search | yes | yes | -- |
| Full-text search | yes | yes | yes |
| Feature engineering, jobs | yes | partly | yes |
| Runs against a local database | yes | yes | **no** |
Two entries in that table do most of the work of explaining the rest.
Three entries in that table do most of the work of explaining the rest.
## SQL needs a deployment {#sql-is-remote}
@@ -39,18 +40,31 @@ db = lancedb.connect("./.lancedb") # no SQL
db = lancedb.connect("db://my-database") # SQL
```
This is the single largest behavioural difference between the embedded and
deployed forms, and the reason the SDKs are the portable choice: SDK code moves
between the two by changing the URI, SQL does not.
This is the largest behavioural difference between the embedded and deployed
forms, and the reason the SDKs are the portable choice: SDK code moves between
the two by changing the URI, SQL does not.
<Badge color="red">Enterprise</Badge> Queries are served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client can reach it and results arrive as Arrow
record batches -- no conversion to Python objects unless you ask for one.
record batches.
## SQL from each language {#sql-clients}
## Vector search has no SQL form {#no-vector-sql}
From Python, the connection runs SQL directly:
Similarity search is a query-builder operation. A query that needs both vector
similarity and SQL analytics runs the search in an SDK and the analytics over
its result. Full-text search *is* available in SQL, as the
[`fts()`](/sql/functions#fts) table function and the
[`contains_tokens`](/sql/functions#contains-tokens) predicate.
## TypeScript has no SQL method {#ts-no-sql}
The TypeScript SDK does not expose SQL, and has no methods for
[functions](/feature-engineering/functions). Reach the endpoint with a Flight
SQL client instead -- see [connecting a client](/search/sql). This is a gap in
the SDK rather than in the endpoint: the same statements work.
## Running SQL from Python {#sql-from-python}
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
@@ -58,20 +72,14 @@ table = reader.read_all()
```
`execute_query` returns a blocking Arrow reader once results start arriving; it
does not wait for the whole query. `execute_query_async` returns a
[job](/basics/jobs) instead, for a statement you would rather not block on.
<Note>
The TypeScript SDK has no SQL method. Reach the endpoint with a Flight SQL
client instead -- see [connecting a client](/search/sql). This is a gap in the
SDK, not in the endpoint: the same statements work.
</Note>
does not wait for the whole query. `execute_query_async` returns a handle
instead -- see [Query and job](/basics/jobs#queries).
## Choosing between them {#choosing}
Reach for **the SDKs** when you are writing data, managing the catalog, running
feature engineering, or building an application that should work the same
embedded and deployed. They are the complete surface.
vector search, or building an application that should work the same embedded and
deployed.
Reach for **SQL** when the question is analytical -- joins across tables,
aggregates, window functions -- or when the thing asking is a BI tool, a
@@ -81,6 +89,43 @@ create a [view](/management/view).
The two compose. A common shape is SDK code to build and maintain tables, and
SQL for the analysis on top.
## Reference {#reference}
### Officially supported SDKs {#supported-sdks}
Python, TypeScript and Rust are supported against both embedded and deployed
LanceDB.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="python" /> [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration |
| <Icon icon="square-js" /> [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` |
| <Icon icon="rust" /> [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance |
### REST API {#rest}
<Badge color="red">Enterprise</Badge> A deployment implements the
[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable
over HTTP by anything that speaks it -- see the
[REST API reference](/api-reference/rest).
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="java" /> [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java |
### Community SDKs {#community-sdks}
Contributed by the community rather than supported by LanceDB, so feature parity
and release cadence vary.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="golang" /> [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK |
| <Icon icon="gem" /> [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings |
| <Icon icon="swift" /> [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK |
| <Icon icon="R" /> [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package |
| <Icon icon="flutter" /> [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings |
## What stays the same {#invariants}
Whichever you pick:
@@ -88,7 +133,7 @@ Whichever you pick:
- **Types are Arrow types.** A vector column is a `FixedSizeList<Float32>`
everywhere, not a SQL type mapped onto storage. See
[Functions and types](/sql/functions).
- **Reads are Arrow.** Nothing is serialized into language objects on the way
out unless you convert it.
- **Reads are Arrow.** Nothing is serialized into language objects on the way out
unless you convert it.
- **The catalog is one catalog.** A table created from TypeScript is visible to
Python and to SQL immediately -- there is no per-language namespace.
docs/web/basics/catalog.mdx
+25 -81
View File
@@ -1,19 +1,24 @@
---
title: "The catalog"
sidebarTitle: "Catalog"
description: "What lives in a LanceDB catalog, how the objects nest, and how you connect to one."
title: "Lakehouse and catalog"
sidebarTitle: "Lakehouse & Catalog"
description: "What a multimodal lakehouse holds, and how the catalog organizes it."
icon: sitemap
keywords: ["catalog", "database", "namespace", "table", "view", "function"]
keywords: ["lakehouse", "catalog", "database", "namespace", "table", "view", "function"]
---
A catalog is what LanceDB knows about. Everything you create lives in one and is
addressed through it.
LanceDB is a **multimodal lakehouse**: data sits in open [Lance](/basics/lance)
tables on object storage, and a catalog gives those tables names, structure, and
the operations that act on them.
The lakehouse part is why images, embeddings, and the columns you derive from
them live in one table rather than in a warehouse plus a vector store plus a
feature store. The catalog part is what this page is about.
## The objects {#objects}
| Object | What it is |
| --- | --- |
| Database | The root. A connection opens exactly one. |
| [Database](/management/database) | The root. A connection opens exactly one. |
| [Namespace](/namespaces) | A named grouping inside a database. Can nest. |
| [Table](/tables) | Rows and columns, versioned. What most operations act on. |
| [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. |
@@ -33,83 +38,22 @@ A namespace can contain further namespaces, so the hierarchy goes as deep as you
need. A table is reached by its path through it.
This page is about what the objects *are*. For the operations on them --
creating, listing, dropping -- see [Management](/management).
connecting, creating, listing, dropping -- see [Management](/management).
## Connecting {#connecting}
## A table is more than rows {#tables}
```python
import lancedb
A Lance table carries things a warehouse table does not, and most of the rest of
these docs is about one of them:
db = lancedb.connect("./.lancedb") # a local directory
db = lancedb.connect("s3://bucket/path") # object storage
db = lancedb.connect("db://my-database", api_key="...") # a deployment
```
| | |
| --- | --- |
| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/computed-columns) from other columns |
| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) |
| **Indexes** | vector, full-text, and scalar, built over the columns |
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
The URI decides what kind of database you get, and that decision reaches further
than it looks. A local or object-store URI opens the data directly from your
process. A `db://` URI talks to a deployment, which is what makes server-side
execution -- [functions](/feature-engineering/functions), remote refreshes,
[SQL](/basics/apis) -- available, and what makes a few local-only features
unavailable.
`connect_async` returns an `AsyncConnection` with the same surface as
coroutines.
### What a table handle actually is {#table-handles}
The URI also decides what `open_table` hands back, which is worth knowing
because the two behave the same but fail differently:
- **`LanceTable`** -- direct access to storage (a local path, `file://`, `s3://`
and similar). Your process resolves and reads the table itself.
- **`RemoteTable`** -- catalog-backed access through a deployment (`db://`).
Resolution and execution happen on the server.
![](/static/assets/images/overview/understanding-tables.png)
Both expose the same table API -- create, open, mutate, query. What differs is
where the work happens, and therefore which operations exist at all.
### Storage options {#storage-options}
`storage_options` configures the underlying store -- credentials, endpoints, and
per-table defaults for tables created through this connection:
```python
db = lancedb.connect(
"s3://bucket/path",
storage_options={"new_table_enable_stable_row_ids": "true"},
)
```
<Warning>
`new_table_enable_stable_row_ids` is worth setting deliberately. Stable row ids
cannot be enabled after a table exists, and a
[materialized view](/feature-engineering/materialized-views) requires its source
to have them. A table created without them cannot later become a view's source
without being recreated.
</Warning>
See [Storage](/storage) for the full set.
### Read consistency {#consistency}
`read_consistency_interval` controls how often a connection checks for versions
committed by other writers. The default reads a snapshot taken when the table was
opened, so a long-lived connection will not see other processes' writes until it
reopens.
Set it when several processes write the same table and readers must observe those
writes:
```python
from datetime import timedelta
db = lancedb.connect(uri, read_consistency_interval=timedelta(seconds=5))
```
The cost is a metadata check on that interval. See
[Consistency](/tables/consistency).
That is why "add a feature column" is a normal operation here rather than a
migration: it writes one new column and leaves the expensive data alone.
## Views and materialized views are different objects {#views}
@@ -131,4 +75,4 @@ latency, or when the query is cheap.
performance decision -- see [Maintenance](/indexing).
**Jobs** are not stored data at all; they are handles to work the server is
doing. See [Jobs](/basics/jobs).
doing. See [Query and job](/basics/jobs).
docs/web/basics/jobs.mdx
+111 -34
View File
@@ -1,39 +1,108 @@
---
title: "Jobs"
sidebarTitle: "Jobs"
description: "The handle every long-running operation returns, and how to wait on, inspect, or cancel one."
title: "Query and job"
sidebarTitle: "Query & Job"
description: "Sync and async in LanceDB: two different distinctions that use the same word."
icon: hourglass-half
keywords: ["job", "async", "background", "create index", "refresh column"]
keywords: ["job", "async", "sync", "query", "background", "await"]
---
"Async" means two unrelated things in LanceDB, and conflating them is the most
common source of confusion about how work gets done.
| | What it is | What it changes |
| --- | --- | --- |
| **Async client** | `connect_async`, `await` | whether *your thread* blocks |
| **Async execution** | `_async` methods returning a [job](#jobs) | whether *the work* outlives the call |
They are independent. A synchronous Python call can start a job that runs for an
hour; an `await` can wait for something that finishes instantly.
## Async client: whether you block {#async-client}
A concern of your language, not of LanceDB. The work is identical either way.
<CodeGroup>
```python Python icon="python"
db = lancedb.connect("db://my-db", api_key="...") # blocking
tbl = db.open_table("documents")
db = await lancedb.connect_async("db://my-db", api_key="...") # coroutines
tbl = await db.open_table("documents")
```
```typescript TypeScript icon="square-js"
// TypeScript is async throughout; there is no blocking form.
const db = await lancedb.connect("db://my-db", { apiKey: "..." });
const tbl = await db.openTable("documents");
```
</CodeGroup>
Python offers both. `AsyncConnection` exposes the same surface as coroutines, so
choosing between them is about the program you are writing -- a service wants
the async form, a script does not care.
## Queries: reading before the query finishes {#queries}
A query is not a job, but it has the same shape of choice: wait for everything,
or take a handle and start reading.
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 1000")
table = reader.read_all()
query = db.execute_query_async("SELECT id, text FROM my_docs")
```
```sql SQL
SELECT id, text FROM my_docs LIMIT 1000
```
</CodeGroup>
`execute_query` returns a blocking Arrow reader **once results start arriving**
-- it does not wait for the whole query to finish. Results stream as record
batches, so a large result set does not have to fit in memory before you see
its first row.
`execute_query_async` returns the query handle instead, for a statement you
would rather not sit on at all. `describe_query` reports on one by id.
<Note>
SQL requires a deployment. See [SDKs and SQL](/basics/apis).
</Note>
Query-builder searches -- vector, full-text, hybrid -- are ordinary calls that
return results. They are fast enough not to need a handle. See
[Query](/search).
## Jobs: work that outlives the call {#jobs}
Some operations take longer than a request should. Building an index over a
large table, refreshing a computed column, registering a function -- these run
on the server after the call that started them has returned.
Each is submitted the same way and returns the same handle. Learn the handle
once and every long-running operation behaves alike.
Each is submitted the same way and returns the same handle.
## The `_async` convention {#convention}
### The `_async` convention {#convention}
An operation that can run in the background has two forms. The plain name
blocks until the work is done and returns its result. The `_async` /
`Async` name submits the work and returns a **job** immediately.
An operation that can run in the background has two forms. The plain name blocks
until the work is done. The `_async` / `Async` name submits the work and returns
a **job** immediately.
| Blocking | Returns a job | What runs |
| Blocking | Returns a job | SQL |
| --- | --- | --- |
| `create_index` | `create_index_async` | index build |
| `refresh_column` | `refresh_column_async` | recompute a [computed column](/feature-engineering/computed-columns) |
| `create_function` | `create_function_async` | register a [function](/feature-engineering/functions) |
| `drop_table` | `drop_table_async` | drop a table |
| `execute_query` | `execute_query_async` | a [SQL](/basics/apis) query |
| `create_index` | `create_index_async` | `CREATE INDEX` |
| `refresh_column` | `refresh_column_async` | `REFRESH COLUMN` |
| `create_function` | `create_function_async` | `CREATE FUNCTION` |
| `drop_table` | `drop_table_async` | `DROP TABLE` |
<Note>
The blocking form is not a different operation -- it submits the same job and
waits. Reach for the `_async` form when you want to do something else
meanwhile, run several at once, or survive the client going away.
waits. Reach for the `_async` form when you want to do something else meanwhile,
run several at once, or survive the client going away.
</Note>
## Submitting and waiting {#waiting}
### Submitting and waiting {#waiting}
<CodeGroup>
```python Python icon="python"
@@ -73,15 +142,15 @@ result = table.refresh_column_async("embedding").wait(timeout=timedelta(minutes=
still running; the handle just gave up watching.
<Note>
The TypeScript `wait()` resolves to `void`. Read the outcome from `result`
after it resolves.
The TypeScript `wait()` resolves to `void`. Read the outcome from `result` after
it resolves.
</Note>
## Inspecting a job {#inspecting}
### Inspecting a job {#inspecting}
Submitting an operation returns only an id, so the detail fields are empty
until the handle fetches the record. `refresh()` does that; `status()` does it
as a side effect.
Submitting an operation returns only an id, so the detail fields are empty until
the handle fetches the record. `refresh()` does that; `status()` does it as a
side effect.
<CodeGroup>
```python Python icon="python"
@@ -110,7 +179,7 @@ job.failure; // why it failed, when it did
Printing the handle renders every field it knows, which is usually faster than
reading them one at a time.
## Events {#events}
### Progress, from events {#events}
`result` only tells you anything once the job has finished. Events are written
as it runs, and they outlive the workers that produced them -- so a job that is
@@ -124,6 +193,10 @@ events = job.events(limit=5000, filter="state = 'claim_complete'")
```typescript TypeScript icon="square-js"
const events = await job.events({ limit: 5000, filter: "state = 'claim_complete'" });
```
```sql SQL
SHOW ERRORS ON my_table FOR JOB 'job-id' LIMIT 100
```
</CodeGroup>
A distributed job records a `claim` / `claim_complete` pair per unit of work,
@@ -142,7 +215,7 @@ silently lose its tail unless you pass `limit`.
Events are recorded server-side, so this raises for an operation that ran
in-process.
## Finding and cancelling jobs {#managing}
### Finding and cancelling jobs {#managing}
The connection lists and reopens jobs, so a handle lost to a restarted process
is recoverable from its id.
@@ -155,9 +228,13 @@ db.cancel_job(job_id) # ask it to stop
```
```typescript TypeScript icon="square-js"
await db.listJobs(); // every job the server knows about
const job = await db.openJob(jobId); // reattach to one
await db.cancelJob(jobId); // ask it to stop
await db.listJobs();
const job = await db.openJob(jobId);
await db.cancelJob(jobId);
```
```sql SQL
SHOW JOBS
```
</CodeGroup>
@@ -172,10 +249,10 @@ parsing it or storing it to resume the job later is not supported.
## In-process jobs {#in-process}
A local connection has no server to run anything, so its operations run in your
process. They still return a handle, so the same code works against both, but
A local database has no server to run anything, so its operations run in your
process. They still return a handle, so the same code works against both -- but
one with nothing behind it: `id` is `None`, `status()` reports `"finished"`,
`wait()` returns immediately, `cancel()` does nothing, and `events()` raises.
This is the difference to expect when moving between a local database and a
deployment -- see [Python, TypeScript, and SQL](/basics/apis).
This is the difference to expect when moving between embedded and deployed --
see [SDKs and SQL](/basics/apis).
docs/web/basics/lance.mdx
+2 -2
View File
@@ -1,6 +1,6 @@
---
title: "What about Lance?"
sidebarTitle: "What about Lance"
title: "Lance format"
sidebarTitle: "Lance Format"
description: "Lance is the open format underneath. LanceDB is what implements a lakehouse on top of it."
icon: "/static/assets/logo/lance-logo-gray.svg"
keywords: ["lance", "format", "open standard", "lakehouse", "namespace", "catalog"]
docs/web/docs.json
+15 -21
View File
@@ -58,6 +58,10 @@
{
"source": "/sql",
"destination": "/basics/apis"
},
{
"source": "/api-reference",
"destination": "/basics/apis"
}
],
"navigation": {
@@ -75,17 +79,25 @@
{
"group": "Basics",
"pages": [
"basics/lance",
"basics/catalog",
"basics/apis",
"basics/jobs",
"basics/agentic-skills",
"basics/lance"
"basics/apis",
{
"group": "REST API",
"expanded": true,
"pages": [
"api-reference/rest/index"
]
},
"basics/agentic-skills"
]
},
{
"group": "Management",
"pages": [
"management/index",
"management/database",
{
"group": "Namespace",
"pages": [
@@ -334,24 +346,6 @@
}
]
},
{
"tab": "API Reference",
"groups": [
{
"group": "API Reference",
"pages": [
"api-reference/index",
{
"group": "REST API",
"expanded": true,
"pages": [
"api-reference/rest/index"
]
}
]
}
]
},
{
"tab": "Agentic Skills",
"groups": [
docs/web/management/database.mdx
+194
View File
@@ -0,0 +1,194 @@
---
title: "Database"
sidebarTitle: "Database"
description: "Connecting to LanceDB: the URI forms, their options, and what each one changes."
icon: database
keywords: ["connect", "database", "uri", "api key", "storage options", "consistency"]
---
A database is the root of a [catalog](/basics/catalog). A connection opens
exactly one, and every other object is reached through it.
Connecting is the one decision that reaches furthest: the URI you pass decides
where queries execute, which features exist, and what a table handle actually
is.
## The three URI forms {#uris}
<CodeGroup>
```python Python icon="python"
import lancedb
db = lancedb.connect("./.lancedb") # local directory
db = lancedb.connect("s3://bucket/path") # object storage
db = lancedb.connect("db://my-database", api_key="...") # a deployment
```
```typescript TypeScript icon="square-js"
import * as lancedb from "@lancedb/lancedb";
const db = await lancedb.connect("./.lancedb");
const db = await lancedb.connect("s3://bucket/path");
const db = await lancedb.connect("db://my-database", { apiKey: "..." });
```
</CodeGroup>
The first two are **embedded**: your process opens Lance tables directly from
storage and does the work itself. The third talks to a **deployment**.
| | Embedded | Deployment |
| --- | --- | --- |
| Where queries run | in your process | on the server |
| Table handle | `LanceTable` | `RemoteTable` |
| [SQL](/basics/apis) | -- | yes |
| Server-side [jobs](/basics/jobs) | -- | yes |
| [Functions](/feature-engineering/functions) | -- | yes |
| [Authorization](/governance/authorization) | storage permissions | granted per object |
![](/static/assets/images/overview/understanding-tables.png)
Both handles expose the same table API -- create, open, mutate, query -- so
application code moves between them by changing the URI. What differs is where
resolution and execution happen, and therefore which operations exist at all.
## Connecting to a deployment {#deployment}
<Badge color="red">Enterprise</Badge>
<CodeGroup>
```python Python icon="python"
db = lancedb.connect(
"db://my-database",
api_key="...",
region="us-east-1",
)
```
```typescript TypeScript icon="square-js"
const db = await lancedb.connect("db://my-database", {
apiKey: "...",
region: "us-east-1",
});
```
</CodeGroup>
| Option | Meaning |
| --- | --- |
| `api_key` / `apiKey` | authenticates the connection -- see [Authentication](/enterprise/authentication) |
| `region` | which region hosts the database. Defaults to `us-east-1` |
| `host_override` / `hostOverride` | point at a specific endpoint instead of the regional one |
| `sql_host_override` | the same, for the [SQL endpoint](/search/sql). Python only |
| `client_config` / `clientConfig` | retries, timeouts, and connection pooling |
An API key belongs in the environment, not in source. `region` is easy to get
wrong silently -- the default is `us-east-1` whether or not that is where your
database lives, and a wrong region fails as a connection error rather than a
helpful one.
## Storage options {#storage-options}
`storage_options` configures the underlying object store -- credentials,
endpoints, and defaults for tables created through this connection:
<CodeGroup>
```python Python icon="python"
db = lancedb.connect(
"s3://bucket/path",
storage_options={"new_table_enable_stable_row_ids": "true"},
)
```
```typescript TypeScript icon="square-js"
const db = await lancedb.connect("s3://bucket/path", {
storageOptions: { new_table_enable_stable_row_ids: "true" },
});
```
</CodeGroup>
<Warning>
`new_table_enable_stable_row_ids` is worth setting deliberately. Stable row ids
cannot be enabled after a table exists, and a
[materialized view](/feature-engineering/materialized-views) requires its source
to have them. A table created without them cannot later become a view's source
without being recreated.
</Warning>
See [Storage](/storage) for the full set, including per-cloud credentials.
## Read consistency {#consistency}
`read_consistency_interval` controls how often a connection checks for versions
committed by other writers.
<CodeGroup>
```python Python icon="python"
from datetime import timedelta
db = lancedb.connect(uri, read_consistency_interval=timedelta(seconds=5))
```
```typescript TypeScript icon="square-js"
const db = await lancedb.connect(uri, { readConsistencyInterval: 5 });
```
</CodeGroup>
The default reads a snapshot taken when the table was opened, so a long-lived
connection **will not see other processes' writes** until it reopens the table.
That is the right default for a batch job and the wrong one for a service.
Set it when several processes write the same table and readers must observe
those writes. The cost is a metadata check on that interval. See
[Consistency](/tables/consistency).
## Database-level operations {#operations}
| | Python | TypeScript | SQL |
| --- | --- | --- | --- |
| list tables | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` |
| the URI it opened | `uri` | `display()` | -- |
| drop every table | `drop_all_tables` | `dropAllTables` | -- |
| drop the database | `drop_database` | -- | -- |
| close | -- | `close`, `isOpen` | -- |
<CodeGroup>
```python Python icon="python"
db.table_names()
tbl = db.open_table("documents")
db.drop_table("documents")
```
```typescript TypeScript icon="square-js"
await db.tableNames();
const tbl = await db.openTable("documents");
await db.dropTable("documents");
```
```sql SQL
SHOW TABLES;
DROP TABLE documents;
```
</CodeGroup>
Each takes a `namespace_path` to act inside a [namespace](/namespaces) rather
than at the root.
<Warning>
`drop_database()` removes every table in the database. There is no confirmation
and no undo -- table [versioning](/tables/versioning) does not survive the table
being dropped.
</Warning>
## Async connections {#async}
`connect_async` returns an `AsyncConnection` with the same surface as
coroutines. The TypeScript SDK is async throughout, so its `connect` already is
one.
```python
db = await lancedb.connect_async("db://my-database", api_key="...")
tbl = await db.open_table("documents")
```
This is a different axis from the `_async` operations that return a
[job](/basics/jobs): `connect_async` is about not blocking your event loop,
while `create_index_async` is about work that outlives the call.
docs/web/namespaces/usage.mdx
+1 -1
View File
@@ -144,6 +144,6 @@ serves more than one database, LanceDB can also use headers such as `x-lancedb-d
## Related references {#related-references}
- [Client SDK API references](/api-reference)
- [Client SDK API references](/basics/apis#reference)
- [REST API Reference](/api-reference/rest)
- [Namespaces and the Catalog Model](/namespaces)
docs/web/quickstart.mdx
+76 -6
View File
@@ -382,6 +382,38 @@ Next, you can query a compact view of the new feature:
The same workflow is used for data preparation tasks when adding derived features, cached model signals, review scores, or dataset
quality indicators.
### Declare now, compute later {#computed-columns}
The call above computes every row as part of the call, which is fine for three
rows. On a table large enough that you would rather choose *when* the work
happens, declare the column instead and fill it separately:
```python
# Commits the column, empty. Nothing is computed yet.
characters.add_columns(computed={"power_score": "(stats.magic + stats.strength) / 2.0"})
# Fills it, and commits again.
characters.refresh_column("power_score")
# RefreshColumnResult(rows_filled=3, version=4)
```
A refresh only gives values to rows that have none, so appending more characters
and refreshing again fills just the new ones -- it does not recompute the rows
already done. That is what makes a backfill safe to retry and cheap to repeat as
data arrives.
On a deployment the same refresh runs server-side, on a cluster, as a
[job](/basics/jobs) you can walk away from:
```python
job = characters.refresh_column_async("power_score")
job.wait()
```
See [Computed columns](/feature-engineering/computed-columns) for expressions,
and [Functions](/feature-engineering/functions) for filling a column with Python
-- an embedding model, a captioner, anything you can register.
## 7\. Store multimodal data {#7-store-multimodal-data}
Multimodal data is a first-class citizen in LanceDB. Binary data (image, audio, video, etc.) is
@@ -424,6 +456,44 @@ These snippets load the local image file and store the bytes in an `image` colum
For more examples, see the [multimodal data](/tables/multimodal) section.
## 8\. Train on the result {#8-train}
The point of the previous steps is this one. The table you searched, filtered,
and added features to is directly usable as a PyTorch
[Dataset](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset)
-- there is no export step, and no second copy of the data to keep in sync.
```python
import torch
from lancedb.util import tbl_to_tensor
dataloader = torch.utils.data.DataLoader(
characters, batch_size=1024, shuffle=True, collate_fn=tbl_to_tensor
)
for batch in dataloader:
... # train
```
Because the loader reads from the table, the feature columns you just
materialized are read straight off storage rather than recomputed every epoch.
That is the whole argument for doing feature engineering in the lakehouse: an
expensive transform runs once, is versioned with the data, and every subsequent
epoch is a read.
Projection matters here. Selecting only the columns a step needs means the
images are never read for an epoch that only wants embeddings:
```python
from lancedb.permutation import Permutation
permutation = Permutation.identity(characters)
dataloader = torch.utils.data.DataLoader(permutation)
```
See [Training](/training) for shuffling, multi-worker loading, and reading from
a remote table.
## Code {#code}
See the full code for these examples (including helper functions) in the
@@ -432,13 +502,13 @@ See the full code for these examples (including helper functions) in the
## Next steps {#next-steps}
You've learned how to install LanceDB, connect, create one table for AI data, retrieve related
examples, curate with metadata, add a derived feature, and represent multimodal records. These same
primitives apply across the AI data lifecycle, from data preparation and feature engineering to
retrieval, evaluation, and training.
You've gone from a table, through search and curation, to derived features and a
training loop -- in one table, with no export step between any two stages. That
arc is the argument for a multimodal lakehouse, and every stage of it scales
past this toy example without changing shape.
Continue to the table and search guides to build on this example with schema options, appends,
updates, versioning, indexing, full-text search, hybrid search, and reranking.
Continue to the guides to build on it with schema options, versioning, indexing,
full-text and hybrid search, reranking, and server-side feature engineering.
<Columns cols={2}>
<Card