rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-30 15:26:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)

Browse Source 
## Summary

- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
  Fix the blob-column default-projection claim and add the previously
  undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
  chained Functions over blob outputs, GPU-produced blobs, Secrets,
  multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
  precise, verified catch-up semantics, and add the missing `ALTER
  MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
  automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
  (52 files); activate the existing availability-comparison mechanism on
  `basics/offerings.mdx` for a full feature-by-feature comparison table.
This commit is contained in:
Jack Ye
2026-10-29 10:16:03 -07:00
committed by GitHub
parent 005a9a353a
commit ddb318567a
133 changed files with 3978 additions and 2918 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/.mintignore
+19 -3
View File
@@ -1,6 +1,22 @@
# Functions are documented before their release. At release, remove these
# entries, add the pages back to the navigation in docs.json, restore the
# /feature-engineering/functions redirect to /management/function, and publish
# the drafts marked "Draft for the Functions release".
# entries, restore the /feature-engineering/functions redirect to
# /management/function, publish the drafts marked "Draft for the Functions
# release", restore `availability` frontmatter (removed while gated, since
# the comparison table on index.mdx links every page that declares one,
# draft or not) to management/function.mdx and sql/ddl/function.mdx -- each
# currently carries a plain `<Badge color="red">Enterprise</Badge>` instead
# -- and re-link curation/chunking.mdx's plain-text `CREATE FUNCTION`/
# `add_columns` mentions to /management/function and /sql/ddl/function#create
# now that those pages have a route. Chunking stays published ahead of
# Functions shipping, so its examples intentionally do not link to either
# gated page -- only an unrouted page would break.
management/function.mdx
sql/ddl/function.mdx
# GRANT/REVOKE currently only covers privileges on Secrets and Functions
# (CREATE SECRET, CREATE FUNCTION, USAGE on a secret or function), which are
# themselves part of the Functions release: see the "Draft for the Functions
# release" block in governance/authorization.mdx#privileges. At release,
# remove this entry, add the page to the navigation in docs.json, and give it
# `availability` frontmatter like the rest of the SQL reference.
sql/ddl/grant.mdx
docs/web/architecture.mdx
-128
View File
@@ -1,128 +0,0 @@
---
title: "Architecture"
sidebarTitle: "Architecture"
description: "How LanceDB Enterprise separates query serving, background work, and storage, and how it compares with LanceDB Embedded."
icon: cubes
keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
---
LanceDB Embedded is an open-source library for working with Lance tables. It runs
in your application's process; your data can live on local disk or in object
storage. LanceDB Enterprise is a multimodal lakehouse for data curation, feature
engineering, search, and training data access. LanceDB Enterprise runs LanceDB
Distributed: query execution, indexing and maintenance run on the Enterprise
service, outside your application. Its architecture separates query serving,
read execution, caching, and background work. Both use the open
[Lance](/basics/lance) format.
Teams can curate datasets, derive features, search multimodal records, and
read the resulting tables into training frameworks. Enterprise separates the
resources that serve shared data from those that build indexes and run
background work, so teams can size them for these different workloads.
![LanceDB Embedded runs the application and embedded library in one process, using local disk or object storage. Enterprise applications connect to query serving, with separate read execution and caching, indexing and background workers, and durable object storage. Both use the Lance format, but they do not share a storage location automatically.](/static/assets/images/overview/lancedb-architecture.svg)
The diagram shows the main compute and storage boundaries, not every request
path. The control plane governs the Enterprise service.
## A shared format, different storage choices {#storage}
Lance stores table data, versions, and indexes. LanceDB Embedded can read and write tables on
local disk or in object storage. Enterprise keeps durable table data and index
artifacts in object storage, independently of the machines serving requests.
Its caches accelerate reads; they are not the durable source of truth.
Sharing a format does not mean that an Enterprise deployment automatically
opens an existing LanceDB Embedded database. Plan how to move or connect your data and
configure access for your deployment.
## LanceDB Embedded: a library inside your application {#embedded}
With an embedded connection, the LanceDB library plans and executes queries,
builds indexes, and updates tables inside your process. There is no separate
database server to deploy.
This fits local retrieval applications, notebooks, development, and
single-node data pipelines. Your application controls the compute resources and
when to run [table maintenance](/maintenance/compaction). You also manage access
to the local files or object storage that hold your tables.
## LanceDB Enterprise: distributed execution for shared workloads {#distributed}
A LanceDB SDK connects to LanceDB Enterprise, and the service runs operations
with LanceDB Distributed. The service handles
requests across a fleet, with components sized for different kinds of work:
- **Query serving** receives requests, resolves tables, and plans operations.
Add serving capacity as more applications and users depend on your data.
- **Read execution and caching** use separately provisioned resources. Executors
cache table data to reduce repeated reads from object storage.
- **Indexing and background workers** build indexes and perform maintenance
outside the query request path. Size this work separately from query serving
as your data grows.
The control plane manages configuration, identity, policy, and cluster
lifecycle. It governs the service rather than executing table queries.
[Authorization](/governance/authorization) lets teams control access to catalog
objects when enabled and configured by the administrator. Deployment settings govern background
[compaction](/maintenance/compaction) and
[cleanup](/maintenance/cleanup), including retention.
These boundaries support different parts of an AI data workflow.
[SQL curation](/curation) requires a deployment with SQL enabled.
[Feature engineering](/feature-engineering) can produce derived columns and
tables; computed columns and materialized views require a deployment that
enables them. [Geneva](/geneva) runs distributed Python feature backfills while
its capabilities continue to transition to LanceDB APIs.
[Training data loaders](/training/data-loading) read tables into model training
frameworks in both offerings. LanceDB provides data access for training; the
training framework runs the model.
Size and monitor each tier for your workload. Read the
[Enterprise architecture](/enterprise/architecture) for the component details
and request flows.
## Application changes for Enterprise {#differences}
Core operations remain familiar: create a table, add data, and run a search.
With LanceDB Embedded, the application executes those operations. With Enterprise, the SDK
sends requests to the service, and the service manages execution and configured
background maintenance.
To use Enterprise, configure the cluster endpoint, database URI and credentials,
and make your data available in the Enterprise database. Review SDK methods
that behave differently with LanceDB Distributed, including result retrieval and
background jobs. [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise)
describes these application changes. If you plan to use materialized views, check the
[stable row ID requirements](/basics/limits#schema) before creating source tables.
## Choose for the workload you need to run {#choosing}
Choose LanceDB Embedded when you want an embedded library and can manage its resources and
maintenance in your application.
Choose Enterprise when multiple applications or teams need a shared multimodal
lakehouse for curation, feature engineering, search, and training data access.
It also fits workloads whose query and indexing demand grow at different rates,
or that need configured service-level access controls and background operations.
You can begin with Enterprise.
[Talk to us about your workload](https://www.lancedb.com/contact) to plan capacity,
deployment options, and the path from your existing data to Enterprise.
## Where to go next {#next}
<CardGroup cols={2}>
<Card title="Enterprise architecture" icon="cubes" href="/enterprise/architecture">
Query serving, execution, and background work in detail.
</Card>
<Card title="Connect to Enterprise" icon="plug" href="/basics/connect#enterprise">
Connect an application to your deployment.
</Card>
<Card title="Guarantees" icon="shield-check" href="/basics/guarantees">
What a write promises, and what a reader sees.
</Card>
<Card title="Lance format" icon="layer-group" href="/basics/lance">
The open format underneath both offerings.
</Card>
</CardGroup>
docs/web/basics/agentic-skills.mdx
+1 -1
View File
@@ -273,7 +273,7 @@ private deployment, or platform-managed operations. The data format and the core
table API stay the same, so the pipeline's table and search code carries over:
the agent connects to an Enterprise `db://` table, leaves maintenance to the
deployment instead of calling `optimize()`, and submits long-running work as
jobs. See [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
[jobs](/basics/jobs).
## What agents are good at here {#workflows}
docs/web/basics/catalog.mdx
+118 -6
View File
@@ -1,16 +1,128 @@
---
title: "Lance Catalog"
sidebarTitle: "Lance Catalog"
description: "Organize, find and control access to your data in LanceDB."
title: "LanceDB Catalog"
sidebarTitle: "LanceDB Catalog"
description: "LanceDB Lakehouse objects are organized into a catalog of databases, namespaces and tables -- connect to a database to work with them."
icon: sitemap
keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"]
keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view", "connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"]
---
import {
PyConnect,
PyConnectAsync,
PyConnectEnterprisePlaceholders,
PyConnectObjectStorage,
PyConnectObjectStorageAsync,
RsConnect,
TsConnect,
TsConnectEnterprisePlaceholders,
TsConnectObjectStorage,
} from '/snippets/connection.mdx';
## Catalog overview {#catalog}
A catalog keeps track of named data objects, such as tables and views, so
applications can discover and access them. In LanceDB, it organizes those
objects into databases and namespaces.
objects into databases and namespaces -- similar to a catalog in Postgres,
Snowflake, or other database systems.
## Connecting {#connecting}
You always run table and query operations within a **database**; cross-database
operations go through SQL instead. The same Python, TypeScript and Rust SDKs
reach a database two ways:
- **Directly, the common case.** `lancedb.connect(uri)` opens one database in a
single call -- a local directory, an object-storage path, or (with a `db://`
URI) a specific Enterprise database.
- **Through the catalog, for admin and cross-database work.** `connect_catalog(endpoint)`
connects to Enterprise's root catalog, which can list, create and drop
databases across the whole deployment. Its `connect_database(name)` and
`create_database(name)` then return the same kind of database connection
`lancedb.connect()` does. Local filesystem and object-store deployments have
no catalog to connect to -- each is already scoped to one database.
The table below shows which URI reaches which kind of database directly.
| URI | Where queries run |
| --- | --- |
| A local path, such as `./data` | Embedded: in your process |
| `s3://`, `gs://` or `az://` | Embedded: in your process, reading object storage |
| `db://` | LanceDB Enterprise: queries run on your cluster |
### A local directory {#local}
Embedded in your application, LanceDB reads and writes the tables in a
directory on disk.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnect}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::connect;\n" }
{ "\n" }
{RsConnect}
</CodeBlock>
</CodeGroup>
### Object storage {#object-storage}
With an object storage URI the library is still embedded: your process runs
every query and reads and writes the bucket directly, with the credentials from
its environment or its storage options. [Configuring storage](/deployment/storage)
covers credentials, endpoints and provider options.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectObjectStorageAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectObjectStorage}
</CodeBlock>
</CodeGroup>
In Rust, pass the same URI to `connect`.
### LanceDB Enterprise {#enterprise}
A `db://` URI connects directly to one database in LanceDB Enterprise, where
your cluster runs queries and background work. Get the database URI, API key,
region and endpoint from your administrator, then replace the placeholders
below. Use your cluster's region in place of `us-east-1`. In application code,
load credentials from environment variables or a secrets manager.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterprisePlaceholders}
</CodeBlock>
</CodeGroup>
In Rust, enable the crate's `remote` feature; the connection builder then takes
the same settings through `api_key`, `region` and `host_override`. To use OAuth
instead of an API key, see
[Authentication](/enterprise/authentication); for every connection option, see
[Database](/management/database#deployment).
## Databases, namespaces and tables {#objects}
@@ -37,7 +149,7 @@ flowchart TD
| [Namespace](/management/namespace) | A named group of tables, views or nested namespaces. |
| [Table](/management/table) | Data stored as rows and columns. |
| [View](/management/view) | A saved query, evaluated when read. Available in Enterprise. |
| [Materialized view](/management/materialized-view) | The stored results of a query, updated by a refresh. |
| [Materialized view](/management/materialized-view) | The stored results of a query, updated by a refresh. Available in Enterprise. |
{/*
Draft for the Functions release. Add this row to the table above:
docs/web/basics/connect.mdx
-135
View File
@@ -1,135 +0,0 @@
---
title: "Connect to LanceDB"
sidebarTitle: "Connect"
description: "Use LanceDB Embedded with tables in a local directory or object storage, or connect to LanceDB Enterprise."
icon: plug
keywords: ["connect", "uri", "embedded", "object storage", "s3", "enterprise", "db://"]
---
import {
PyConnect,
PyConnectAsync,
PyConnectEnterprisePlaceholders,
PyConnectEnterpriseQuickstart,
PyConnectObjectStorage,
PyConnectObjectStorageAsync,
RsConnect,
TsConnect,
TsConnectEnterprisePlaceholders,
TsConnectEnterpriseQuickstart,
TsConnectObjectStorage,
} from '/snippets/connection.mdx';
The URI you pass to `connect` decides where the work runs. LanceDB Embedded
runs in your application's process; your data can live on local disk or in
object storage. LanceDB Enterprise runs LanceDB Distributed: query execution,
indexing and maintenance run on the Enterprise service, outside your
application. The same Python, TypeScript and Rust SDKs open all three kinds;
[Embedded and Enterprise](/basics/offerings) describes how the two offerings differ.
| URI | Where queries run |
| --- | --- |
| A local path, such as `./data` | LanceDB Embedded: in your process |
| `s3://`, `gs://` or `az://` | LanceDB Embedded: in your process, reading object storage |
| `db://` | LanceDB Enterprise: LanceDB Distributed runs queries on your cluster |
## A local directory {#local}
Embedded in your application, LanceDB reads and writes the tables in a
directory on disk.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnect}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{ "use lancedb::connect;\n" }
{ "\n" }
{RsConnect}
</CodeBlock>
</CodeGroup>
## Object storage {#object-storage}
With an object storage URI the library is still embedded: your process runs
every query and reads and writes the bucket directly, with the credentials from
its environment or its storage options. [Configuring storage](/deployment/storage)
covers credentials, endpoints and provider options.
<CodeGroup>
<CodeBlock filename="Python (sync)" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Python (async)" language="Python" icon="python">
{PyConnectObjectStorageAsync}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectObjectStorage}
</CodeBlock>
</CodeGroup>
In Rust, pass the same URI to `connect`.
## LanceDB Enterprise {#enterprise}
A `db://` URI connects to LanceDB Enterprise, where your cluster runs queries
and background work. Get the database URI, API key, region and endpoint from
your administrator, then replace the placeholders below. Use your cluster's region
in place of `us-east-1`. In application code, load credentials from environment
variables or a secrets manager.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterprisePlaceholders}
</CodeBlock>
</CodeGroup>
In Rust, enable the crate's `remote` feature; the connection builder then takes
the same settings through `api_key`, `region` and `host_override`. To use OAuth
instead of an API key, see
[Authentication](/enterprise/authentication); for every connection option, see
[Database](/management/database#deployment).
<Accordion title="Load production credentials from your environment">
Set these values in the shell that starts your application. Replace the URI, API
key, region and endpoint with your Enterprise connection settings. In production, a secrets
manager can supply these environment variables for you.
```shell
export LANCEDB_URI="db://your-database"
export LANCEDB_API_KEY="your-api-key"
export LANCEDB_REGION="us-east-1"
export LANCEDB_HOST_OVERRIDE="https://your-endpoint"
```
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterpriseQuickstart}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{ "import * as lancedb from \"@lancedb/lancedb\";\n\n" }
{TsConnectEnterpriseQuickstart}
</CodeBlock>
</CodeGroup>
</Accordion>
docs/web/basics/guarantees.mdx
+2 -2
View File
@@ -87,5 +87,5 @@ cannot undo `drop_table`.
- [Table versioning](/management/table#versioning): inspect and restore history.
- [Cleanup](/maintenance/cleanup): configure retention and understand protected versions.
- [Jobs](/basics/jobs): distinguish submission, completion and cancellation.
- [Column refresh](/feature-engineering/refresh-column): understand partial progress.
- [Architecture](/architecture): understand storage and background maintenance.
- [Refresh a computed column](/feature-engineering/refresh-column): understand partial progress.
- [Architecture](/#architecture): understand storage and background maintenance.
docs/web/basics/jobs.mdx
+44 -5
View File
@@ -5,11 +5,8 @@ description: "Track background work from submission to completion."
icon: hourglass-half
keywords: ["job", "background", "progress", "events", "cancel"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Job handles track operations in LanceDB Embedded and LanceDB Enterprise. Enterprise also
supports listing server jobs, reopening them by ID and reading their events.
---
## Job handles {#convention}
@@ -26,6 +23,14 @@ so check its status before
waiting or requesting cancellation. See the operation's API reference for when
the method returns.
```python
job = tbl.create_index_async("vector", replace=True)
print(job.id)
print(job.job_type)
print(job.status())
```
## Submit and wait for a job {#waiting}
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
@@ -39,6 +44,22 @@ In Python, `job.status()` reports the current state and `job.wait()` waits for
completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting.
The timeout does not cancel the server job.
```python
from datetime import timedelta
from lancedb.exceptions import JobFailedError, JobCancelledError
job = tbl.create_index_async("vector", replace=True)
try:
result = job.wait(timeout=timedelta(minutes=10))
except JobFailedError:
print(f"job failed: {job.failure}")
except JobCancelledError:
print("job was cancelled")
except TimeoutError:
print(f"still running: {job.status()}")
```
<span id="inspecting" />
<span id="events" />
@@ -54,6 +75,24 @@ The timeout does not cancel the server job.
Reopening a job retrieves the existing operation; it does not restart failed
work. Treat the ID as an opaque identifier and pass it unchanged to the jobs API.
```python
job = db.open_job(job_id) # raises JobNotFoundError if unknown
job.refresh() # pull the latest status from the server
print(job.status())
events = job.events(limit=100)
print(events.to_pylist())
job.cancel()
```
List jobs on a connection, paging through the results:
```python
for info in db.list_jobs(page_limit=50):
print(info.job_id, info.job_type, info.state)
```
Event responses have [row limits](/basics/limits#jobs). An explicit limit can
request more than the default, but it does not guarantee a complete event
history.
@@ -97,7 +136,7 @@ workers. That placement does not change the handle's query-backed identity.
## Learn more {#next}
- [Build and repair indexes](/indexing/reindexing): submit index work.
- [Column refresh](/feature-engineering/refresh-column): populate derived values.
- [Refresh a computed column](/feature-engineering/refresh-column): populate derived values.
- [Background jobs](/maintenance/background-jobs): understand Enterprise maintenance.
- [SQL job commands](/sql/ddl/job): list jobs and inspect errors.
- [SDKs](/basics/sdk): find the job API for your client.
docs/web/basics/lakehouse.mdx
-75
View File
@@ -1,75 +0,0 @@
---
title: "LanceDB Lakehouse"
sidebarTitle: "LanceDB Lakehouse"
description: "Bring multimodal data, search and feature engineering together for AI workloads."
icon: layer-group
keywords: ["lakehouse", "multimodal", "search", "compute", "training", "feature engineering"]
---
## Multimodal storage and processing {#multimodal}
LanceDB brings storage, search and data processing together for AI workloads.
You can store multimodal data, derive features, search your records and prepare
training datasets using tables built on the open [Lance format](/basics/lance).
The format provides the storage foundation. The [catalog](/basics/catalog)
organizes your tables. LanceDB provides the tools to work with that data.
## Example: a video search workflow {#capabilities}
Consider a team building a video search application:
<span id="ingest" />
<span id="compute" />
<span id="search" />
1. **Ingest the clips.** Store video clips with metadata such as timestamps,
source and labels.
2. **Add features.** Generate captions and embeddings with your models, and
store them alongside each clip.
3. **Use the dataset.** Search for relevant clips using embeddings and metadata
filters, or select a dataset to read into a training framework.
```mermaid
flowchart TD
clips["Video clips and metadata"] --> table["Store in a Lance table"]
table --> features["Add captions and embeddings"]
features --> search["Search for relevant clips"]
features --> training["Select data for training"]
```
The clips, metadata and derived features stay associated in the same table,
so retrieval and training workflows can use the same dataset. Your training
framework runs the model; LanceDB provides access to the training data.
## Shared compute and maintenance in Enterprise {#where}
LanceDB Embedded runs as an embedded library in your application. LanceDB Enterprise
provides shared infrastructure for applications and teams working with larger
multimodal workloads.
<span id="maintenance" />
- **Scale resources for different workloads.** Provision query serving and
background workers independently as search traffic and indexing demand grow.
- **Share data across teams.** Give applications access to shared tables, with
Enterprise permissions controlling who can read or change them.
- **Manage ongoing maintenance.** Configure background indexing, compaction
and cleanup as your datasets grow.
See [Architecture](/architecture) for how Enterprise separates storage, query
serving and background work.
## Learn more {#next}
- [Data ingestion](/ingestion): load and update your data.
- [Search](/search): retrieve records with vector, full-text and hybrid search.
- [Feature engineering](/feature-engineering): derive columns and datasets.
- [Training data access](/training/data-loading): read datasets into training workflows.
- [Enterprise](/enterprise): explore deployment options for shared workloads.
{/*
Draft for the Functions release. Add this item to the list above, after
Feature engineering:
- [Functions](/management/function): run Python to compute column values.
*/}
docs/web/basics/lance.mdx
+2 -2
View File
@@ -2,7 +2,7 @@
title: "Lance format"
sidebarTitle: "Lance Format"
description: "The open format for multimodal data behind LanceDB."
icon: "/static/assets/logo/lance-logo-gray.svg"
icon: "/static/assets/logo/lance-logo-gray.png"
keywords: ["lance", "format", "multimodal", "versioning", "column evolution"]
---
@@ -42,4 +42,4 @@ those fields together.
- [Lance format specification](https://lance.org/format): how the format works.
- [Table versioning](/management/table#versioning): read and manage table versions.
- [Computed columns](/feature-engineering/add-computed-column): add derived features to your data.
- [Architecture](/architecture): how LanceDB Embedded and LanceDB Enterprise build on Lance.
- [Architecture](/#architecture): how LanceDB Embedded and Enterprise build on Lance.
docs/web/basics/limits.mdx
+3 -3
View File
@@ -38,9 +38,9 @@ See [SQL job commands](/sql/ddl/job#errors) and
Data-type and index constraints belong to the operation you choose. Check
[Data types](/sql/data-types) and [Vector indexes](/indexing/vector-index)
before choosing a schema or index. For materialized views, check
[stable row ID requirements](/feature-engineering/materialized-views#stable-row-ids)
before creating source tables.
before choosing a schema or index. For materialized views, confirm your
source table meets your deployment's requirements with your administrator
before creating one over it.
## Configurable limits {#not-quotas}
docs/web/basics/offerings.mdx
-73
View File
@@ -1,73 +0,0 @@
---
title: "LanceDB Embedded or Enterprise?"
sidebarTitle: "Embedded and Enterprise"
description: "Start with an embedded library, or use the LanceDB Enterprise multimodal lakehouse for shared AI workloads."
icon: compass
keywords: ["oss", "enterprise", "embedded", "deployment", "availability", "comparison"]
---
LanceDB gives you two ways to work with multimodal data, both built on the open
[Lance](/basics/lance) format.
**LanceDB Embedded (open source)** is a library that runs in your
application's process; your data can live on local disk or in object storage.
You control the application,
its resources, and table maintenance, without a separate database service.
[Try the quickstart](/quickstart) to create a table and run your first search.
**LanceDB Enterprise** is a multimodal lakehouse for data curation, feature
engineering, search, and training data access. It adds query-serving capacity,
data caching, and background operations for teams working with shared
multimodal datasets. LanceDB Enterprise runs LanceDB Distributed: query
execution, indexing and maintenance run on the Enterprise service, outside your
application.
[Connect to Enterprise](/basics/connect#enterprise) when your cluster is ready.
<span id="labels" />
<span id="availability" />
## Why choose Enterprise? {#why-enterprise}
### Serve growing workloads {#serving}
Add query-serving capacity as more applications and users depend on your data.
Enterprise spreads query traffic across a fleet and caches frequently read
table data near compute, reducing repeated reads from object storage. Query
serving and background work use separate resources, so you can size each for
your workload. See [Enterprise architecture](/enterprise/architecture).
### Reduce operational work {#operations}
Let the service handle routine table maintenance instead of scheduling it in
application code. Enterprise handles background [compaction](/maintenance/compaction) and can
[clean up old table versions](/maintenance/cleanup) according to your retention
policy. Configure these maintenance tasks for your Enterprise cluster so each
application does not have to schedule them.
### Fit your deployment requirements {#deployment}
Choose a managed deployment or bring LanceDB into your own cloud account.
Enterprise integrates with OpenTelemetry for metrics and traces, helping your
team monitor the service alongside your applications. Talk to us about your
cloud, networking, and operational requirements to plan the right deployment.
The [Enterprise overview](/enterprise) explains the available options.
### Connect your data workflows {#workflows}
Use the same tables for retrieval, dataset preparation, and training data access.
With its [SQL endpoint](/basics/sql) enabled, Enterprise supports analytical
[curation](/curation), including joins, grouping, and sampling.
[Feature engineering](/feature-engineering) adds derived columns and tables;
computed columns and materialized views require the corresponding features to
be enabled for your Enterprise cluster.
[Geneva](/geneva) runs Python feature backfills as distributed jobs.
[Training data loaders](/training/data-loading), available in both offerings,
read the resulting datasets for model training.
## Plan your Enterprise deployment {#get-started}
[Talk to us about your workload](https://www.lancedb.com/contact) to discuss your
data, query traffic, pipelines, and deployment requirements.
For a closer look at how the service runs, read the
[Enterprise architecture](/enterprise/architecture).
docs/web/basics/query.mdx
-55
View File
@@ -1,55 +0,0 @@
---
title: "Queries"
sidebarTitle: "Query"
description: "Choose how to query your data and read the results."
icon: magnifying-glass
keywords: ["query", "search", "SQL", "async", "streaming", "results"]
---
## Querying tables {#queries}
A query selects data from your tables. You can filter rows, choose columns,
search for similar content or combine those operations. For example, a video
application might find clips similar to a sample and return only those recorded
in the last month.
## SDK searches and SQL queries {#searches}
| Interface | When to use it |
| --- | --- |
| **SDK query builders** | Build vector, full-text and hybrid searches, apply filters and choose the fields to return. Available in LanceDB Embedded and LanceDB Enterprise. |
| **Enterprise SQL** | Write SQL queries for filtering, aggregation, joins and other data curation tasks. Connect through Python or a Flight SQL client. |
Both interfaces work with LanceDB tables. Choose the one that supports your
operation and fits your application. See [SQL](/basics/sql) for connection
instructions and SQL search support, and [Search](/search) for the query builders.
## Collect or stream results {#results}
- **Collect the results** when you want a list, Arrow table or DataFrame that
fits in memory. Use a result limit and select only the columns you need.
- **Read batches** when you want to process a large result incrementally.
Reading batches avoids loading the entire result into application memory.
It does not guarantee that the query can return its first batch immediately.
### Asynchronous clients {#async-client}
An async client lets your application do other work while a request is waiting.
That is separate from whether you collect results or read them in batches.
Python offers synchronous and asynchronous clients; TypeScript uses async APIs.
Check the SDK reference for the methods available in your client.
### Query handles and background jobs {#jobs}
An Enterprise SQL query handle lets you inspect query execution and request
cancellation. Background operations such as index builds return a
[job](/basics/jobs) handle instead. Use the API for the returned handle to
check status and retrieve results.
## Learn more {#next}
- [Quickstart](/quickstart): run a complete create, insert and search example.
- [Search](/search): choose vector, full-text or hybrid retrieval.
- [Filtering](/search/filtering): narrow results with predicates.
- [SQL](/basics/sql): connect a client and run SQL queries.
- [SDKs](/basics/sdk): choose a language and find its API reference.
docs/web/basics/sdk.mdx
+37 -72
View File
@@ -4,12 +4,6 @@ sidebarTitle: "SDKs"
description: "Choose a LanceDB SDK and find the API reference for your language."
icon: code-compare
keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"]
availability:
oss: available
enterprise: available
summary: >-
Python, TypeScript and Rust support both LanceDB Embedded and LanceDB Enterprise.
Java provides a client for the Enterprise REST API.
---
Use a LanceDB SDK to create tables, add data, build indexes and run searches
@@ -21,16 +15,15 @@ run on the Enterprise service, outside your application.
The [quickstart](/quickstart) includes installation instructions and complete
examples in all three languages. For connection settings, see
[Connect to LanceDB](/basics/connect).
[Connect to LanceDB](/basics/catalog).
## Official SDKs {#supported-sdks}
| SDK | Use it with | API reference |
| --- | --- | --- |
| Python | Python applications, notebooks, pandas and PyArrow | [Python reference](https://lancedb.github.io/lancedb/python/python/) |
| TypeScript | Node.js and TypeScript applications | [TypeScript reference](https://lancedb.github.io/lancedb/js/) |
| Rust | Rust applications using the native library | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) |
| Java | Applications using the Enterprise REST API | [Java reference](https://lancedb.github.io/lancedb/java/java/) |
| SDK | API reference |
| --- | --- |
| Python | [Python reference](https://lancedb.github.io/lancedb/python/python/) |
| TypeScript | [TypeScript reference](https://lancedb.github.io/lancedb/js/) |
| Rust | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) |
Guides show the available language examples together. Use the API reference for
method signatures and options; SDKs do not expose every feature through the same
@@ -41,12 +34,35 @@ For SQL joins, aggregations and window functions, use
languages can use a Flight SQL client. TypeScript's table query API does not
provide a SQL execution method.
## REST API {#rest}
## Lance Namespace SDKs {#lance-namespace-sdks}
LanceDB Enterprise also exposes an HTTP API for catalog and table operations.
Use it when integrating a language or tool without a LanceDB SDK. The
[REST API reference](/api-reference/rest) describes its endpoints and request
formats. For SQL queries, connect through [Flight SQL](/basics/sql).
LanceDB Enterprise's catalog and table operations follow
[Lance Namespace](https://lance.org/format/namespace), an open specification
for describing access to tables in a multimodal lakehouse, including its REST
Namespace Spec. LanceDB Enterprise exposes this as an HTTP API for catalog and
table operations -- the [REST API reference](/api-reference/rest) describes
its endpoints and request formats. Use it when integrating a language or tool
without a LanceDB SDK; for SQL queries, connect through
[Flight SQL](/basics/sql) instead. Any client that implements a Lance
Namespace REST client can integrate with LanceDB this way, not only the
official SDKs above.
| Language | Package | Used by |
| --- | --- | --- |
| Java | `org.lance:lance-namespace-core` | Spark, Flink, Kafka, Trino, Presto; also LanceDB's Java client |
| Python | `lance-namespace` | Core interface and connect functionality |
| Rust | `lance-namespace` | Core interface, in [lance-format/lance](https://github.com/lance-format/lance) |
LanceDB's Java client builds on this Lance Namespace Java core: table creation,
listing and most other catalog operations go through the generated
`LanceNamespace` interface talking to Enterprise's REST Namespace endpoint.
The [real time engine](/tables/realtime-engine) is the one area with its own,
LanceDB-specific Java client on top, since RTE's LSM write path is not part of
the Lance Namespace specification.
See [lance.org/format/namespace](https://lance.org/format/namespace) for the
full spec and its other catalog implementations (Directory, Apache Polaris,
Unity Catalog, Apache Hive Metastore, Apache Iceberg REST Catalog, and more).
## Community SDKs {#community-sdks}
@@ -61,66 +77,15 @@ features, installation instructions and releases before choosing one.
| R | [lancedb](https://github.com/CathalByrneGit/lancedb) |
| Flutter | [flutter_lancedb](https://github.com/Alexcn/flutter_lancedb) |
## Python multiprocessing {#multiprocessing}
<Warning>
Use `spawn` when starting Python worker processes with `multiprocessing`.
LanceDB uses threads internally, and forking a process with active threads can
cause deadlocks. Create a connection inside each worker instead of sharing an
open connection from the parent process.
</Warning>
For PyTorch worker setup and connection factories, see
[PyTorch integration](/training/torch).
## Moving to LanceDB Enterprise {#moving-to-enterprise}
The core table workflow stays familiar: connect, create or open a table, write
records, and query them. Enterprise runs those operations with LanceDB
Distributed and manages background maintenance.
- **Configure the connection.** Replace the local path with your database URI,
and supply the endpoint, region and credentials provided by your administrator.
[Connect to Enterprise](/basics/connect#enterprise) shows these settings.
- **Retrieve query results.** Python tables on Enterprise do not support whole-table
`to_arrow()` or `to_pandas()` calls. Use a query and select the columns and
number of rows you need, for example `table.search().limit(10).to_arrow()`.
- **Handle table listings and namespaces.** Enterprise table listings are paginated.
Use the namespace and pagination options described in
[Namespaces](/management/namespace).
- **Use server-side jobs.** Index builds and feature refreshes can run as
[jobs](/basics/jobs). Each operation's guide explains how to submit work and
track completion.
- **Let Enterprise manage maintenance.** Use
[background maintenance](/maintenance/background-jobs) rather than calling
the embedded table's `optimize()` method.
{/* Draft for the remote option validation release.
- **Remove local storage and cache settings.** `db://` connections reject local
sessions, table cache settings, and per-table storage and write options. Omit
these options so Enterprise manages storage and caching. Connection-level
storage options supported by your Enterprise deployment remain available.
*/}
For SQL connection details, see [SQL](/basics/sql). For the operating model and
deployment options, see [LanceDB Enterprise](/enterprise).
## Shared data across languages {#invariants}
LanceDB uses Arrow schemas across its SDKs. A fixed-size vector column stores
vectors of a consistent dimension; the [data types reference](/sql/data-types)
describes the supported element types and other column types.
Applications in different languages can work with the same tables. Connect to
the same database and namespace, use the appropriate credentials, and account
for [read consistency](/tables/consistency) when another process writes data.
## Related guides {#next}
<CardGroup cols={2}>
<Card title="SQL" icon="table-list" href="/basics/sql">
Query Enterprise tables with SQL from Python or a Flight SQL client.
</Card>
<Card title="PyTorch Data Loader" icon="fire" href="/training/data-loading">
Use LanceDB tables in a DataLoader, including multiple worker processes.
</Card>
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
Use the LanceDB plugin to help a coding agent work with your data.
</Card>
docs/web/basics/sql.mdx
+115 -33
View File
@@ -4,11 +4,6 @@ sidebarTitle: "SQL"
description: "Query LanceDB Enterprise with SQL from Python or an Arrow Flight SQL client."
icon: table-list
keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"]
availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports SQL through Arrow Flight SQL. Use the SDK table APIs for LanceDB Embedded tables. See [Connection settings](#sql-is-remote) to connect.
---
Use SQL in LanceDB Enterprise to filter records, join tables, calculate
@@ -16,26 +11,42 @@ aggregates, and prepare datasets. SQL queries work with the same Lance tables
you create and update through the SDKs, without exporting the data to another
system.
You can run SQL through the Python SDK or an Arrow Flight SQL client. This
guide covers connection settings, a first query, and how SQL fits alongside
the SDK table APIs.
This guide covers the two ways to run SQL, connection settings, a first query,
parameterized queries, and how to read and manage the results.
## Connect a SQL client {#flight-sql}
## Access LanceDB with SQL {#access}
Enterprise exposes an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint. A compatible client sends SQL statements to this endpoint and
receives results as Arrow record batches.
There are two ways to run SQL against LanceDB Enterprise, both against the
same [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, returning results as Arrow record batches:
- **Python:** use `execute_query()` on an Enterprise connection, as shown below.
- **TypeScript:** use a separate Flight SQL client. The LanceDB TypeScript SDK
does not expose a SQL execution method.
- **Other languages and tools:** use a Flight SQL client or a tool with Flight
SQL integration. A LanceDB-specific SDK is not required to submit SQL.
1. **`db.execute_query()`** on the Python SDK's Enterprise connection, shown
below. This is LanceDB's native SQL execution method, and we recommend it
for the best performance -- see [Adaptive query execution](#adaptive).
2. **An Arrow Flight SQL client.** Use this from TypeScript and other
languages, whose LanceDB SDKs do not expose a SQL execution method, or from
any tool with its own Flight SQL integration. A LanceDB-specific SDK is not
required to submit SQL.
The [SQL reference](/sql#connecting) shows a query from Python. See
[SDKs](/basics/sdk) for the LanceDB language clients.
### Adaptive query execution {#adaptive}
A standard Flight SQL client calls `GetFlightInfo` to plan a query and then
`DoGet` to fetch rows -- two round trips every time, even when the server
could stream results back immediately.
`execute_query()` skips that extra round trip when it can. It calls `DoGet`
directly for a query with no parameters, or `DoExchange` when
[binding parameters](#parameters), and sends a
`lancedb-sql-protocol: adaptive-v1` header. The server decides per query
whether to stream results on that same call or defer execution and return a
poll ticket; when it defers, `execute_query()` follows up with
`PollFlightInfo` automatically, with no extra code on your side. A generic
Flight SQL client does not send that header, so it always takes the slower
`GetFlightInfo` path.
### Connection settings {#sql-is-remote}
Get the following values from your Enterprise administrator:
@@ -49,7 +60,7 @@ The Flight SQL endpoint must be enabled and reachable from your client. Pass
it explicitly as `sql_host_override` in Python, as in the example below. The
API and SQL endpoints may use different hosts or ports.
See [Connect to Enterprise](/basics/connect#enterprise) for other connection options.
See [Connect to Enterprise](/basics/catalog#enterprise) for other connection options.
SQL runs in Enterprise. Connections to a local directory or an object-storage
bucket use LanceDB Embedded and cannot execute SQL statements. Use
@@ -59,10 +70,11 @@ SDK table operations to query those tables.
Install the SDK with `pip install lancedb`. Replace the connection placeholders
with your Enterprise settings, including the region and both endpoint URLs.
This example reads an existing table named `my_docs` with `id` and `text`
columns. Change the query to match your table.
This example reads an existing `frames` table, with a `video_id` and a
`caption` column describing each frame. Change the query to match your table.
```python
<CodeGroup>
```python Python icon="python"
import lancedb
db = lancedb.connect(
@@ -73,21 +85,77 @@ db = lancedb.connect(
sql_host_override="https://your-sql-endpoint:443",
)
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
reader = db.execute_query("SELECT frame_id, video_id, caption FROM frames LIMIT 10")
table = reader.read_all()
print(table.to_pylist())
```
</CodeGroup>
In application code, load credentials from environment variables or a secrets
manager. The [connection guide](/basics/connect#enterprise) shows that setup.
manager. The [connection guide](/basics/catalog#enterprise) shows that setup.
`execute_query()` returns a blocking Arrow reader when results start arriving.
The server can stream rows on the first call or return result tickets that the
SDK follows automatically, with the same behavior for parameterized queries.
The example uses `read_all()` to collect the small result in memory. For larger
results, consume the reader in batches. To submit a query and check its status
separately, use `execute_query_async()`. See [Queries](/basics/query) for query
handles, streaming, and cancellation.
`execute_query()` returns a blocking Arrow reader; see
[Adaptive query execution](#adaptive) for how it starts streaming. The example
above uses `read_all()` to collect the small result in memory. For larger
results, or to track and cancel a running query, see
[Read and manage results](#results). To submit a query and check its status
separately, use `execute_query_async()`.
## Parameters in SQL queries {#parameters}
Pass a `parameters` argument to bind values into a query, instead of
formatting them into the SQL string:
```python
import numpy as np
reader = db.execute_query(
"SELECT id FROM docs ORDER BY distance(vector, $1) LIMIT $2",
parameters=[np.array([0.1, 0.2, 0.3], dtype=np.float32), 10],
)
```
Values travel as Arrow, not interpolated SQL text, so a vector binds as a
fixed-size list with no manual formatting and no injection risk. Reference a
parameter with `$1`, `$2`, ... for positional binding, or `$name` for named
binding:
- A list or tuple binds positionally, in order, to `$1`, `$2`, ...
- A dict binds by key to `$name`.
- A one-row `pyarrow.RecordBatch` or `Table` binds its columns, by name and by
position.
- A 1-D NumPy or Arrow array becomes a single fixed-size-list value -- use
this form for a vector argument, as shown above.
<Note>
`parameters` is a LanceDB-specific mechanism, separate from the Flight SQL
protocol's `PREPARE`/bind step, which
[a deployment does not implement](/sql/prepared-statements#parameters). Bind
values with `parameters` from the Python SDK; a generic Flight SQL client has
no equivalent.
</Note>
## Read and manage results {#results}
- **Collect the results** when you want a list, Arrow table or DataFrame that
fits in memory. Use a result limit and select only the columns you need.
- **Read batches** when you want to process a large result incrementally.
Reading batches avoids loading the entire result into application memory.
It does not guarantee that the query can return its first batch immediately.
### Asynchronous clients {#async-client}
An async client lets your application do other work while a request is
waiting. That is separate from whether you collect results or read them in
batches. Python offers synchronous and asynchronous clients; TypeScript uses
async APIs. Check the SDK reference for the methods available in your client.
### Query handles and background jobs {#jobs}
An Enterprise SQL query handle lets you inspect query execution and request
cancellation. Background operations such as index builds return a
[job](/basics/jobs) handle instead. Use the API for the returned handle to
check status and retrieve results.
## Choose SQL or SDK table operations {#choosing}
@@ -100,6 +168,14 @@ You can combine them. For example, create and populate a table with an SDK,
then query it with SQL to prepare a training dataset. In Python, both workflows
can use the same Enterprise connection.
Both interfaces work with the same Lance tables, through the same Arrow
schema. Applications in different languages can work with the same tables:
connect to the same database and namespace, use the appropriate credentials,
and account for [read consistency](/tables/consistency) when another process
writes data. A fixed-size vector column stores vectors of a consistent
dimension; see the [data types reference](/sql/data-types) for the supported
element types and other column types.
### Common SQL operations {#sdk-parity}
Several tasks are available through both interfaces, but the methods and SQL
@@ -152,8 +228,14 @@ and data types. For SDK methods and their arguments, use the
## Next steps {#next}
<CardGroup cols={2}>
<Card title="Queries" icon="magnifying-glass" href="/basics/query">
Read results, track long-running queries, and cancel work.
<Card title="Quickstart" icon="rocket" href="/quickstart">
Run a complete create, insert and search example.
</Card>
<Card title="Search" icon="magnifying-glass" href="/search">
Choose vector, full-text or hybrid retrieval.
</Card>
<Card title="Filtering" icon="filter" href="/search/filtering">
Narrow results with predicates.
</Card>
<Card title="SDKs" icon="code-compare" href="/basics/sdk">
Compare language clients and find their API references.
docs/web/curation/aggregate.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Measure coverage, missing values and class balance before selectin
icon: chart-column
keywords: ["aggregate", "group by", "class balance", "count", "profiling", "distinct"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports SQL aggregation.
---
Use SQL aggregates to summarize a dataset: count records, find missing values,
docs/web/curation/changes.mdx
+93
View File
@@ -0,0 +1,93 @@
---
title: "Scope curation to what changed"
sidebarTitle: "Changes"
description: "Process only the rows added since your last curation run, instead of rescanning the whole table."
icon: clock-rotate-left
keywords: ["changes", "incremental", "cdc", "version", "append-only"]
availability:
embedded: unavailable
enterprise: available
---
A curation task you run repeatedly -- chunking, deduplication, profiling --
does not need to rescan rows it already processed. `CHANGES` reads only the
rows appended to a table since an earlier
[version](/management/table#versioning), so a recurring job can scope itself
to what is new.
## Example data {#example-data}
`curation_batches` gets a batch of new documents appended periodically, with
no updates or deletes between appends:
```sql
CREATE TABLE curation_batches (doc_id BIGINT, source VARCHAR, text VARCHAR);
INSERT INTO curation_batches VALUES
(1, 'web', 'A cat on a chair'),
(2, 'web', 'A dog in a field');
-- note the version after this insert, for example 5
INSERT INTO curation_batches VALUES
(3, 'books', 'A bird in a tree'),
(4, 'books', 'A cat by a window');
-- the table is now at a later version, for example 7
```
## Read only what is new {#read-new}
`CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)` reads the rows
appended after version 5 -- documents 3 and 4 -- without rescanning 1 and 2:
```sql
SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)
```
Record the table's version after each curation run, then start the next
run's `CHANGES` from it. See [Versioning](/management/table#versioning) for
the APIs that read a table's current version.
## Chain it into a curation task {#chain}
`CHANGES` is a relation, like any other table reference, so it composes with
the rest of this page: chunk, dedupe, or join only the new rows, instead of
the whole table. Materializing the result keeps a running, append-only table
of work already done:
```sql
INSERT INTO curated_batches
SELECT doc_id, source, text
FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)
WHERE source = 'books'
```
A [chunker Function](/curation/chunking) or a [`dedupe`](/sql/dql/dedupe)
call works the same way: put the `CHANGES` relation where the source table
would otherwise go, and only the new rows flow through it.
## Strictly append-only {#strict}
The clause is named for what it requires, not just what it does. If anything
other than a pure append happened to the table between the starting version
and the current one -- an update, a delete, a merge -- the query is refused
rather than silently returning an incomplete or wrong result:
```sql
-- refused if any row in curation_batches was updated or deleted after version 5
SELECT * FROM curation_batches CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 5)
```
When that happens, either re-run from a later starting version that is
itself after the non-append write, or read the table in full instead of
incrementally for that run. See
[Changes since a version](/sql/dql/changes#strict) for the complete set of
refused forms.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="CHANGES" icon="clock-rotate-left" href="/sql/dql/changes">
The full SQL syntax, placement rules, and unsupported forms.
</Card>
<Card title="Table" icon="table" href="/management/table#versioning">
How table versions work, and how to find one to start from.
</Card>
</CardGroup>
docs/web/curation/chunking.mdx
+210
View File
@@ -0,0 +1,210 @@
---
title: "Chunk records into multiple rows"
sidebarTitle: "Chunking"
description: "Expand one source row into many derived rows -- documents into passages, videos into clips, images into tiles."
icon: diagram-subtask
keywords: ["chunking", "function", "udf", "lateral", "unnest", "1:N", "passages", "clips", "video", "frames"]
availability:
embedded: unavailable
enterprise: available
---
Every other curation task on this page keeps the row count the same or
smaller: profiling, sampling, joining, deduplicating. Chunking goes the other
way -- one source row becomes many derived rows, each carrying its parent's
columns along with it. Use it to split a document into passages for
embedding, a video into clips, or an image into tiles.
## Define a chunking Function {#function}
A Function that returns a `list` produces more than one value per input row.
A chunker is a Function whose output is a list -- of structs, of strings, of
anything -- registered like any other Function. See
[Create a Function](/curation/create-function) for CPU, GPU and remote-service
Functions, and scalar versus batch calling, if you have not registered one
before.
A function that splits text into fixed-size passages turns a `documents`
table into one row per passage:
**Source: `documents`**
| doc_id | title | text |
| --- | --- | --- |
| 1 | "Intro to AI" | "Machine learning is a method of..." (long text) |
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
from lancedb.functions import udf
@udf(
output_schema=pa.list_(
pa.struct([
pa.field("chunk_index", pa.int64(), nullable=False),
pa.field("chunk_text", pa.utf8(), nullable=False),
])
),
)
def split_into_passages(text: str) -> list[dict]:
size = 200
return [
{"chunk_index": i, "chunk_text": text[i : i + size]}
for i in range(0, len(text), size)
]
version = db.create_function_async(split_into_passages).wait()
```
</CodeGroup>
Registration details, environment and dependency declarations, and how a
version stays immutable once rows are filled from it, are the same for a
chunker as for any other Function. The same function can be registered from
SQL with `CREATE FUNCTION`:
<CodeGroup>
```sql SQL
CREATE FUNCTION split_into_passages(text utf8)
RETURNS list<struct<chunk_index: int64, chunk_text: utf8>>
LANGUAGE python
AS 'def split_into_passages(text): size = 200; return [{"chunk_index": i, "chunk_text": text[i:i+size]} for i in range(0, len(text), size)]'
```
</CodeGroup>
## Explode it in a query {#explode}
Call the function in the `FROM` clause, after the source table, with an
alias: a Function there yields one row per element of the list it returns,
with every other column repeated. Reference the function's output fields
through the alias:
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("""
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
""")
print(reader.read_all().to_pylist())
```
</CodeGroup>
A document with five 200-character passages becomes five output rows, each
carrying `doc_id` and `title` from its parent. This is the fastest way to
check a chunker before committing to a materialized view: run it over a few
rows and inspect the output shape.
## Video into frames {#video}
Chunking a blob column is the same shape, with the chunker producing
timestamps rather than text spans, and a second Function pulling the actual
frame for each one. `videos` holds raw clips in a
[Blob v2](/management/table#managed-blobs) column, with each row's duration
already known:
**Source: `videos`**
| video_id | video | duration_seconds |
| --- | --- | --- |
| "clip-1" | (Blob v2 bytes) | 12.0 |
A chunker that samples one timestamp per second needs no blob access at all
-- it only looks at `duration_seconds`:
<CodeGroup>
```python Python icon="python"
@udf(
output_schema=pa.list_(
pa.struct([
pa.field("frame_index", pa.int64(), nullable=False),
pa.field("ts", pa.float64(), nullable=False),
])
),
)
def sample_frame_timestamps(duration_seconds: float) -> list[dict]:
every = 1.0 # one frame per second
count = int(duration_seconds // every)
return [{"frame_index": i, "ts": i * every} for i in range(count)]
version = db.create_function_async(sample_frame_timestamps).wait()
```
</CodeGroup>
A second Function reads the blob -- incrementally, through a `BlobReader`,
rather than loading the whole clip -- and returns one JPEG per call:
<CodeGroup>
```python Python icon="python"
import lancedb
from datetime import timedelta
from lancedb import BlobReader
@udf(
name="extract_frame",
gpu=True,
pip=["av==13.1.0", "pillow==11.0.0"],
input_schema=pa.schema([
lancedb.blob("video"),
pa.field("ts", pa.float64(), nullable=False),
]),
output_schema=lancedb.blob("jpeg", nullable=False),
)
def extract_frame(video: BlobReader, ts: float) -> bytes:
... # decode the frame at ts, encode as JPEG, return its bytes
extract_version = db.create_function_async(extract_frame).wait(timeout=timedelta(minutes=15))
```
</CodeGroup>
Compose both in one query: explode `videos` into one row per sampled
timestamp, then call `extract_frame` on each row to produce its image:
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("""
SELECT video_id, f.frame_index, f.ts, extract_frame(video, f.ts) AS jpeg
FROM videos, sample_frame_timestamps(duration_seconds) AS f
""")
```
</CodeGroup>
A 12-second clip becomes 12 output rows, each with its own extracted JPEG. A
Function in `FROM` position yields one row per list element the same way
regardless of what the list holds; whether the second step produces a blob,
a plain value, or another struct does not change the explosion.
## Persist it as a materialized view {#persist}
Wrap either query in [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view)
to keep a real table that stays current as the source changes, instead of
recomputing the explosion on every query:
<CodeGroup>
```sql SQL
CREATE MATERIALIZED VIEW chunks AS
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
```
</CodeGroup>
**Derived: `chunks`** (1:N expansion)
| doc_id | title | chunk_index | chunk_text |
| --- | --- | --- | --- |
| 1 | "Intro to AI" | 0 | "Machine learning is a method..." |
| 1 | "Intro to AI" | 1 | "...of teaching computers..." |
The view refreshes like any other materialized view: new source rows are
chunked and inserted, deleted source rows drop their children, and an
updated row's old children are replaced by freshly chunked ones on the next
[refresh](/feature-engineering/refresh-column). A single Function call or
`UNNEST(column)` is the only lateral source a materialized view can carry --
not both, and not combined with `GROUP BY` -- so keep aggregation or further
curation in a second step over the chunked table.
## Chunk, then curate {#next}
A freshly chunked table is a normal table: [dedupe](/curation/dedupe) it to
collapse near-identical chunks, [sample](/curation/sampling) it for a quick
look, or [profile it](/curation/aggregate) to check how many chunks each
source row produced. Continue with
[choosing a curation task](/curation#questions).
docs/web/curation/create-function.mdx
+213
View File
@@ -0,0 +1,213 @@
---
title: "Create a Function"
sidebarTitle: "Create Function"
description: "Register Python as a column: CPU, GPU, and remote-service Functions, called once per row or once per batch."
icon: code
keywords: ["function", "udf", "cpu", "gpu", "batch", "scalar", "blob", "remote api", "embedding", "rate limiter"]
availability:
embedded: unavailable
enterprise: available
---
Some curation steps cannot be written as a SQL expression: an embedding
model, an image decode, a call to an external service. A Function is a
Python callable registered with the database, versioned, and then applied as
a column or called in a query -- the code runs where the data is, not pulled
into your process first. This page covers how to shape one; later curation
tasks such as [Chunking](/curation/chunking) assume you have read it.
## Scalar: once per row {#scalar}
The plain form takes one row's values and returns one result. Types are
inferred from the annotations:
<CodeGroup>
```python Python icon="python"
from lancedb import udf
@udf(pip=["numpy==2.2.0"])
def score(value: float) -> float:
return value * 2
version = db.create_function_async(score).wait()
table.add_columns({"doubled": version(value=col("value"))})
```
</CodeGroup>
`pip` declares the worker's dependencies; they are part of the function's
immutable version, so changing one produces a new version rather than
altering rows a previous version already filled.
## Batch: once per call, many rows at a time {#batch}
A function is called once per batch instead of once per row when its
parameters are annotated `pyarrow.Array`, or when it takes a single
`pyarrow.RecordBatch` parameter to read more than one column together. The
same doubling logic, vectorized over a whole batch at once:
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
import pyarrow.compute as pc
from lancedb import udf
@udf(pip=["numpy==2.2.0"])
def score_batch(value: pa.Array) -> pa.Array:
return pc.multiply(value, 2)
```
</CodeGroup>
Batching amortizes per-call overhead across many rows, which matters most
once a function also does something with real fixed cost per call -- loading
a model, opening a connection, making a network round trip.
## Stateful Functions: load once, call many times {#stateful}
A `@udf`-decorated **class** is the same callable with state. Each remote
instance calls `__init__` once, calls `__call__` for every row or batch it
processes, and calls `close()`, if defined, once when it retires:
<CodeGroup>
```python Python icon="python"
from lancedb import udf
@udf
class scale:
def __init__(self, factor: float, offset: float = 0.0):
self.factor = factor
self.offset = offset
def __call__(self, value: float) -> float:
return value * self.factor + self.offset
```
</CodeGroup>
Constructor parameters are **initialization fields**, bound once per binding
rather than read per row -- mixed with per-row columns in the same call:
`version(value=col("value"), factor=2.0)`. Load models, open clients, and
build rate limiters in `__init__`; every batch the instance serves then
reuses them, since an instance's state is shared across its own batches but
never across other instances or workers. This is the pattern GPU inference
and remote-API Functions both build on below.
## GPU: batch inference, loaded once {#gpu}
`gpu=True` requires a GPU-capable worker for every execution. Combined with
the batch and stateful forms above, a realistic embedding model loads once
in `__init__` and runs one forward pass per batch, not one per row:
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
from lancedb import udf
@udf(gpu=True, pip=["torch==2.12.0", "sentence-transformers==5.7.0"])
class embed_text:
def __init__(self, model_name: str = "all-MiniLM-L6-v2"):
import torch
from sentence_transformers import SentenceTransformer
device = "cuda" if torch.cuda.is_available() else "cpu"
self.model = SentenceTransformer(model_name, device=device)
def __call__(self, text: pa.Array) -> pa.Array:
texts = [t or "" for t in text.to_pylist()]
embeddings = self.model.encode(texts)
return pa.array(embeddings.tolist(), type=pa.list_(pa.float32(), list_size=384))
version = db.create_function_async(embed_text).wait()
table.add_columns({"embedding": version(text=col("caption"))})
```
</CodeGroup>
The model loads once per worker instance, in `__init__`; `__call__` only
encodes the batch it was handed. Registering the same logic as a plain
function instead would reload the model on every single row.
## Calling a remote service {#remote}
A function that calls an external API needs a credential without putting it
in the function's source. A Secret is a database-scoped named credential:
`db.create_secret` stores the value once, and binding it at registration time
puts it in the environment variable the function reads, without ever
returning it from any API or recording it in the function's own definition.
Combine it with the stateful form so the HTTP client -- and any rate limiter
around it -- is built once, not reopened on every row:
<CodeGroup>
```python Python icon="python"
import os
from lancedb import EnvVarSecret, udf
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
@udf
class embed_remote:
def __init__(self, model: str = "text-embedding-3-small"):
import os
from openai import OpenAI
self.client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
self.model = model
def __call__(self, text: str) -> list[float]:
response = self.client.embeddings.create(input=text, model=self.model)
return response.data[0].embedding
version = db.create_function_async(
embed_remote,
secrets=[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")],
).wait()
```
</CodeGroup>
A mistyped Secret name is caught at registration; a mistyped environment
variable name only surfaces on the first rows, since nothing records which
variables a function reads. There is no built-in retry or per-row
error-skip policy -- a function that calls a flaky service should handle
timeouts and retries itself, inside `__call__`.
## Blob v2 input and output {#blob}
A Function can bind a [Blob v2 column](/management/table#managed-blobs) and
return another one, including a blob nested inside a struct field, not only
at the top level. In row mode, a blob parameter arrives as Python `bytes`
(or a `BlobReader` for incremental, seekable access, so a function never
has to materialize a payload it only partially needs):
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
import lancedb
from lancedb import BlobReader, udf
@udf(
gpu=True,
pip=["av==13.1.0", "pillow==11.0.0"],
input_schema=pa.schema([
lancedb.blob("video"),
pa.field("ts", pa.float64(), nullable=False),
]),
output_schema=lancedb.blob("jpeg", nullable=False),
)
def extract_frame(video: BlobReader, ts: float) -> bytes:
... # decode the frame at ts, encode as JPEG, return its bytes
```
</CodeGroup>
`output_schema=lancedb.blob(...)` is what makes the result Blob v2 rather
than plain Arrow `binary` -- returning `bytes` with no explicit blob output
schema still holds the right bytes, but skips the Managed storage path and
`fetch_blobs`. Declare it explicitly on every function whose output should
be a blob.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Chunk records into multiple rows" icon="diagram-subtask" href="/curation/chunking">
A Function whose output is a list, exploded into one row per element.
</Card>
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Filling the column a Function is bound to, and re-filling it as inputs change.
</Card>
</CardGroup>
docs/web/curation/dedupe.mdx
+142
View File
@@ -0,0 +1,142 @@
---
title: "Deduplicate near-duplicate records"
sidebarTitle: "Dedupe"
description: "Collapse records that are only approximately alike, judged by embedding distance, without a pairwise scan."
icon: clone
keywords: ["dedupe", "duplicate_pairs", "deduplication", "near-duplicate", "embedding", "distance_threshold"]
availability:
embedded: unavailable
enterprise: available
---
[Ranking and deduplication](/curation/windowing#dedup) removes records by an
explicit rule, such as keeping the latest version of a known key. This page
covers the other kind of duplicate: records with no shared key that are still
nearly identical, judged by distance between their embeddings -- a stopped
camera holding one pose for a few seconds, or two scraped pages with the same
paragraph reworded. `dedupe` and `duplicate_pairs` find these directly over a
vector index, without a pairwise scan.
## Example data {#example-data}
Connect with an [Enterprise connection](/basics/catalog#enterprise). A small
fixture keeps the distances easy to check by hand.
<Accordion title="Create the example table">
`curation_embeddings` has eight records. Rows 10 and 20 are near-identical;
rows 30, 40 and 50 form one looser cluster; the rest are distinct.
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
from lancedb.index import IvfFlat
schema = pa.schema([
pa.field("id", pa.int64(), nullable=False),
pa.field("caption", pa.utf8()),
pa.field("embedding", pa.list_(pa.float32(), 2)),
])
table = db.create_table("curation_embeddings", [
{"id": 10, "caption": "a cat on a chair", "embedding": [0.0, 0.0]},
{"id": 20, "caption": "a cat on the chair", "embedding": [0.0, 0.03]},
{"id": 30, "caption": "a dog in a field", "embedding": [5.0, 5.0]},
{"id": 40, "caption": "a dog in the field", "embedding": [5.0, 5.05]},
{"id": 50, "caption": "a dog out in a field", "embedding": [5.0, 5.10]},
{"id": 60, "caption": "a bird in a tree", "embedding": [10.0, 10.0]},
{"id": 70, "caption": "a boat on the water", "embedding": [20.0, 20.0]},
{"id": 80, "caption": "a car on the road", "embedding": [30.0, 30.0]},
], schema=schema)
table.create_index("embedding", config=IvfFlat(num_partitions=1, distance_type="l2"))
table.wait_for_index(["embedding_idx"])
```
</CodeGroup>
`dedupe` and `duplicate_pairs` read the vector index, so the column needs one
before either function runs; a query against unindexed fragments is rejected
rather than silently returning stale or wrong results.
</Accordion>
## Collapse near-duplicates {#dedupe}
`dedupe` returns one representative row per cluster of records within
`distance_threshold` of each other -- rows 10/20 collapse to one, 30/40/50
collapse to one, and the four distinct rows pass through unchanged:
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("""
SELECT id, caption FROM dedupe('curation_embeddings', 'embedding', distance_threshold => 0.06)
ORDER BY id
""")
print(reader.read_all().to_pylist())
# [{'id': 10, 'caption': 'a cat on a chair'}, {'id': 30, 'caption': 'a dog in a field'},
# {'id': 60, ...}, {'id': 70, ...}, {'id': 80, ...}]
```
</CodeGroup>
Every other column survives unchanged, since `dedupe` returns full rows, not
just which ones matched. Clustering is transitive: 30 and 40 are within
`distance_threshold` of each other, and so are 40 and 50, so all three
collapse into one representative even though 30 and 50 alone sit outside the
threshold. Pass `resolution => 'direct_representative'`
for a simpler, non-transitive selection instead, if a chained cluster like that
should not merge into one representative.
Which row survives as the representative is not guaranteed to stay the same
across appends: a later row that bridges two previously separate clusters can
change which id represents the merged cluster. Do not treat a representative's
id as a stable identifier for its cluster.
## Inspect the pairs instead {#pairs}
`duplicate_pairs` returns the matching edges themselves, without collapsing
anything -- useful for review, or for a threshold that is still being tuned:
<CodeGroup>
```python Python icon="python"
db.execute_query("""
SELECT id_a, id_b, distance FROM duplicate_pairs(
'curation_embeddings', 'embedding', distance_threshold => 0.06, id_column => 'id'
)
ORDER BY id_a
""")
```
</CodeGroup>
This returns the pairs `(10, 20)`, `(30, 40)` and `(40, 50)` -- note 30 and 50
are not a direct pair here (they sit `0.10` apart, outside the threshold),
even though `dedupe` merges all three transitively above.
## Keep it current with a materialized view {#materialized}
Wrap either call in a [materialized view](/management/materialized-view) so
the deduplicated set stays current as new records arrive, instead of
recomputing it from scratch on every curation pass:
<CodeGroup>
```python Python icon="python"
db.execute_query("""
CREATE MATERIALIZED VIEW curation_embeddings_deduped AS
SELECT * FROM dedupe('curation_embeddings', 'embedding', distance_threshold => 0.06)
""")
```
</CodeGroup>
See [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
for manual and automatic refresh options to keep it current as new records land.
## At production scale {#production}
[LeRobotDataset](/integrations/lerobotdataset) runs the same `dedupe`
call over 27.6 million real embedding rows to remove near-static frames from a
robotics training set in under 30 seconds -- the mechanism does not change
with scale, only the table it runs against.
## Continue with the curated data {#next}
[Profile the result](/curation/aggregate) to check how much a threshold
removes before committing to it, or go back to
[choosing a curation task](/curation#questions).
docs/web/curation/index.mdx
+5 -1
View File
@@ -17,10 +17,14 @@ records belong, and save the selected data or the rules used to produce it.
| Task | What it helps you decide |
| --- | --- |
| [Create a Function](/curation/create-function) | How to run your own Python -- on CPU, on GPU, or against a remote service -- as part of SQL. |
| [Chunk records](/curation/chunking) | How to split one record into several -- passages, clips, or tiles. |
| [Scope to what changed](/curation/changes) | How to process only the rows added since your last run. |
| [Profile a dataset](/curation/aggregate) | Whether coverage, missing values and class balance suit the task. |
| [Sample records](/curation/sampling) | Which subset to inspect or use for an experiment. |
| [Join datasets](/curation/join) | How to attach labels, metadata or scores stored elsewhere. |
| [Rank and deduplicate](/curation/windowing) | Which records to keep when several describe the same item. |
| [Rank and deduplicate](/curation/windowing) | Which records to keep when several share an explicit key. |
| [Deduplicate near-duplicates](/curation/dedupe) | Which records to collapse when they only look alike, with no shared key. |
The analytical examples in this section use **Enterprise SQL**. Connect with
the [SQL guide](/basics/sql). For sampling through the Python SDK in LanceDB Embedded or
docs/web/curation/join.mdx
+33 -3
View File
@@ -5,10 +5,8 @@ description: "Combine records with labels or metadata, and check for missing or
icon: link
keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports SQL joins.
---
A SQL join combines records from two tables using a shared key. Use joins to
@@ -127,6 +125,38 @@ Use [Explain a query plan](/search/explain) to see how the query reads and
combines data. Check the plan to see whether the query uses indexes and how it performs the
join. An index on a key does not guarantee that the join will use it.
### Defer wide columns until after the join {#late-materialization}
When one side of a join carries a BlobV2 column, an embedding, or both, join
on a narrow key first -- `row_id`, `_rowid`, or another selective column --
and project the wide columns only in the final select list, rather than
selecting them alongside the join condition. The planner can then defer
reading those columns until after the join has already narrowed down which
rows survive, instead of reading them for every candidate row.
`frames` is wide: a BlobV2 `image` and a vector `embedding` alongside its
key. `selected` is narrow: the frame IDs an earlier curation step already
kept, such as a sample or a dedupe result. Join on the key, project the wide
columns last:
```sql
SELECT s.frame_id, f.image, f.embedding
FROM selected AS s
JOIN frames AS f ON f.frame_id = s.frame_id
```
`selected` narrows the row set before `image` or `embedding` is ever touched
on the `frames` side, rather than reading every row's blob and vector just
to discard most of them after the join.
This is query planning the optimizer may or may not apply to a given query,
not a guarantee from writing the join this way. Confirm it per query with
[`EXPLAIN`](/search/explain): a plan containing `RemoteTake` deferred the
wide read; a plan without it read the wide columns eagerly, even for a query
shaped exactly like the one above. See
[Late materialization for wide columns](/sql/dql/join#late-materialization)
in the SQL reference for the underlying mechanism.
## Save the result or continue curating {#reference}
Keep the query for reuse or [save the selected data](/curation#next).
docs/web/curation/sampling.mdx
+4 -6
View File
@@ -1,14 +1,12 @@
---
title: "SQL sampling"
sidebarTitle: "SQL Sampling"
title: "Sampling"
sidebarTitle: "Sampling"
description: "Choose a sample for exploration or evaluation and preserve the selected records when reproducibility matters."
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports SQL `TABLESAMPLE`. For sampling in LanceDB Embedded and LanceDB Enterprise through the Python SDK, see [Sample without SQL](#training).
---
import { PySampleDataset } from '/snippets/curation_sampling.mdx';
@@ -163,7 +161,7 @@ promise the same IDs across SDK versions or changes to the source dataset.
</Accordion>
For shuffled training reads, worker configuration and data loaders, see
[Loading data for model training](/training/data-loading).
[PyTorch Data Loader](/training/data-loading).
## Related guides {#reference}
docs/web/curation/windowing.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Choose records with explicit ranking rules and deterministic tie h
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports SQL window functions.
---
Window functions calculate values across related rows while keeping each row
docs/web/deployment/storage.mdx
+5 -12
View File
@@ -1,23 +1,16 @@
---
title: "Embedded storage configuration"
sidebarTitle: "Embedded Storage"
description: "Choose and configure local disk or object storage for tables accessed by LanceDB Embedded."
title: "Storage configuration"
sidebarTitle: "Storage Configuration"
description: "Choose and configure local disk or object storage for your tables."
icon: database
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
availability:
oss: available
enterprise: unavailable
summary: >-
The embedded library chooses and configures its storage backend from the
connection URI and `storage_options`. In LanceDB Enterprise, storage is part of
the deployment's own configuration instead.
---
This page is about LanceDB Embedded, where your connection chooses and
configures the storage backend. A LanceDB Enterprise deployment sets up its own
storage when it is deployed, so a `db://` connection passes none of these
options except an Azure storage account name. See
[Deployment](/enterprise/deployment) for how a deployment is set up.
[LanceDB Enterprise](/enterprise) for how a deployment is set up.
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
@@ -124,7 +117,7 @@ import {
<Note>
**LanceDB Enterprise storage configuration**
In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow.
In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Kubernetes deployment guide](/enterprise/deployment/kubernetes) for the Enterprise flow.
</Note>
### Object stores {#object-stores}
docs/web/docs.json
+76 -29
View File
@@ -27,6 +27,42 @@
},
"favicon": "/static/favicon.ico",
"redirects": [
{
"source": "/architecture",
"destination": "/#architecture"
},
{
"source": "/enterprise/architecture",
"destination": "/#architecture"
},
{
"source": "/enterprise/deployment/azure",
"destination": "/enterprise/deployment/kubernetes"
},
{
"source": "/basics/query",
"destination": "/basics/sql#results"
},
{
"source": "/basics/offerings",
"destination": "/#embedded-and-enterprise"
},
{
"source": "/enterprise/deployment",
"destination": "/enterprise#deployment"
},
{
"source": "/feature-engineering/data-evolution",
"destination": "/feature-engineering/add-computed-column"
},
{
"source": "/training/torch",
"destination": "/training/data-loading"
},
{
"source": "/basics/lakehouse",
"destination": "/"
},
{
"source": "/tables/update",
"destination": "/ingestion/update"
@@ -209,7 +245,7 @@
},
{
"source": "/sql/refresh",
"destination": "/sql/dml/refresh"
"destination": "/sql/dml/refresh-column"
},
{
"source": "/sql/show",
@@ -245,13 +281,11 @@
"group": "Getting Started",
"pages": [
"index",
"basics/offerings",
"basics/connect",
"quickstart",
"architecture",
"basics/sdk",
"basics/sql",
"basics/agentic-skills"
"basics/agentic-skills",
"deployment/support"
]
},
{
@@ -259,9 +293,8 @@
"pages": [
"basics/lance",
"basics/catalog",
"basics/lakehouse",
"basics/query",
"basics/jobs",
"tables/realtime-engine",
"tables/consistency",
"basics/guarantees",
"basics/limits",
@@ -276,7 +309,8 @@
"ingestion/insert",
"ingestion/merge-insert",
"ingestion/update",
"ingestion/delete"
"ingestion/delete",
"ingestion/realtime-write"
]
},
{
@@ -288,35 +322,42 @@
"search/hybrid-search",
"search/filtering",
"reranking/index",
"search/explain"
"search/explain",
"search/realtime-read",
"enterprise/distributed-search"
]
},
{
"group": "Data Curation",
"pages": [
"curation/index",
"curation/create-function",
"curation/chunking",
"curation/changes",
"curation/aggregate",
"curation/sampling",
"curation/join",
"curation/windowing"
"curation/windowing",
"curation/dedupe"
]
},
{
"group": "Feature Engineering",
"pages": [
"feature-engineering/index",
"feature-engineering/data-evolution",
"feature-engineering/add-computed-column",
"feature-engineering/refresh-column",
"feature-engineering/materialized-views"
"feature-engineering/view",
"feature-engineering/materialized-views",
"feature-engineering/refresh-materialized-view"
]
},
{
"group": "Training",
"pages": [
"training/why-lancedb",
"training/permutation",
"training/data-loading",
"training/torch"
"training/distributed-training"
]
},
{
@@ -327,19 +368,24 @@
"management/namespace",
"management/table",
"tables/branching",
"tables/realtime-engine",
"management/materialized-view",
"management/view"
"management/view",
"management/materialized-view"
]
},
{
"group": "Maintenance",
"pages": [
"indexing/index",
"indexing/vector-index",
"indexing/fts-index",
"indexing/scalar-index",
"indexing/reindexing",
{
"group": "Indexing",
"icon": "layer-group",
"pages": [
"indexing/index",
"indexing/vector-index",
"indexing/fts-index",
"indexing/scalar-index",
"indexing/reindexing"
]
},
"maintenance/compaction",
"maintenance/cleanup",
"maintenance/background-jobs"
@@ -349,14 +395,10 @@
"group": "Deployment",
"pages": [
"enterprise/index",
"enterprise/architecture",
"enterprise/distributed-search",
"enterprise/deployment/index",
"deployment/storage",
"deployment/monitoring",
"enterprise/deployment/azure",
"enterprise/benchmarks",
"deployment/support"
"enterprise/deployment/kubernetes",
"enterprise/benchmarks"
]
},
{
@@ -375,6 +417,7 @@
{
"group": "DDL",
"pages": [
"sql/ddl/database",
"sql/ddl/namespace",
"sql/ddl/table",
"sql/ddl/view",
@@ -391,7 +434,8 @@
"sql/dml/update",
"sql/dml/delete",
"sql/dml/copy",
"sql/dml/refresh"
"sql/dml/refresh-column",
"sql/dml/refresh-materialized-view"
],
"icon": "database"
},
@@ -404,7 +448,10 @@
"sql/dql/aggregation",
"sql/dql/window",
"sql/dql/sampling",
"sql/dql/changes",
"sql/dql/fts",
"sql/dql/dedupe",
"sql/dql/duplicate-pairs",
"sql/dql/explain"
],
"icon": "filter"
docs/web/embedding/index.mdx
+1 -5
View File
@@ -4,12 +4,8 @@ sidebarTitle: "Managing Embeddings"
description: "Use the embedding API in LanceDB: registry, functions, schemas, and multi-language SDK support."
icon: "bars"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Embedding functions run in your process, whether the table is local or on a
deployment, which stores their configuration but never runs them. Python
and TypeScript also embed query text; in Rust you embed queries yourself.
---
import {
docs/web/enterprise/architecture.mdx
-23
View File
@@ -1,23 +0,0 @@
---
title: "Enterprise architecture"
sidebarTitle: "Architecture"
description: "The tiers a deployment runs, and how they scale independently."
icon: cubes
keywords: ["architecture", "control plane", "data plane", "indexer"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
A deployment separates compute from storage so each scales on its own: table
data and indexes live in object storage, while query serving and background work
run on machines that can be replaced without any of them owning the data.
[Architecture](/architecture) covers this alongside the embedded form, which is
the comparison most readers want first.
See [Distributed search](/enterprise/distributed-search) for how Enterprise
distributes index execution and table-data reads.
docs/web/enterprise/authentication.mdx
+1 -5
View File
@@ -5,12 +5,8 @@ description: "Authentication modes for LanceDB Enterprise: API keys and OAuth cr
icon: key
keywords: ["authentication", "oauth", "api key", "authorization code", "device code", "azure managed identity", "client credentials"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Requests to a deployment carry an API key or an OAuth token, whichever it is
configured to accept. LanceDB Embedded has no server to sign in to; see
[Architecture](/architecture).
---
A deployment authenticates each request over a `db://` connection before
docs/web/enterprise/benchmarks.mdx
+1 -1
View File
@@ -10,7 +10,7 @@ keywords: ["benchmarks", "performance", "throughput", "latency"]
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
[Architecture](/#architecture) for what differs between the two.
Benchmark results are specific to a deployment's shape (cluster size, storage
class, dataset), so they are published with LanceDB Enterprise.
docs/web/enterprise/deployment/azure.mdx
-17
View File
@@ -1,17 +0,0 @@
---
title: "Azure deployment guide"
sidebarTitle: "Azure"
description: "Deploying LanceDB Enterprise on Azure."
icon: cloud
keywords: ["azure", "deployment", "aks"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
The Azure deployment procedure is published with LanceDB Enterprise. See
[Deployment](/enterprise/deployment) for the general shape, and
[contact us](https://www.lancedb.com/contact) to get started.
docs/web/enterprise/deployment/index.mdx
-20
View File
@@ -1,20 +0,0 @@
---
title: "Deployment guide"
sidebarTitle: "Deployment Models"
description: "Standing up a LanceDB Enterprise deployment."
icon: server
keywords: ["deployment", "install", "cluster", "kubernetes"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
Deployment is specific to your cloud and account, so the procedure is published
with LanceDB Enterprise. [Contact us](https://www.lancedb.com/contact) to get
started.
LanceDB Embedded has nothing to deploy: it is a library your process links
against. See [Architecture](/architecture#embedded).
docs/web/enterprise/deployment/kubernetes.mdx
+17
View File
@@ -0,0 +1,17 @@
---
title: "Kubernetes deployment guide"
sidebarTitle: "Kubernetes"
description: "Deploying LanceDB Enterprise on Kubernetes."
icon: cloud
keywords: ["kubernetes", "k8s", "aks", "deployment", "helm"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/#architecture) for what differs between the two.
The Kubernetes deployment procedure is published with LanceDB Enterprise. See
[LanceDB Enterprise](/enterprise#deployment) for the general shape, and
[contact us](https://www.lancedb.com/contact) to get started.
docs/web/enterprise/distributed-search.mdx
+3 -6
View File
@@ -4,17 +4,14 @@ sidebarTitle: "Distributed search"
description: "How LanceDB Enterprise distributes index execution and table-data reads."
icon: "network-wired"
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
LanceDB Enterprise distributes index execution and table-data reads across
executors. LanceDB Embedded executes searches in the application process.
---
Distributed search spreads index execution, table-data reads, and caching across
multiple serving machines in LanceDB Enterprise. Query nodes plan queries and
combine results. Executors cache index structures and table data and execute
the distributed work. See [Enterprise architecture](/enterprise/architecture).
combine results. Plan executors cache index structures and table data and execute
the remote work. See [Architecture](/#architecture).
## Embedded search {#embedded-search}
docs/web/enterprise/index.mdx
+11 -2
View File
@@ -10,7 +10,7 @@ keywords: ["enterprise", "deployment", "cluster"]
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
[Architecture](/#architecture) for what differs between the two.
LanceDB Enterprise runs the lakehouse as a service: a control plane for
configuration, identity and policy; a data plane serving reads and writes; and
@@ -30,4 +30,13 @@ Draft for the Functions release. Add this row to the table above, after SQL:
| [Functions](/management/function) | Python running on workers |
*/}
See [Architecture](/architecture) for the two shapes side by side.
See [Architecture](/#architecture) for the two shapes side by side.
## Getting a deployment {#deployment}
Deployment is specific to your cloud and account, so the procedure is
published with LanceDB Enterprise rather than here.
[Contact us](https://www.lancedb.com/contact) to get started; see the
[Kubernetes deployment guide](/enterprise/deployment/kubernetes) for a worked example.
Embedded LanceDB has nothing to deploy: it is a library your process links
against -- see [LanceDB Embedded](/#1-lancedb-oss).
docs/web/enterprise/security.mdx
+1 -1
View File
@@ -10,7 +10,7 @@ keywords: ["compliance", "security", "soc2", "encryption", "audit"]
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it. See
[Architecture](/architecture) for what differs between the two.
[Architecture](/#architecture) for what differs between the two.
Compliance posture (certifications, data residency, encryption at rest and in
transit, audit logging) is a property of a deployment, so it is documented
docs/web/feature-engineering/add-computed-column.mdx
+109 -8
View File
@@ -1,18 +1,33 @@
---
title: "Add a computed column"
sidebarTitle: "Computed Columns"
title: "Create a computed column"
sidebarTitle: "Create Computed Column"
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
icon: table-columns
keywords: ["computed columns", "add_columns", "transforms", "schema"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Local computed columns require a source build with the experimental-functions Cargo feature. Enterprise requires the deployment to enable them.
---
A computed column is declared with an expression and starts with no values.
Declaring it is cheap on any table size, because nothing is evaluated until
import {
PySchemaAddSetup,
TsSchemaAddSetup,
RsSchemaAddSetup,
PyAddColumnsCalculated,
TsAddColumnsCalculated,
RsAddColumnsCalculated,
} from '/snippets/tables.mdx';
Lance can write a new column without rewriting the columns already in the
table. That matters most on a table that also holds large images, video or
embeddings, where rewriting everything just to add one small derived field
would be wasteful. See [Lance format](/basics/lance) for the storage model
this relies on.
A computed column is how LanceDB formalizes that capability for feature
engineering specifically: instead of writing the derived values immediately,
it is declared with an expression and starts with no values. Declaring it is
cheap on any table size, because nothing is evaluated until
[a refresh](/feature-engineering/refresh-column) asks for it.
<Note>
@@ -56,6 +71,69 @@ including that you can travel back to before the column existed.
`computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is
large enough that you would rather control when the work happens.
Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` alongside
`computed` to have the server keep the column refreshed instead of calling
[`refresh_column`](/feature-engineering/refresh-column) yourself:
```python
table.add_columns(
computed={"doubled": "x * 2"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
A local table stores these settings but does not act on them -- automatic
scheduling runs on LanceDB Cloud or Enterprise. See
[Automatic refresh](/feature-engineering/refresh-column#automatic) for the
full behavior, including how to change the setting later.
An eager `transforms` call looks like this, starting from a small product
table:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PySchemaAddSetup}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsSchemaAddSetup}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsSchemaAddSetup}
</CodeBlock>
</CodeGroup>
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyAddColumnsCalculated}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsAddColumnsCalculated}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsAddColumnsCalculated}
</CodeBlock>
</CodeGroup>
For feature engineering on large existing tables, group related derived
features into one `add_columns` call instead of running many separate
writes: this creates one new table version for the schema change and
computes every new column from the existing rows in it, rather than growing
the table's version history with many small updates.
<Info>
`add_columns` does not accept Python callables, batch UDFs, or PyArrow
`RecordBatch` iterators for populating new columns, whether through
`transforms` or `computed`. New column values must be defined with SQL
expressions, or added as NULL columns from an Arrow field or schema. If your
transformation cannot be expressed in SQL, compute the values outside
`add_columns` before writing them back through another workflow.
</Info>
## The expression {#expressions}
The expression is SQL, evaluated over the row. It can reference any column in
@@ -111,12 +189,35 @@ image decode -- the column is filled by a registered
[function](/management/function) instead of an expression. Declaring it looks
the same; what differs is what fills it. It needs a deployment with Functions
enabled.
The call passed to `add_columns` is the tell: a SQL expression in `computed`
is a string, keyed by the column name --
<CodeGroup>
```python Python icon="python"
table.add_columns(computed={"caption_length": "length(caption)"})
```
</CodeGroup>
-- while a Function application is the registered version itself, called with
its parameters bound to source columns through `col(...)`, keyed the same way
but passed positionally rather than through `computed`:
<CodeGroup>
```python Python icon="python"
table.add_columns({"caption": caption_version(jpeg=col("jpeg"))})
```
</CodeGroup>
Both accept the same `auto_refresh_enabled` and `auto_refresh_interval_seconds`
options, and both refresh and toggle the same way; see
[Refresh a computed column](/feature-engineering/refresh-column#automatic).
*/}
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Fill the rows that lack values.
</Card>
<Card title="Geneva" icon="layer-group" href="/geneva">
docs/web/feature-engineering/data-evolution.mdx
-134
View File
@@ -1,134 +0,0 @@
---
title: "Data evolution"
sidebarTitle: "Data Evolution"
description: "Derive new columns from data already in the table, without rewriting what is already there."
icon: wand-magic-sparkles
keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
availability:
oss: available
enterprise: available
summary: >-
Adding columns computed from SQL expressions works in the embedded library
and in LanceDB Enterprise.
---
import {
PySchemaAddSetup,
TsSchemaAddSetup as TsSchemaAddSetup,
RsSchemaAddSetup as RsSchemaAddSetup,
PyAddColumnsCalculated as AddColumnsCalculated,
TsAddColumnsCalculated as TsAddColumnsCalculated,
RsAddColumnsCalculated as RsAddColumnsCalculated,
PyAddFeatureColumnsSql as AddFeatureColumnsSql,
TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
} from '/snippets/tables.mdx';
Add derived columns to calculate features from data already in a table.
For example, compute a product score from existing fields without reading
and rewriting the table in your application.
Lance can write the new columns without rewriting the existing columns.
This is useful when the table also contains large images, video or embeddings.
See [Lance format](/basics/lance) for the storage model.
<Note>
In SQL, on an Enterprise cluster that enables computed columns, this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
</Note>
## Set up the example table {#set-up-the-example-table}
The examples use `db`, a connection opened as in
[Connect to LanceDB](/basics/connect), and create a small product table in it:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PySchemaAddSetup}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsSchemaAddSetup}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsSchemaAddSetup}
</CodeBlock>
</CodeGroup>
## Add derived columns {#add-derived-columns}
You can add new columns that are derived from existing data using SQL expressions.
For feature engineering on large existing tables, group related derived features into
one `add_columns` operation instead of running many separate writes. This creates one
new table version for the schema change and computes the new columns from the existing
rows, which avoids growing the table's version history with many small updates.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{AddColumnsCalculated}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsAddColumnsCalculated}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsAddColumnsCalculated}
</CodeBlock>
</CodeGroup>
The same call can add multiple derived columns at once. For example, if you are
building several lightweight features from existing product fields, pass all of the
new column expressions together:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{AddFeatureColumnsSql}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsAddFeatureColumnsSql}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsAddFeatureColumnsSql}
</CodeBlock>
</CodeGroup>
<Info>
LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
PyArrow `RecordBatch` iterators for populating new columns. New column values must be
defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
If your transformation cannot be expressed in SQL, compute the values outside
`add_columns` before writing them back through another workflow.
</Info>
## When SQL is not enough {#beyond-sql}
The expressions above are evaluated once, as part of the call. For other workflows:
- A [computed column](/feature-engineering/add-computed-column) declares the
expression and leaves the column empty, so you choose when the work happens.
Use it when you want to schedule the computation separately.
- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for
values SQL cannot express: an embedding, a model call, an image decode.
{/*
Draft for the Functions release. Replaces the Geneva item in the list above:
- A [function](/management/function) fills a column with Python, for values SQL
cannot express -- an embedding, a model call, an image decode.
*/}
A deployment has to enable computed columns before you use them.
{/* Draft for the Functions release: Functions need enabling there too. */}
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
Declare now, fill later.
</Card>
<Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
Change the table's shape instead of its contents.
</Card>
</CardGroup>
docs/web/feature-engineering/index.mdx
+24 -15
View File
@@ -24,7 +24,7 @@ then refresh the derived data when its inputs change.
*/}
<Note>
In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it.
In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH COLUMN](/sql/dml/refresh-column) to fill it.
</Note>
## Computed columns and materialized views {#the-shape}
@@ -36,8 +36,8 @@ refresh costs:
| | Computed column | Materialized view |
| --- | --- | --- |
| Declaring or creating it | records the expression; no values are written | runs the query and stores its rows, unless created empty |
| A refresh | fills rows with no value and recomputes rows whose inputs changed | applies source changes or rebuilds the view, depending on the query and SDK version |
| A refresh when nothing changed | fills nothing | can be a no-op when incremental refresh is supported |
| A refresh | fills rows with no value and recomputes rows whose inputs changed | rebuilds the view completely from its source query |
| A refresh when nothing changed | fills nothing | still rebuilds the view and commits a new version |
| In LanceDB Enterprise | runs as a [job](/basics/jobs) | runs as a server-side materialization job |
### Computed columns fill incrementally {#incremental}
@@ -65,21 +65,19 @@ up-to-date features.
A [materialized view](/feature-engineering/materialized-views) has no separate
declaration step: creating one runs its query and stores the result, so the view
is ready to read. A refresh updates those stored results. Python SDK 0.40.0 can apply
changes incrementally for supported queries; other queries require a full rebuild.
See [refresh behavior](/feature-engineering/materialized-views#incremental) for
version-specific details.
is ready to read. A refresh rebuilds the stored result completely from the
source query; see [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
for the full behavior, including automatic refresh.
Enterprise runs materialization on the Enterprise service. Use the SDK and refresh
options supported by your server version; see
[Enterprise refreshes](/feature-engineering/materialized-views#availability).
Enterprise runs materialization on remote compute, as a server-side job.
## Choose a feature engineering workflow {#which-one}
| | What it produces | Filled by | Available |
| --- | --- | --- | --- |
| [View](/feature-engineering/view) | a saved query, no stored rows of its own | nothing -- it runs on every read | Enterprise |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Embedded and Enterprise |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Enterprise |
| [Geneva](/geneva) | columns filled by Python | the `geneva` package | <Badge color="red">Enterprise</Badge> |
{/*
@@ -88,6 +86,11 @@ Draft for the Functions release. Replace the Geneva row of the table above with:
| [Geneva](/geneva) | columns filled by Python, for existing pipelines | the `geneva` package | <Badge color="red">Enterprise</Badge> |
*/}
**A view** has no refresh and no staleness window, because it stores the
query rather than its result. Reach for one first, before a computed column
or a materialized view, when the query is cheap enough to simply re-run on
every read.
**Computed columns** are the common case: a column declared with a SQL
expression, filled by [a refresh](/feature-engineering/refresh-column).
@@ -147,20 +150,26 @@ should handle that itself, inside the function body.
When a refresh fails, fix the cause and run the refresh
again. The new refresh fills what is still empty or stale. How much of the
failed run was kept depends on where it ran (see
[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry
[Refresh a computed column](/feature-engineering/refresh-column#failures)), and a retry
can recompute values that were already written, so do not count on it costing
only the rows that did not complete.
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
<Card title="Create a view" icon="eye" href="/feature-engineering/view">
A saved query, with no storage or refresh of its own.
</Card>
<Card title="Create a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
Declare a column from a SQL expression.
</Card>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Fill the rows that lack values, locally or as a job.
</Card>
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/materialized-views">
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
Store a query's result as a table.
</Card>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
Bring a derived table up to date with its source.
</Card>
{/*
docs/web/feature-engineering/materialized-views.mdx
+62 -89
View File
@@ -1,122 +1,95 @@
---
title: "Refresh a materialized view"
sidebarTitle: "Materialized View Refresh"
description: "Create a materialized view and refresh it when the source data changes."
title: "Create a materialized view"
sidebarTitle: "Create Materialized View"
description: "Store the result of a query as a table, ready to query, index and search."
icon: table
keywords: ["materialized view", "create_materialized_view", "refresh", "full refresh"]
keywords: ["materialized view", "create_materialized_view"]
availability:
oss: available
embedded: unavailable
enterprise: available
summary: >-
Creation and refresh in LanceDB Embedded require a source build with the experimental-functions Cargo feature. Enterprise runs the work on the Enterprise service. See [Enterprise refreshes](#availability) for server compatibility requirements.
---
import {
PyMaterializedViewCreate as MaterializedViewCreate,
PyMaterializedViewRefresh as MaterializedViewRefresh,
} from '/snippets/materialized_view.mdx';
A materialized view stores the result of a query as a table. Use it to keep a
filtered or transformed dataset that you can query, index and search repeatedly.
Refresh the view after its source data changes.
filtered or transformed dataset that you can query, index and search
repeatedly, instead of recomputing the query every time you read it.
<Note>
In the 0.41 SDK release, materialized-view creation and refresh in LanceDB Embedded require a
source build with the `experimental-functions` Cargo feature. Published Python
and TypeScript packages leave it disabled. Opening, reading, listing and dropping
existing views remain available, as do LanceDB Enterprise operations.
In SQL this is [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view#create).
</Note>
<Note>
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
</Note>
## Create a view {#create}
## Create and refresh {#create-and-refresh}
<CodeGroup>
```python Python icon="python"
view = db.create_materialized_view(
"frames_captioned",
source="frames",
select=["frame_id", "video_id", "ts", "caption"],
where="selected = true",
)
```
<CodeBlock filename="Python" language="Python" icon="python">
{MaterializedViewCreate}
</CodeBlock>
```typescript TypeScript icon="square-js"
const view = await db.createMaterializedView("frames_captioned", "frames", {
select: ["frame_id", "video_id", "ts", "caption"],
where: "selected = true",
});
```
</CodeGroup>
`source` names the table the view reads. `select` takes column names,
`(alias, expression)` pairs, or a dict of the same; omitting it selects every
source column, expanded against the schema as it was at creation time.
`where` filters source rows, and `limit` caps the view.
Creating the view computes its rows, so it is ready to read. Pass
`with_no_data=True` to create it empty instead. Later changes to the source
reach the view through a refresh:
`with_no_data=True` to create it empty instead; a later
[refresh](/feature-engineering/refresh-materialized-view) populates it.
<CodeBlock filename="Python" language="Python" icon="python">
{MaterializedViewRefresh}
</CodeBlock>
Your administrator manages stable row IDs and other source-table
requirements for materialized views on the server; confirm support for your
source table's shape before creating a view over it.
The example works with Python SDK 0.40.0. It creates the source with stable
row IDs, which that release requires for ordinary materialized-view sources.
## Automatic refresh at creation {#auto-refresh}
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
Omitting it selects every source column, expanded against the schema as it was
at creation time. `where` filters source rows; `limit` caps the view.
Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` to schedule
refreshes instead of calling
[`refresh`](/feature-engineering/refresh-materialized-view) yourself:
## Prepare the source table {#stable-row-ids}
```python
view = db.create_materialized_view(
"frames_captioned",
source="frames",
select=["frame_id", "video_id", "ts", "caption"],
where="selected = true",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
In Python SDK 0.40.0, create the source with
`storage_options={"new_table_enable_stable_row_ids": "true"}`, as the example
above does. Stable row IDs let the view track source rows across updates and
compaction. Set this option when creating the table; it cannot be added later.
A source with the [realtime engine](/tables/realtime-engine) enabled or retained, uncompacted write-ahead-log
rows is not accepted by the local materialized-view implementation.
## Refresh behavior {#incremental}
With Python SDK 0.40.0, a refresh applies source changes incrementally when the
query supports it, and rebuilds the view otherwise. In the example, adding Bea
writes one new view row. Refreshing again without source changes writes none.
Use `refresh(full=True)` to force a rebuild.
`rows_written` counts rows written by that refresh, not all rows in the view.
Use `view.table.count_rows()` to get the total. Pass `source_version=N` to
refresh against a particular source version instead of the latest.
<Note>
Refresh behavior has changed between SDK versions. The earlier 0.40.0b13
development build rebuilt every local view on refresh and did not accept
`full`; 0.39.0 created views empty. Use the reference for your installed SDK
when maintaining an existing pipeline.
</Note>
## Enterprise refreshes {#availability}
<Badge color="red">Enterprise</Badge>
Enterprise runs materialized-view creation and refresh on the Enterprise service.
For SQL, see [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view) and
[`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view).
Use an SDK version supported by your Enterprise deployment. Materialized-view
APIs and stored definitions have changed across releases; a local SDK example
alone does not establish compatibility with an older server. Before upgrading
an existing pipeline, confirm the supported refresh options and whether its
views need to be recreated with your administrator.
Stable row IDs and feature configuration for Enterprise sources are managed on
the server. The source-table storage option in the local example does not
configure the Enterprise deployment, and Python SDK 0.40.0's `create_table` on a
`db://` connection does not accept `storage_options`: leave it out, and confirm
with your administrator that the deployment gives new tables stable row IDs.
`list_materialized_views` requests the server's view listing. On a local
database it reads table schemas to find views.
Omit `auto_refresh_enabled` to let the server decide whether to enable it,
and omit `auto_refresh_interval_seconds` to let the server decide the
pacing. See
[Automatic refresh](/feature-engineering/refresh-materialized-view#auto-refresh)
for how scheduling behaves once the view exists.
## Working with the view {#working-with-the-view}
`view.table` is the underlying table, for querying, indexing, or search.
`view.definition` is the recorded query: the select list, predicate, and
limit. Use it to inspect the query without inferring it from the rows. Both are properties; on an async view, the definition is read
with `await view.definition()`.
limit. Use it to inspect the query without inferring it from the rows. Both
are properties; on an async view, the definition is read with
`await view.definition()`.
An existing view is reopened with `db.open_materialized_view(name)`.
An existing view is reopened with `db.open_materialized_view(name)`. Creation,
refresh and drop also have async, job-returning SDK methods; see
[Materialized view](/management/materialized-view#manage).
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Derive a column instead of a whole table.
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
Bring the view up to date with its source.
</Card>
<Card title="Versioning" icon="clock-rotate-left" href="/management/table#versioning">
Inspect the table history and read a particular version.
docs/web/feature-engineering/refresh-column.mdx
+36 -21
View File
@@ -1,14 +1,12 @@
---
title: "Refresh a column"
sidebarTitle: "Column Refresh"
title: "Refresh a computed column"
sidebarTitle: "Refresh Computed Column"
description: "Populate a computed column and update it when its inputs change."
icon: arrows-rotate
keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Local column refresh requires a source build with the experimental-functions Cargo feature. Enterprise runs it as a server job that must be enabled for your Enterprise cluster.
---
Refresh a [computed column](/feature-engineering/add-computed-column) to fill
@@ -64,10 +62,6 @@ That is what makes the call idempotent and safe to retry. However, unlike a
spreadsheet formula, nothing is recomputed until you refresh, so a derived
column is stale between an update and the next refresh.
{/*
Draft for automatic computed-column refresh. Keep the released manual-refresh
instructions above until the server feature is available.
## Automatic refresh {#automatic}
Set scheduling options when declaring a computed column:
@@ -80,17 +74,17 @@ table.add_columns(
)
```
Both options apply to every computed output declared in that call, including
registered Function outputs. Automatic refresh is enabled by default. When
the interval is omitted, the server determines when to refresh. LanceDB Embedded
tables persist only an explicitly configured interval; automatic execution requires a
LanceDB Enterprise deployment with its separate `column_refresh.enabled` switch
turned on. That server switch is disabled in the base configuration.
These options apply to every computed output declared in that call. Automatic
refresh is enabled by default. When the interval is omitted, the server
determines when to refresh. Local tables store the setting, but automatic
execution requires a Cloud or Enterprise server with its separate
`column_refresh.enabled` switch turned on. That server switch is disabled in
the base configuration.
The interval is a positive minimum delay after the last successful refresh
of that output. A first refresh is immediately eligible. The scheduler still
checks input freshness and waits for computed ancestors to settle; the timer
does not make unchanged inputs require recomputation. Each Function output's
does not make unchanged inputs require recomputation. Each declared output's
interval is independent, so refreshing a sibling does not postpone it.
Pacing uses retained live Job Registry history. If completed jobs are archived
@@ -107,10 +101,31 @@ ALTER TABLE my_table ALTER COLUMN doubled SET auto_refresh_interval_seconds = 60
ALTER TABLE my_table ALTER COLUMN doubled SET auto_refresh_enabled = true;
```
Disabling automatic refresh prevents new automatic jobs. Inflight jobs may
finish, and manual refresh remains available. Re-enabling wakes maintenance
to consider accumulated changes under the interval and dependency rules.
*/}
<Note>
The current setting is in the column's own field metadata,
`computed_column.auto_refresh_enabled`, readable from `table.schema` without
submitting a job to ask.
</Note>
Setting the flag is a metadata-only change and always succeeds, independent of
whether the scheduler can actually refresh the column -- a deployment issue
that keeps the background job from running (for example, a worker that cannot
reach object storage) does not stop the `ALTER` from committing. Check the
column's job history, not just its metadata, if values stop appearing after
re-enabling.
**Disabling does not skip or drop writes.** Rows written while disabled
commit with no computed value, the same as a row written before the column
existed; they do not queue anywhere separate. Inflight jobs may finish, and
manual refresh remains available throughout.
**Re-enabling catches up the entire backlog in one pass**, not just rows
written after that point: the next scheduler tick processes every row left
unfilled since the column was disabled, in a single refresh, with no
further write required to trigger it. This is "catch up," not "resume
cadence" -- a column disabled for an hour and re-enabled fills everything
that accumulated during that hour, immediately, rather than only picking up
new rows from the moment it is re-enabled.
## Refreshing part of a table {#partial}
@@ -196,7 +211,7 @@ symptoms, which rarely name their cause.
<Card title="Jobs" icon="hourglass-half" href="/basics/jobs">
Waiting on, inspecting, and cancelling server-side work.
</Card>
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/materialized-views">
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/refresh-materialized-view">
The same idea, for a whole derived table.
</Card>
</CardGroup>
docs/web/feature-engineering/refresh-materialized-view.mdx
+88
View File
@@ -0,0 +1,88 @@
---
title: "Refresh a materialized view"
sidebarTitle: "Refresh Materialized View"
description: "Bring a materialized view up to date after its source data changes."
icon: arrows-rotate
keywords: ["materialized view", "refresh", "auto_refresh_enabled", "auto_refresh_interval_seconds"]
availability:
embedded: unavailable
enterprise: available
---
Refresh a [materialized view](/feature-engineering/materialized-views) to
bring its stored rows up to date after the source table changes.
<Note>
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh-materialized-view).
</Note>
## Refresh a view {#refresh}
```python
view.refresh()
```
Every refresh rebuilds the view completely from its source query -- there is
no incremental refresh. Calling it again with no source changes still
rebuilds and commits a new table version.
The result reports what the refresh did:
```python
result = view.refresh()
result.rows_written # rows in the rebuilt view
result.source_version # the source table version the view now reflects
result.version # the view's own new table version
```
Pass `source_version=N` to refresh against a particular source version
instead of the latest:
```python
view.refresh(source_version=12)
```
Use `view.table.count_rows()` for the view's row count at any time; it
always matches `rows_written` from the most recent refresh.
## Automatic refresh {#auto-refresh}
Set scheduling options when [creating the view](/feature-engineering/materialized-views#auto-refresh):
```python
view = db.create_materialized_view(
"frames_captioned",
source="frames",
select=["frame_id", "video_id", "ts", "caption"],
where="selected = true",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
`auto_refresh_interval_seconds` is a positive minimum delay after the last
successful refresh before the next automatic one is eligible; omit it to let
the server decide. The whole view toggles together -- there is no
per-column granularity the way a computed column has one setting per output.
The view's *initial* population, at creation or when the server first runs
it, happens regardless of these settings; only refreshes after that are
gated. Manual `view.refresh()` keeps working whether or not automatic
refresh is enabled.
## When a refresh fails {#failures}
A refresh replaces the view's stored rows in one commit, so a failure leaves
the view as it was after its last successful refresh. Fix the cause and
refresh again.
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
Define the query a view stores.
</Card>
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
The same idea, for a single derived column.
</Card>
</CardGroup>
docs/web/feature-engineering/view.mdx
+73
View File
@@ -0,0 +1,73 @@
---
title: "Create a view"
sidebarTitle: "Create View"
description: "Define a derived dataset as a saved query, with no storage or refresh of its own."
icon: eye
keywords: ["view", "create view", "feature engineering", "saved query"]
availability:
embedded: unavailable
enterprise: available
---
A view is the lightest way to derive a feature-engineered dataset: it stores
a **query**, not rows, so there is nothing to refresh and nothing that can
fall out of date. Reach for one before
[a computed column](/feature-engineering/add-computed-column) or
[a materialized view](/feature-engineering/materialized-views) when the
underlying query is cheap enough to simply re-run on every read.
## Define a view {#create}
A view over a products table, selecting and deriving a few columns:
```sql
CREATE VIEW "discounted_products" AS
SELECT id, name, price, price * 0.9 AS sale_price
FROM "products"
WHERE in_stock = true
```
The statement resolves the view's schema from the query at creation time.
Anything the [SQL dialect](/sql/functions) supports can define one: joins,
predicates, aggregates, window functions -- the same expressiveness as
[a computed column's](/feature-engineering/add-computed-column#expressions)
SQL expression, just over a whole derived table instead of one column.
## Read it like a table {#reading}
```sql
SELECT * FROM "discounted_products" WHERE sale_price < 20
```
Every read runs the saved query against current data, so `sale_price`
reflects whatever `price` is right now -- there is no staleness window to
reason about, unlike a materialized view between refreshes. A predicate
applied when reading the view composes with the query inside it, rather than
filtering the view's output afterward, so a selective filter does not pay
for rows the view would have produced and then discarded.
## When to use a view instead {#choosing}
| | View | Computed column / materialized view |
| --- | --- | --- |
| Stores | the query | derived values |
| Freshness | always current | as of the last refresh |
| Read cost | the full query, every read | a column or table scan |
| Write cost | none | a refresh |
Reach for a view when the query is cheap, or when reading a stale value would
be wrong for the workload. Reach for a computed column or a materialized view
when the query is expensive and the data changes less often than it is read.
See [View](/sql/ddl/view#choosing) for the full SQL reference comparison,
including the separate privileges each one needs.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="View" icon="eye" href="/sql/ddl/view">
CREATE VIEW, DROP VIEW, and listing views alongside tables.
</Card>
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
The stored-rows counterpart, and when a refresh is worth paying for.
</Card>
</CardGroup>
docs/web/governance/authorization.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Who may do what: privileges granted on catalog objects to principa
icon: shield-halved
keywords: ["authorization", "authz", "privileges", "rbac", "access control"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports catalog privileges when its administrator configures enforcement. LanceDB Embedded has no catalog authorization layer: access is whatever the object store allows.
---
[Authentication](/enterprise/authentication) establishes who is calling.
docs/web/index.mdx
+137 -81
View File
@@ -19,29 +19,6 @@ Training data and experimentation slow down when raw data, metadata, embeddings,
artifacts live in separate systems. LanceDB keeps them together in one versioned multimodal table, so AI teams spend less
time stitching infrastructure together and more time improving datasets, testing features, and keeping GPUs fed.
![Training data lifecycle: Curation, Feature Engineering, Search and Retrieval, Training](/static/assets/images/overview/training-data-lifecycle.svg)
Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs.
Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble
GPU-ready batches from a tagged dataset version.
For a deeper look at how this works in training pipelines, start with [Why LanceDB for training](/training/why-lancedb).
## LanceDB suite {#lancedb-suite}
The LanceDB suite includes LanceDB Embedded (open source), a retrieval library, and LanceDB Enterprise,
a multimodal lakehouse platform for the full AI data lifecycle.
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
It is easy to set up on a local machine for search and regular-scale workflows. LanceDB Enterprise is built
for teams that need scale without building bespoke infrastructure for curation,
feature engineering, search and retrieval, and efficient training data access.
LanceDB Enterprise runs LanceDB Distributed: query execution, indexing and maintenance
run on the Enterprise service, outside your application.
![LanceDB suite: LanceDB Embedded search and Enterprise multimodal lakehouse on Lance format](/static/assets/images/overview/lancedb-suite.svg)
## Why teams use LanceDB {#why-teams-use-lancedb}
<Steps>
<Step title="One table for the whole AI data loop">
Store images, video, audio, text, annotations, embeddings, and model-generated features together in one schema-enforced table.
@@ -58,70 +35,149 @@ run on the Enterprise service, outside your application.
</Step>
</Steps>
Use the same table to curate training data, add derived features, retrieve examples, and feed training jobs that rely on expensive GPUs.
Training workloads can sample, shuffle, and scan projected columns from local storage or object storage, then assemble
GPU-ready batches from a tagged dataset version. Training pipelines need curation, derived features, reproducible
splits, fast random access, and a clean path into frameworks such as PyTorch -- LanceDB keeps all of it connected
through the same table model, whether a workflow uses one table or several related ones.
## Architecture {#architecture}
LanceDB Embedded and Enterprise are both built on [Lance](/basics/lance), which stores table data, versions, and
indexes in open, columnar files. The format and table model are the same across both offerings -- what changes is
how storage and compute are organized around them.
```mermaid
%%{init: {"flowchart": {"curve": "basis", "nodeSpacing": 45, "rankSpacing": 55, "padding": 12}}}%%
flowchart TB
subgraph Embedded["LanceDB Embedded"]
EmbApp(["Your application<br/>+ LanceDB SDK"])
end
subgraph Enterprise["LanceDB Enterprise"]
EntSdk(["LanceDB SDK"]) --> EntEngine{{"Enterprise Distributed Engine"}}
end
Lance[("Lance format<br/>table data, versions, indexes")]
Embedded --> Lance
Enterprise --> Lance
```
Embedded reads and writes tables on local disk or in object storage from inside your application process.
Enterprise keeps durable table data and index artifacts in object storage, independently of the machines serving
requests, and separates query serving, read execution and caching, and indexing and background work into their
own independently sized components. Caches accelerate reads; they are never the durable source of truth. With
multiple serving replicas, Enterprise routes traffic to surviving nodes when one fails; losing a cache node
temporarily raises latency while its replacement warms, rather than causing an outage.
Sharing a format does not mean an Enterprise deployment automatically opens an existing Embedded database --
plan how to move or connect your data and configure access for your deployment.
## Embedded and Enterprise {#embedded-and-enterprise}
LanceDB ships as two offerings on the same Lance format and table model. Start locally with Embedded, then move to
Enterprise when your team needs distributed scale, managed infrastructure, private deployment, or higher-throughput
curation, feature engineering, search and retrieval, and training workflows.
### 1\. LanceDB Embedded {#1-lancedb-oss}
**LanceDB Embedded** is an open-source, embedded retrieval library, with client SDKs in Python, TypeScript and
Rust -- the application itself plans queries, builds indexes, and runs background work, with no separate database
server to deploy. This fits local retrieval applications, notebooks, development, and single-node data pipelines,
where your application controls its own compute resources and [table maintenance](/maintenance/compaction).
### 2\. LanceDB Enterprise {#2-lancedb-enterprise}
[LanceDB Enterprise](/enterprise) is a petabyte-scale (and beyond), distributed **multimodal lakehouse** platform
built for search, curation, feature engineering, and high-throughput training data access workflows on top of the
same core table abstraction, without building bespoke infrastructure. Applications connect through a LanceDB SDK
while the service handles requests across a fleet, with query serving, read execution and caching, and indexing
and background work sized and scaled as separate components. A control plane governs configuration, identity
and cluster lifecycle, separately from the data-plane components in the diagram below. See
[Distributed search](/enterprise/distributed-search) for how query nodes and plan executors divide a request.
```mermaid
%%{init: {"flowchart": {"curve": "basis", "nodeSpacing": 45, "rankSpacing": 55, "padding": 12}}}%%
flowchart TB
Clients(["Clients"])
QueryFleet{{"Query Fleet<br/>query serving"}}
ExecutorRing{{"Executor Ring<br/>read execution + cache"}}
MonitoringFleet{{"Monitoring Fleet"}}
JobWorkerFleet{{"Job Worker Fleet<br/>indexing, column + MV refresh, compaction"}}
Storage[("Cloud Object Storage")]
Clients --> QueryFleet
QueryFleet -- reads --> ExecutorRing
QueryFleet -- writes --> Storage
QueryFleet -- notifies --> MonitoringFleet
MonitoringFleet -- creates jobs --> JobWorkerFleet
ExecutorRing -- reads --> Storage
JobWorkerFleet -- writes --> Storage
```
Teams choose Enterprise to:
- **Serve growing workloads** -- add query-serving capacity and let Enterprise cache frequently read table data near
compute as traffic grows.
- **Reduce operational work** -- let the service handle background [compaction](/maintenance/compaction) and
[cleanup](/maintenance/cleanup) instead of scheduling them in application code.
- **Fit deployment requirements** -- run a managed deployment or bring LanceDB into your own cloud account, with
OpenTelemetry metrics and traces.
- **Connect data workflows** -- use the same tables for retrieval, curation, and training data access, with
[SQL](/basics/sql), [feature engineering](/feature-engineering), and [Geneva](/geneva) feature backfills.
To set up LanceDB Enterprise in your organization, [reach out to us](https://www.lancedb.com/contact).
### Compare features {#compare}
{/* availability-comparison */}
## Projects using LanceDB {#projects-using-lancedb}
### Training {#training}
- [stable-worldmodel](https://github.com/galilai-group/stable-worldmodel) -- a platform for reproducible world-model
research built on a LanceDB data layer, reporting 3-4x faster data loading on Push-T versus HDF5 / MP4 at a
fraction of the disk footprint.
- [le-wm](https://github.com/lucas-maes/le-wm) -- a joint-embedding predictive world model from pixels, trained on
the stable-worldmodel platform and its LanceDB data layer.
- [lerobot-lancedb](https://github.com/lancedb/lerobot-lancedb) -- a drop-in LanceDB backend for Hugging Face
LeRobot datasets with faster loading across robotics datasets.
### Agentic applications {#agentic-applications}
- [OpenClaw LanceDB Plugin](https://docs.openclaw.ai/plugins/memory-lancedb) -- stores an agent's long-term memory
in LanceDB, automatically recalling relevant memories before a model turn and capturing new facts afterward.
- [Cognee](https://github.com/topoteretes/cognee) -- an open-source AI memory platform that gives agents persistent
memory across sessions, using LanceDB as one of its vector store backends.
- [CrewAI](https://github.com/crewAIInc/crewAI) -- a multi-agent framework that
[rebuilt its agent memory on LanceDB](https://www.lancedb.com/blog/crewai-rebuilt-agent-memory-on-lancedb).
Across these projects, LanceDB and Lance provide the multimodal data layer that keeps raw observations, annotations,
features, and training or memory access patterns in one format instead of scattering them across task-specific
stores.
## Start with your workload {#start-with-your-workload}
<CardGroup cols={2}>
<Card title="Train and fine-tune models" icon="fire" href="/training/why-lancedb">
Learn why LanceDB works well as the data layer for training workloads.
</Card>
<Card title="Load data into PyTorch" icon="boxes-stacked" href="/training/data-loading">
Use LanceDB tables and permutations for projected, shuffled, random-access training reads.
</Card>
<Card title="Browse ready-to-use datasets" icon="database" href="/datasets">
Explore Lance-formatted multimodal datasets with raw bytes, metadata, embeddings, and indices.
<Card title="Ingest your data" icon="inbox" href="/ingestion">
Create tables, append records, and update or delete rows as your data changes.
</Card>
<Card title="Build search and retrieval" icon="search" href="/search/">
Use vector search, full-text search, hybrid search, reranking, filtering, and SQL.
</Card>
<Card title="Curate training data" icon="filter" href="/curation">
Profile, sample, chunk, join, and deduplicate records before training on them.
</Card>
<Card title="Engineer features in place" icon="layer-group" href="/feature-engineering">
Derive computed columns and materialized views without moving data elsewhere.
</Card>
<Card title="Load data into PyTorch" icon="boxes-stacked" href="/training/data-loading">
Use LanceDB tables and permutations for projected, shuffled, random-access training reads.
</Card>
<Card title="Manage catalog objects" icon="folder-tree" href="/management">
Create, list, and drop tables, namespaces, and views from Python, TypeScript, or SQL.
</Card>
</CardGroup>
## From local development to production scale {#from-local-development-to-production-scale}
LanceDB Embedded and LanceDB Enterprise share the same Lance format and table model. Start locally with
LanceDB Embedded, then move to Enterprise when your team needs distributed scale, managed infrastructure, private deployment,
or higher-throughput curation, feature engineering, search and retrieval, and training workflows.
### 1\. LanceDB Embedded {#1-lancedb-oss}
The fastest way to get started is the open-source embedded library, with client SDKs in Python, TypeScript
and Rust. Run it locally in just a few steps, which lets you explore datasets, curate data, and run search and retrieval workloads
for agents. Start here:
<Columns cols={2}>
<Card
title="Quickstart"
icon="rocket"
href="/quickstart"
>
Get started with LanceDB in minutes.
</Card>
<Card
title="Basic Table Operations"
icon="table"
href="/management/table"
>
Create tables, evolve schemas, version data, and modify rows in LanceDB.
</Card>
</Columns>
### 2\. LanceDB Enterprise {#2-lancedb-enterprise}
[LanceDB Enterprise](/enterprise) is a petabyte-scale (and beyond), distributed **multimodal lakehouse** platform built for
search, curation, feature engineering, and high-throughput training data access workflows on top of the same core table
abstraction. This eliminates the need for teams to build bespoke infrastructure to manage large multimodal datasets.
To set up LanceDB Enterprise in your organization,
[reach out to us](https://www.lancedb.com/contact).
<Info>
**Built with scale, performance, and security in mind.**
LanceDB Enterprise is designed for very large-scale, high-performance, distributed workloads in
private deployments, and can operate under strict [security requirements](/enterprise/security).
</Info>
<Card
title="Quickstart"
icon="rocket"
href="/quickstart"
>
Get started with LanceDB in minutes, including Enterprise `db://` connections.
</Card>
docs/web/indexing/fts-index.mdx
+1 -5
View File
@@ -4,12 +4,8 @@ sidebarTitle: "FTS Index"
description: "Create and tune BM25-based full-text search indexes in LanceDB."
icon: "book"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Full-text indexes are built in the embedded library and in LanceDB Enterprise.
The embedded library builds the index before the call returns; a
deployment builds it in the background.
---
import { PyFtsIndexAsync as FtsIndexAsync, PyFtsIndexCreate as FtsIndexCreate, PyFtsIndexNested as FtsIndexNested, PyFtsIndexWait as FtsIndexWait } from '/snippets/indexing.mdx';
docs/web/indexing/index.mdx
+1 -4
View File
@@ -4,11 +4,8 @@ sidebarTitle: "Overview"
description: "Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more."
icon: "list"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Indexes are built in the embedded library and in LanceDB Enterprise. Enterprise builds them in the background, and builds fewer vector index types than the
embedded library.
---
Indexes speed up searches and filters by organizing data for specific query
docs/web/indexing/reindexing.mdx
+1 -6
View File
@@ -5,13 +5,8 @@ description: "Learn how to keep your indexes up-to-date in LanceDB using increme
icon: "refresh"
keywords: ["reindexing", "re-index", "incremental indexing", "reindex"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Indexes are kept up to date in the embedded library and in LanceDB Enterprise. In
the embedded library you add new rows to them with `optimize()`; a
deployment does it in the background, and `optimize()` is not available
there.
---
import {
docs/web/indexing/scalar-index.mdx
+1 -5
View File
@@ -4,12 +4,8 @@ sidebarTitle: "Scalar Index"
description: "Learn how to use scalar indexes in LanceDB for efficient metadata filtering and query optimization."
icon: "tree"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Scalar indexes are built in the embedded library and in LanceDB Enterprise. A
deployment builds them in the background and keeps most of them up to date
itself; in the embedded library you update them with `optimize()`.
---
import {
docs/web/indexing/vector-index.mdx
+1 -6
View File
@@ -4,13 +4,8 @@ sidebarTitle: "Vector Index"
description: "Build and manage LanceDB vector indexes."
icon: "arrow-up-right-dots"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Vector indexes are built in the embedded library and in LanceDB Enterprise. The
embedded library builds an index before `create_index()` returns; a
deployment builds it in the background, keeps it up to date, and builds
only `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and `IVF_HNSW_SQ`.
---
import {
docs/web/ingestion/create-table.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Create a table from records or a schema and choose input formats."
icon: table
keywords: ["create table", "ingest", "arrow", "pandas", "polars", "pydantic", "schema"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Create and open tables in LanceDB Embedded and LanceDB Enterprise.
---
import {
docs/web/ingestion/delete.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Delete"
description: "Select and delete records, verify the remaining rows, and understand storage retention."
icon: trash
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Delete rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Storage reclamation follows compaction and retention policy.
---
import { RsExample as RsDeleteRows } from '/snippets/ingestion_delete.mdx';
docs/web/ingestion/index.mdx
+2 -4
View File
@@ -5,10 +5,8 @@ description: "Choose how to create a table, append records or change existing da
icon: inbox
keywords: ["ingest", "create table", "add", "insert", "arrow", "pandas", "polars", "pydantic"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Create tables and write records through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also provides SQL write statements.
---
import { PyWriteWorkflow, TsWriteWorkflowBody, TsWriteWorkflowTail, RsWriteWorkflow } from '/snippets/ingestion_examples.mdx';
@@ -32,7 +30,7 @@ with a stable key when the incoming data should update existing records.
## Connect and prepare the data {#connect-to-lancedb}
Use the [connection guide](/basics/connect) to connect to LanceDB Embedded or LanceDB Enterprise.
Use the [connection guide](/basics/catalog) to connect to Embedded or Enterprise.
The SDK write operations in this section work with both. Enterprise also
supports [SQL write statements](/sql#dml); an SDK filter expression does not
require a separate SQL connection.
docs/web/ingestion/insert.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Insert"
description: "Append records, load file-backed data or batches, and verify ingestion."
icon: plus
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Append records through the SDKs in LanceDB Embedded and LanceDB Enterprise. For large Enterprise writes, see [large upload settings](#bulk-ingests-into-a-remote-table).
---
import {
docs/web/ingestion/merge-insert.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Merge Insert"
description: "Match incoming records by key and choose which rows to update, insert or remove."
icon: code-merge
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Merge insert is available through the SDKs in LanceDB Embedded and LanceDB Enterprise. Use it for key-based updates and inserts.
---
import {
docs/web/ingestion/realtime-write.mdx
+196
View File
@@ -0,0 +1,196 @@
---
title: "Real Time writes"
sidebarTitle: "Real Time Write"
description: "Enable the real time engine on a table and absorb small, frequent writes cheaply, each one committed in milliseconds."
icon: "bolt"
keywords: ["rte", "real time engine", "wal", "memwal", "lsm", "write-ahead log", "streaming writes", "upsert"]
availability:
embedded: unavailable
enterprise: available
---
The [real time engine](/tables/realtime-engine) puts an LSM-style write tier in
front of a table, turning many small writes into a few large commits. This
page covers enabling it and writing to a table; for querying an RTE table, see
[Real Time reads](/search/realtime-read).
## Getting started {#getting-started}
### 1. Create a table with a primary key {#primary-key}
RTE deduplicates writes by primary key. Create the table with a non-nullable
key column, then declare it with `set_unenforced_primary_key`. The key is a
single `int32`, `int64`, string, or binary column, and cannot be changed once
set.
The table below holds frames captioned and embedded as they stream in from a
live camera feed, such as a robot recording episodes into a dataset shaped
like [LeRobotDataset](/integrations/lerobotdataset)'s `frames` table —
`frame_id` is the primary key so a re-captioned frame upserts in place:
<CodeGroup>
```python Python icon="python"
from datetime import timedelta
import lancedb
import numpy as np
import pyarrow as pa
from lancedb import RTEWriteSpec
from lancedb.index import FTS, BTree, IvfPq
db = lancedb.connect("db://my_db", api_key="...", host_override="https://...")
dim = 768
schema = pa.schema([
pa.field("frame_id", pa.int64(), nullable=False),
pa.field("video_id", pa.utf8(), nullable=False),
pa.field("embedding", pa.list_(pa.float32(), dim)),
pa.field("caption", pa.utf8()),
])
frames = db.create_table("frames", schema=schema)
frames.set_unenforced_primary_key("frame_id")
```
</CodeGroup>
Or in SQL:
<CodeGroup>
```sql SQL
ALTER TABLE frames SET UNENFORCED PRIMARY KEY (frame_id)
```
</CodeGroup>
### 2. Create indexes {#indexes}
By default, RTE maintains every supported index on the table, including
indexes created after it is enabled. A scalar index on the primary key is
required for partial updates and deletes.
<CodeGroup>
```python Python icon="python"
frames.create_index("embedding", config=IvfPq())
frames.create_index("caption", config=FTS())
frames.create_index("frame_id", config=BTree())
frames.wait_for_index(["embedding_idx", "caption_idx", "frame_id_idx"], timeout=timedelta(minutes=5))
```
</CodeGroup>
### 3. Enable RTE {#enable}
<CodeGroup>
```python Python icon="python"
frames.enable_rte(RTEWriteSpec.unsharded())
```
</CodeGroup>
Or in SQL:
<CodeGroup>
```sql SQL
ALTER TABLE frames SET RTE
```
</CodeGroup>
### 4. Write {#write}
<Warning>
On an RTE table, an upsert through `merge_insert` is the only way to write
rows. Use the form below exactly: `when_matched_update_all()` with no filter,
plus `when_not_matched_insert_all()`. Other `merge_insert` forms, such as
conditional updates or deleting rows missing from the source, are rejected.
</Warning>
Writes go through `merge_insert` on the primary key, and are visible to queries
immediately — see [Real Time reads](/search/realtime-read) for querying them
back:
<CodeGroup>
```python Python icon="python"
new_frames = pa.table({
"frame_id": pa.array([2000, 2001], pa.int64()),
"video_id": pa.array(["clip-9", "clip-9"]),
"embedding": pa.FixedSizeListArray.from_arrays(
pa.array(np.random.rand(2 * dim).astype("float32")), dim),
"caption": pa.array(["a gripper closing on a cup", "a cup set down on the table"]),
}, schema=schema)
frames.merge_insert("frame_id") \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute(new_frames)
```
</CodeGroup>
**Partial updates.** Send the primary key plus only the columns you want to
change; omitted columns keep their current values. This requires a scalar
index on the primary key. A re-captioning job, for example, only touches
`caption`:
<CodeGroup>
```python Python icon="python"
patch = pa.table(
{"frame_id": [2000, 2001], "caption": ["a two-finger gripper closing on a cup", "the cup resting upright"]},
schema=pa.schema([schema.field("frame_id"), schema.field("caption")]),
)
frames.merge_insert("frame_id").when_matched_update_all().when_not_matched_insert_all().execute(patch)
```
</CodeGroup>
**Deletes.** The predicate must bind the primary key to literal values. Ranges,
`OR`, and filters on other columns are rejected. Deletes require a scalar index
on the primary key.
<CodeGroup>
```python Python icon="python"
frames.delete("frame_id IN (2000, 2001)")
```
</CodeGroup>
### Disabling RTE {#disable}
Stop writes and wait for in-flight writes to finish, then flush pending writes
into the base table and disable. `checkpoint_rte()` only drains writes that
existed when it started, so anything written during the checkpoint can be left
behind. `disable_rte()` is rejected while the write-ahead log still holds rows,
so run `checkpoint_rte()` first.
<CodeGroup>
```python Python icon="python"
frames.checkpoint_rte()
frames.disable_rte()
```
</CodeGroup>
## Write performance {#write-performance}
RTE acknowledges a write as soon as it is durable in the write-ahead log, so
ingest is no longer limited by how fast the table can commit.
**Compared with writing directly to the table.** On tables ranging from
hundreds of thousands to tens of millions of rows, RTE delivered 4–12× the
write throughput and 24–52× lower p99 write latency, depending on the mix of
inserts, updates, and deletes.
**Scaling with shards.** An RTE table can be split into
[shards](/sql/ddl/table#properties) that write independently, and a single
table's ingest grows with its shard count. In our testing on AWS, with every
index on the table (vector, full-text, and BTree) kept current, a single table
reached these peak rates:
| Shards | Peak ingest |
| --: | --: |
| 1 | 90 MB/s |
| 2 | 170 MB/s |
| 4 | 340 MB/s |
| 8 | 625 MB/s |
| 16 | 690 MB/s |
| 32 | 885 MB/s |
Adding shards scales a table's ingest nearly linearly.
## Next steps {#next-steps}
- [Real Time reads](/search/realtime-read) — query an RTE table
- [How the real time engine works](/tables/realtime-engine#how-it-works)
- [Merge insert](/ingestion/merge-insert)
- [`ALTER TABLE ... SET RTE`](/sql/ddl/table#properties)
docs/web/ingestion/update.mdx
+2 -4
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Update"
description: "Select existing rows, change values or expressions, and verify the changes."
icon: pen
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Update rows through the SDKs in LanceDB Embedded and LanceDB Enterprise. Enterprise also supports the SQL UPDATE statement.
---
import {
@@ -83,7 +81,7 @@ connection or table handle, account for its
Nested-field updates are not supported by these operations. For schema changes,
see [schema evolution](/management/table#schema). Treat derived values separately:
changing an input field is not a guarantee that every stored feature has been
recomputed. See [Refresh a column](/feature-engineering/refresh-column).
recomputed. See [Refresh a computed column](/feature-engineering/refresh-column).
Updates can change index coverage and query cost. After a large update,
inspect the workload and follow [index maintenance](/indexing/reindexing)
docs/web/integrations/ai/huggingface.mdx
+2 -5
View File
@@ -3,11 +3,8 @@ title: "Hugging Face Hub"
sidebarTitle: "Hugging Face"
description: "Use LanceDB directly on Lance datasets hosted on the Hugging Face Hub for multimodal search and retrieval."
availability:
oss: available
enterprise: unavailable
summary: >-
The embedded library opens Lance datasets on the Hugging Face Hub in place
through `hf://` URIs. A deployment does not read `hf://` datasets.
embedded: available
enterprise: available
---
[Hugging Face Hub](https://huggingface.co/datasets?format=format:lance&sort=trending) is a popular platform for sharing machine learning datasets, models, and other resources.
docs/web/integrations/data/pandas_and_pyarrow.mdx
+1 -5
View File
@@ -2,12 +2,8 @@
title: "Pandas and PyArrow"
sidebarTitle: "Pandas & PyArrow"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
DataFrames go into `create_table` and come back from query results the same
way for local and deployment tables.
---
import {
docs/web/integrations/data/polars_arrow.mdx
+1 -6
View File
@@ -2,13 +2,8 @@
title: "Polars"
sidebarTitle: "Polars & Arrow"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Polars data goes into `create_table`, and query results come back as Polars
DataFrames, for local and deployment tables alike; a LazyFrame over a whole
table is local only.
---
import {
docs/web/integrations/data/pydantic.mdx
+1 -6
View File
@@ -2,13 +2,8 @@
title: "Pydantic"
sidebarTitle: "Pydantic"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
The SDK converts `LanceModel` schemas to Arrow, and query rows back to
models, in your process, so they work the same with local and deployment
tables.
---
import {
docs/web/integrations/lerobotdataset.mdx
+19
View File
@@ -138,6 +138,25 @@ class EmbedFromBlob: # SigLIP2 loads once per worker
...
```
**Deduplication** removes near-static runs before they skew curation or
training. A stopped gripper holding one pose for a few seconds produces many
frames whose `emb_siglip2` vectors are nearly identical; LanceDB Enterprise's
native [`dedupe`](/sql/dql/dedupe) collapses them into one
representative per cluster, directly over the index, without a pairwise scan:
```python
db.execute_query("""
CREATE MATERIALIZED VIEW frames_deduped AS
SELECT * FROM dedupe('frames', 'emb_siglip2', distance_threshold => 0.01)
""")
```
Every column survives the dedup unchanged -- `jerk_score`, the episode and
camera metadata, the video blob reference -- since `dedupe` returns full
rows, not just which ones matched. Wrap it in a materialized view, as above,
to keep the deduplicated set current as new episodes land; see
[Materialized views](/feature-engineering/materialized-views).
**Search** is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms.
**Curation** is one query across all of it. Semantic search with a predicate on the derived score, no join, no second system:
docs/web/maintenance/background-jobs.mdx
+3 -6
View File
@@ -5,16 +5,13 @@ description: "Understand the Enterprise background jobs that maintain tables, in
icon: gears
keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
LanceDB Enterprise schedules background maintenance jobs according to your
cluster configuration.
---
The jobs documented elsewhere in these docs ([Build & Repair](/indexing/reindexing),
[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup),
[Column Refresh](/feature-engineering/refresh-column)) start from a call you
[Refresh Computed Column](/feature-engineering/refresh-column)) start from a call you
make or, for compaction, cleanup and index updates, from the deployment's own
maintenance. This page is the rest: jobs the deployment runs to keep itself
correct and fast, which you may see named in [`SHOW JOBS`](/sql/ddl/job#show)
@@ -27,7 +24,7 @@ also start cache-warming jobs through the admin API.
## Write-ahead log maintenance {#wal}
For a table with the [realtime engine](/tables/realtime-engine) enabled, two jobs keep the
For a table with the [real time engine](/tables/realtime-engine) enabled, two jobs keep the
log from growing without bound:
| Job type | What it does |
docs/web/maintenance/cleanup.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Pruning old versions and the files only they referenced."
icon: broom
keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Both offerings support cleanup. Embedded `optimize()` uses a retention window of 7 days by default; Enterprise uses configured background cleanup and retention, and does not support `optimize()`.
---
Every write keeps the version before it, which is what makes
docs/web/maintenance/compaction.mdx
+2 -6
View File
@@ -4,12 +4,8 @@ sidebarTitle: "Compaction"
description: "Merge small files, prune old versions, and fold new rows into existing indexes."
icon: broom
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Tables are compacted in the embedded library and in LanceDB Enterprise. In the
embedded library you call `optimize()`; a deployment compacts tables in the
background, and `optimize()` is not available there.
---
import { PyStorageCompactionTuning, PyStorageRebuildVectorIndex, RsStorageCompactionTuning } from '/snippets/doc_storage_examples.mdx';
@@ -98,7 +94,7 @@ continuous small writes needs it on a schedule.
<Note>
Compaction rewrites files, which conflicts with a concurrent
[computed-column refresh](/feature-engineering/add-computed-column) or
[view refresh](/feature-engineering/materialized-views) planned against the
[view refresh](/feature-engineering/refresh-materialized-view) planned against the
version it replaces. Sequence them rather than running both at once.
</Note>
docs/web/management/database.mdx
+5 -10
View File
@@ -5,12 +5,8 @@ description: "Connect to LanceDB and configure storage, credentials and read con
icon: database
keywords: ["connect", "database", "uri", "api key", "storage options", "consistency"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
LanceDB Embedded opens tables in a local directory or object storage.
LanceDB Enterprise connects with a database URI and credentials and runs
operations with LanceDB Distributed.
---
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
@@ -27,11 +23,10 @@ objects, and configure how your application reaches the data.
| `s3://bucket/path`, `gs://bucket/path`, `az://container/path` | LanceDB Embedded, with object storage | In your application |
| `db://my-database` | LanceDB Enterprise | On the Enterprise service (LanceDB Distributed) |
[Connect to LanceDB](/basics/connect) has examples for each connection type.
A connection to object storage still uses LanceDB Embedded: your process
reads and writes the bucket directly. LanceDB Enterprise runs LanceDB
Distributed: query execution, indexing and maintenance run on the Enterprise
service, outside your application.
[Connect to LanceDB](/basics/catalog) has examples for each connection type.
A connection to object storage still uses the embedded library: your process
reads and writes the bucket directly. Enterprise runs operations remotely and
manages the supporting compute and maintenance.
## Connect to Enterprise {#deployment}
docs/web/management/function.mdx
+268 -59
View File
@@ -4,11 +4,6 @@ sidebarTitle: "Function"
description: "Register a Python function once and apply it as a column, for work SQL cannot express."
icon: code
keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"]
availability:
oss: unavailable
enterprise: available
summary: >-
Enterprise supports Functions when enabled for your Enterprise cluster; LanceDB Embedded cannot register or run them.
---
import {
@@ -110,6 +105,26 @@ with `db.get_function(name, version=...)`.
the function is baked into an image before the version becomes usable.
</Note>
## Try it in SQL {#try-sql}
A registered function is callable in SQL by its name, the same as any other
scalar function. Before applying it as a column, call it directly over the
table to check the registration succeeded, without committing a schema
change:
<CodeGroup>
```python Python icon="python"
reader = db.execute_query("SELECT value, double(value) AS doubled FROM function_demo")
print(reader.read_all().to_pylist())
# [{'value': 1.0, 'doubled': 2.0}, {'value': 2.0, 'doubled': 4.0}, {'value': 3.0, 'doubled': 6.0}]
```
</CodeGroup>
<Note>
Equivalent SQL: `SELECT value, double(value) AS doubled FROM function_demo`.
See [Querying with SQL](/basics/sql) for `execute_query` and other languages.
</Note>
## Apply it as a column {#apply}
Calling the registered version binds each parameter, by name, to a column with
@@ -124,6 +139,51 @@ function application keeps its scalar or named-struct result in the named
column. A bare named-struct application expands its result fields as one atomic
binding, with aliases from `rename(columns=...)`.
### Bind more than one input, or fill more than one column {#multi}
Parameters bind by name, so a function that takes several arguments binds
several columns in one call:
<CodeGroup>
```python Python icon="python"
@udf
def ratio(numerator: float, denominator: float) -> float:
return numerator / denominator
version = db.create_function_async(ratio).wait()
table.add_columns({"ratio": version(numerator=col("value"), denominator=col("count"))})
```
</CodeGroup>
A named-struct output fills more than one column from a single application.
Declare the fields with `output_schema`, then alias each field to the column
it fills with `rename(columns=...)` and pass the application itself, not a
mapping, so both columns come from one atomic binding:
<CodeGroup>
```python Python icon="python"
import pyarrow as pa
@udf(
output_schema=pa.schema([
pa.field("half", pa.float64(), nullable=False),
pa.field("double", pa.float64(), nullable=False),
]),
)
def scale(value: float):
return {"half": value / 2, "double": value * 2}
version = db.create_function_async(scale).wait()
application = version(value=col("value")).rename(
columns={"half": "value_half", "double": "value_double"}
)
table.add_columns(application)
```
</CodeGroup>
A single refresh fills `value_half` and `value_double` together: there is no
state where one is filled and the other is still empty.
## Fill the column {#refresh}
A refresh runs the function on the cluster as a [job](/basics/jobs). `wait()`
@@ -188,46 +248,61 @@ undecorated class, so local tests can create an instance and call it directly.
A Function can bind a [Blob v2 column](/management/table#managed-blobs) and
return another blob. In row mode, a top-level blob parameter arrives as Python
`bytes`, or `None` for a nullable input. The worker reads the payload before
calling your function. Return bytes and let the destination table store the
Managed payload and descriptor.
`bytes`, or `None` for a nullable input, after the worker has read the
payload. Return bytes and let the destination table store the Managed
payload and descriptor.
Declare blob inputs and outputs explicitly. This example reverses the bytes
to demonstrate the round trip without requiring an image library. Here `db`
is an Enterprise database and `table` contains an `image` Blob v2 column
with table schema metadata `lance.blob.managed=true`, as in the
[table creation example](/management/table#managed-blobs-read-write):
```python
from datetime import timedelta
The rest of this section extracts a frame from a stored video and captions
it, continuing as one example across chaining, persisting and incremental
reads below. `videos` holds raw clips in a `video` Blob v2 column; `frames`
names one timestamp per clip to extract, with a `selected` flag to skip some:
<CodeGroup>
```python Python icon="python"
import lancedb
from lancedb import col
from lancedb.functions import udf
import pyarrow as pa
@udf(
name="reverse_image",
python_version="3.12",
pip=["pyarrow==26.0.0"],
input_schema=pa.schema([lancedb.blob("image")]),
output_schema=lancedb.blob("reversed", nullable=False),
)
def reverse_image(image):
return b"missing" if image is None else image[::-1]
function = db.create_function_async(reverse_image).wait(
timeout=timedelta(minutes=15)
)
table.add_columns({"reversed": function(image=col("image"))})
result = table.refresh_column_async("reversed").wait(
timeout=timedelta(minutes=15)
)
assert result.rows_failed == 0
hits = table.search().select(["id", "reversed"]).with_row_id(True).to_arrow()
values = table.fetch_blobs("reversed", hits).to_pylist()
videos = db.create_table("videos", schema=pa.schema([
pa.field("id", pa.utf8(), nullable=False),
lancedb.blob("video", nullable=False),
]))
frames = db.create_table("frames", [
{"frame_id": 1, "video_id": "clip-1", "ts": 0.5, "selected": True},
{"frame_id": 2, "video_id": "clip-1", "ts": 1.5, "selected": True},
])
```
</CodeGroup>
### Produce a blob {#produce-blobs}
Declare blob inputs and outputs explicitly. `extract_frame` reads one frame
out of a video at a timestamp and returns it as a JPEG, without loading the
whole clip first -- see [Read a blob incrementally](#incremental) below for
how the `video: BlobReader` parameter does that:
<CodeGroup>
```python Python icon="python"
from datetime import timedelta
from lancedb import BlobReader, col
from lancedb.functions import udf
@udf(
name="extract_frame",
gpu=True,
pip=["av==13.1.0", "pillow==11.0.0"],
input_schema=pa.schema([
lancedb.blob("video"),
pa.field("ts", pa.float64(), nullable=False),
]),
output_schema=lancedb.blob("jpeg", nullable=False),
)
def extract_frame(video: BlobReader, ts: float) -> bytes:
... # decode the frame at ts, encode as JPEG, return its bytes
version = db.create_function_async(extract_frame).wait(timeout=timedelta(minutes=15))
```
</CodeGroup>
An empty byte string is a filled result. A scalar output cannot be nullable,
because a physical NULL marks an unfilled computed row. To return a nullable
@@ -235,29 +310,93 @@ blob as a value, use a named-struct output schema with a nullable blob field;
the struct itself records whether the result has been filled. Read a nested
blob with `table.fetch_blobs("payload.image", hits)`.
On deployments that support computed-column inputs, another Function can bind
to `col("reversed")`. Refresh the producer first, then the consumer. Once
outputs are current, a later refresh fills appended rows. A filtered refresh
preserves payload references for unselected Managed outputs. An unfiltered
refresh can recompute a whole fragment when it cannot establish that the
fragment's outputs are current.
<Warning>
`output_schema=lancedb.blob(...)` is what makes a persisted result Blob v2
rather than plain Arrow `binary`. Returning `bytes` from a function whose
output type is inferred from a bare annotation, with no explicit
`lancedb.blob(...)` output schema, persists as ordinary `binary`: the column
still holds the right bytes, but it skips the Managed storage path, descriptor
reads, and `fetch_blobs`. Declare it explicitly on every function whose output
should be a blob, not just the first one in a chain.
</Warning>
### Chain a Function on its output {#chain}
A second Function can bind a blob another Function produced. Register
`caption` the same way, consuming a blob instead of producing one:
<CodeGroup>
```python Python icon="python"
@udf(gpu=True, pip=["transformers==4.47.0"])
def caption(jpeg: bytes) -> str:
... # run a vision-language model over the JPEG, return one sentence
caption_version = db.create_function_async(caption).wait(timeout=timedelta(minutes=15))
```
</CodeGroup>
Compose both in one SQL statement over a join, instead of applying each as
its own column. The join supplies `video` and `ts` from two different
tables, and `caption` reads `extract_frame`'s result as it is computed,
before anything is written:
<CodeGroup>
```sql SQL
SELECT f.frame_id, extract_frame(v.video, f.ts) AS jpeg,
caption(extract_frame(v.video, f.ts)) AS caption
FROM videos v JOIN frames f ON v.id = f.video_id
WHERE f.selected = true
```
</CodeGroup>
[`EXPLAIN`](/sql/dql/explain) on a query like this shows `RemoteTake`: the
blob is read once, where the worker runs, and never pulled into the query
node first.
### Persist it as a column, or a materialized view {#persist}
Apply the same chained expression with `add_columns` to persist it as an
ordinary table column:
<CodeGroup>
```python Python icon="python"
table.add_columns({"jpeg": extract_frame(video=col("video"), ts=col("ts"))})
result = table.refresh_column_async("jpeg").wait(timeout=timedelta(minutes=15))
assert result.rows_failed == 0
hits = table.search().select(["frame_id", "jpeg"]).with_row_id(True).to_arrow()
values = table.fetch_blobs("jpeg", hits).to_pylist()
```
</CodeGroup>
Or wrap the join and both Functions in a
[materialized view](/sql/ddl/materialized-view) instead, so the extracted
frame and its caption stay current as new videos and frames arrive. See
[Automatic refresh](/feature-engineering/refresh-column#automatic) for
keeping either one fresh without calling refresh yourself.
On deployments that support computed-column inputs, another Function can
bind to `col("jpeg")` directly, the same as the SQL join above does through
its own expression. Refresh the producer first, then the consumer. Once
outputs are current, a later refresh fills appended rows. A filtered
refresh preserves payload references for unselected Managed outputs. An
unfiltered refresh can recompute a whole fragment when it cannot establish
that the fragment's outputs are current.
Function output blobs follow the same compaction and cleanup lifecycle as
ingested Managed blobs. Retained versions continue to own their payloads;
cleanup reclaims an object only after its retained references are gone.
### Draft: incremental blob inputs (upcoming)
This feature requires an SDK that exports `BlobReader` and a compatible
Enterprise runtime with reader input support enabled. Coordinate the runtime
upgrade before registering or executing reader Functions.
### Read a blob incrementally {#incremental}
Annotate a row Function parameter with `BlobReader` to read a Blob v2 input
incrementally. The signature selects the access mode without another
registration option. For example, read the first MP4 box type without loading
the whole video:
incrementally, the way `extract_frame` above does, instead of materializing
the whole payload first. The signature selects the access mode without
another registration option. A minimal example, reading only the first MP4
box type without loading the rest of the video:
```python
<CodeGroup>
```python Python icon="python"
from lancedb import BlobReader, udf
@@ -266,12 +405,15 @@ def first_box_type(video: BlobReader) -> bytes:
video.seek(4)
return video.read(4)
```
</CodeGroup>
The worker passes a seekable reader backed by the stored blob. `read(size)`
reads from the current cursor; `read_range(offset, length)` reads a byte range
without moving it. `size()` returns the payload size without reading it.
Each row call receives its own reader, which closes when the call returns or
raises. Do not retain it for another invocation.
The worker passes a seekable reader backed by the stored blob --
`video.seekable()` is `True`. `read(size)` reads from the current cursor;
`read_range(offset, length)` reads a byte range without moving it, the same
bytes a `seek(offset)` followed by `read(length)` would read. `size()`
returns the payload size without reading it. Each row call receives its own
reader, which closes when the call returns or raises. Do not retain it for
another invocation.
Use `video: bytes` when the Function needs the complete payload. Bytes inputs
materialize before invocation and remain subject to eager input limits.
@@ -293,6 +435,14 @@ dependencies are part of its definition:
| `python_version` | Python major/minor for the execution environment, defaulting to yours |
| `gpu` | whether every execution requires a GPU |
<CodeGroup>
```python Python icon="python"
@udf(gpu=True)
def extract_frame(video: BlobReader, ts: float) -> bytes:
... # runs only on a GPU-capable worker
```
</CodeGroup>
These are part of the **immutable function version**. Changing a dependency
produces a new version rather than altering the one already registered, so a
column filled by version 1 is unaffected by version 2.
@@ -322,6 +472,46 @@ The client version defaults `python_version`, so a function registered from a
right and occasionally surprising. Pin it when the function is sensitive to it.
</Note>
## Remote calls and credentials {#secrets}
A function that calls an external API needs a credential without putting it
in the function's source or its `env` map in plain text. A Secret is a
database-scoped named credential; `db.create_secret` stores the value once,
and an `EnvVarSecret` binds it to the environment variable the function
reads. Binding is a registration-time decision, passed to
`create_function_async`, not to `@udf`:
<CodeGroup>
```python Python icon="python"
import os
from lancedb import EnvVarSecret
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
@udf
def embed(text: str) -> list[float]:
import os
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.embeddings.create(input=text, model="text-embedding-3-small")
return response.data[0].embedding
version = db.create_function_async(
embed,
secrets=[EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")],
).wait()
```
</CodeGroup>
No API returns a Secret's value, and it never appears in the function's
registration: `embed` reads `OPENAI_API_KEY` the way it always did, and the
binding is what puts a value there. A mistyped Secret name is caught at
registration; a mistyped environment variable name surfaces only on the
first rows, since nothing records which variables a function reads.
## Failure handling {#failures}
A function that raises fails the job. There is no per-row retry or skip policy
@@ -350,6 +540,25 @@ In practice this means a function should be self-contained: take its inputs as
arguments, import what it needs at module level, and avoid capturing state from
the surrounding scope.
Pass `code=[module, ...]` to ship additional helper modules into the same
image, for logic the function imports but that does not belong inlined into
it -- a model-loading helper shared by several Functions, for example.
Continuing `caption` from [above](#chain), with its model-loading code
factored into a sibling module instead of inlined:
<CodeGroup>
```python Python icon="python"
import caption_worker # a sibling module with the model-loading logic
@udf(gpu=True, code=[caption_worker])
def caption(jpeg: bytes) -> str:
return caption_worker.run(jpeg)
```
</CodeGroup>
Each module in `code` is captured and shipped the same way the function
itself is: as source, snapshotted at registration.
## Drop it {#drop}
`db.drop_function(name, version=...)` drops the function's name and the object
docs/web/management/index.mdx
+1 -1
View File
@@ -64,7 +64,7 @@ Enterprise, it requests the server's view listing.
</Note>
See [Materialized views](/management/materialized-view) for creating one, and
[Refresh a materialized view](/feature-engineering/materialized-views) for keeping
[Refresh a materialized view](/feature-engineering/refresh-materialized-view) for keeping
it current.
### Views {#views}
docs/web/management/materialized-view.mdx
+42 -22
View File
@@ -5,10 +5,8 @@ description: "Create, open, list and drop a table whose contents are defined by
icon: table
keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"]
availability:
oss: available
embedded: unavailable
enterprise: available
summary: >-
Both offerings support materialized views. See [Enterprise compatibility](#deployment) for connection and server requirements.
---
A materialized view is a table whose contents are defined by a query over
@@ -19,10 +17,10 @@ The result is an ordinary table: queryable, indexable, searchable, and present
in `table_names()`. Nothing about reading it is special.
This page covers creating and managing one. Keeping it current is
[Refresh a materialized view](/feature-engineering/materialized-views).
[Refresh a materialized view](/feature-engineering/refresh-materialized-view).
<Note>
In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show).
In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show).
</Note>
## Create a materialized view {#create}
@@ -30,25 +28,25 @@ In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-
<CodeGroup>
```python Python icon="python"
view = db.create_materialized_view(
"recent_docs",
source="documents",
select=["id", "title", "embedding"],
where="created_at > TIMESTAMP '2026-01-01 00:00:00'",
"frames_captioned",
source="frames",
select=["frame_id", "video_id", "ts", "caption"],
where="selected = true",
)
```
```typescript TypeScript icon="square-js"
const view = await db.createMaterializedView("recent_docs", "documents", {
select: ["id", "title", "embedding"],
where: "created_at > TIMESTAMP '2026-01-01 00:00:00'",
const view = await db.createMaterializedView("frames_captioned", "frames", {
select: ["frame_id", "video_id", "ts", "caption"],
where: "selected = true",
});
```
</CodeGroup>
`source` names the table the view reads. With Python SDK 0.40.0, create that
table with stable row IDs before creating the view. The
[complete example](/feature-engineering/materialized-views#create-and-refresh)
shows the setup. `select`, `where` and `limit` define the query.
`source` names the table the view reads — here, a `frames` table with a
`caption` column, filtered to the frames selected for captioning. See
[Create a materialized view](/feature-engineering/materialized-views#create)
for the full walkthrough. `select`, `where` and `limit` define the query.
## Open, list and drop views {#manage}
@@ -60,6 +58,30 @@ shows the setup. `select`, `where` and `limit` define the query.
| refresh | `refresh` | `refresh` | `REFRESH MATERIALIZED VIEW` |
| drop | `drop_table` | `dropTable` | `DROP TABLE` |
<Badge color="red">Enterprise</Badge> Creating, refreshing and dropping a
materialized view each have an async, job-returning counterpart --
`create_materialized_view_async`, `view.refresh_async()`,
`drop_materialized_view_async` -- the same pattern as
[`refresh_column_async`](/feature-engineering/refresh-column#async): each
returns a [job](/basics/jobs) you `wait()` on instead of blocking. The async
counterpart of the example [above](#create):
<CodeGroup>
```python Python icon="python"
from datetime import timedelta
job = db.create_materialized_view_async(
"frames_captioned", source="frames",
select=["frame_id", "video_id", "ts", "caption"], where="selected = true",
)
view = job.wait(timeout=timedelta(minutes=15))
```
</CodeGroup>
`create_materialized_view_async` also accepts `auto_refresh_enabled` and
`auto_refresh_interval_seconds`; see
[Automatic refresh](/feature-engineering/refresh-materialized-view#auto-refresh).
<Note>
On a local database, `list_materialized_views` reads table schemas to find
views. On Enterprise, it asks the server for the view listing, as
@@ -78,16 +100,14 @@ That also means dropping one removes stored rows, unlike dropping a
## Enterprise compatibility {#deployment}
Enterprise supports creating and refreshing materialized views on the
Enterprise service. Use the SDK version supported by your deployment, or the
[SQL interface](/sql/ddl/materialized-view). See
[Enterprise refreshes](/feature-engineering/materialized-views#availability)
for compatibility considerations when upgrading an existing pipeline.
Enterprise supports creating and refreshing materialized views on remote
compute. Use the SDK version supported by your deployment, or the
[SQL interface](/sql/ddl/materialized-view).
## Related guides {#next}
<CardGroup cols={2}>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
How a refresh works, and what it writes.
</Card>
<Card title="Views" icon="eye" href="/management/view">
docs/web/management/namespace.mdx
+1 -4
View File
@@ -5,11 +5,8 @@ description: "Grouping tables into a hierarchy, and addressing them through it."
icon: folder-tree
keywords: ["namespace", "catalog", "hierarchy", "directory namespace", "rest namespace"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Namespaces work in the embedded library, through directory or REST
catalogs, and in LanceDB Enterprise.
---
import {
docs/web/management/table.mdx
+3 -7
View File
@@ -5,12 +5,8 @@ description: "The object most operations act on: creating, opening, renaming, cl
icon: table
keywords: ["table", "create_table", "open_table", "clone", "rename", "drop"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Tables are created, evolved, versioned and dropped in the embedded library
and in LanceDB Enterprise. Renaming, cloning, dropping every table and pruning old
versions differ between the two.
---
import {
@@ -953,8 +949,8 @@ First, let's create a sample table with product data to demonstrate schema evolu
<Note>
Columns derived from existing data (SQL expressions over other columns) are
covered in [Data evolution](/feature-engineering/data-evolution), because
deriving values is a feature-engineering job rather than a schema one.
covered in [Create a computed column](/feature-engineering/add-computed-column),
because deriving values is a feature-engineering job rather than a schema one.
</Note>
#### Add columns with default values {#schema-add-columns-with-default-values}
docs/web/management/view.mdx
+2 -4
View File
@@ -1,14 +1,12 @@
---
title: "Views"
sidebarTitle: "View"
description: "A saved query, evaluated when read. The non-materialized counterpart to a materialized view."
description: "A saved query, evaluated when read."
icon: eye
keywords: ["view", "create view", "sql", "catalog"]
availability:
oss: unavailable
embedded: unavailable
enterprise: available
summary: >-
Enterprise supports views when enabled for your Enterprise cluster; the embedded library cannot create them.
---
A view saves a query that you can refer to by name. Reading the view executes
docs/web/quickstart.mdx
+110 -77
View File
@@ -4,26 +4,24 @@ sidebarTitle: "Quickstart"
description: "Install LanceDB, create a table, and run your first vector search."
icon: rocket
---
import { PyConnect, TsConnect, RsConnect, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartVectorSearch1, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1 } from '/snippets/quickstart.mdx';
import { PyConnect, TsConnect, RsConnect, PyConnectObjectStorage, TsConnectObjectStorage, RsConnectObjectStorage, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartAddFeature, PyQuickstartComputedFeature, PyQuickstartComputedFeatureEnterprise, PyQuickstartMultimodalBytes, PyQuickstartTraining, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, TsQuickstartCurateWithMetadata, TsQuickstartAddFeature, TsQuickstartQueryFeature, TsQuickstartMultimodalBytes, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1, RsQuickstartCurateWithMetadata, RsQuickstartAddFeature, RsQuickstartQueryFeature, RsQuickstartMultimodalBytes } from '/snippets/quickstart.mdx';
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
Follow the steps below, or jump to the [complete examples](#code) to copy a runnable program.
## 1\. Install LanceDB {#1-install-lancedb}
Choose your language. Python uses Polars to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
<CodeGroup>
```bash Python (pip) icon="python"
pip install lancedb polars
pip install lancedb pandas
```
```bash Python (uv) icon="python"
uv venv
uv pip install lancedb polars
uv pip install lancedb pandas
```
```bash TypeScript icon="square-js"
@@ -112,7 +110,7 @@ uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`
## 2\. Connect to a database {#2-connect-to-a-lancedb-database}
### Use a local directory {#connect-via-local-directory-path}
### Local directory {#connect-via-local-directory-path}
LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed.
@@ -130,10 +128,27 @@ LanceDB creates an `ex_lancedb` directory to store your tables. No server is nee
</CodeBlock>
</CodeGroup>
The [complete examples](#code) include the imports and entry points for each language. For Python's async API, see [Connecting to a database](/basics/connect).
For Python's async API, see [Connecting to a database](/basics/catalog).
<Accordion title="Connect to LanceDB Enterprise instead">
### Connect to LanceDB Enterprise {#connect-to-lancedb-enterprise}
### Object storage {#connect-via-object-storage-uris}
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Configuring storage](/deployment/storage) for credentials and provider-specific options.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsConnectObjectStorage}
</CodeBlock>
</CodeGroup>
### Enterprise {#connect-to-lancedb-enterprise}
Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`.
@@ -152,27 +167,20 @@ Use the database URI, API key, and endpoint provided by your administrator or th
</CodeGroup>
<Note>
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/connect#enterprise) for an example.
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/catalog#enterprise) for an example.
</Note>
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between Embedded and Distributed execution, see [Moving to LanceDB Enterprise](/basics/sdk#moving-to-enterprise).
</Accordion>
<Accordion title="Connect directly to object storage instead">
### Connect via object storage URIs {#connect-via-object-storage-uris}
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Configuring storage](/deployment/storage) for credentials and provider-specific options.
</Accordion>
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [LanceDB Enterprise](/enterprise).
## 3\. Create a table and add data {#3-create-a-new-table}
Each row describes a character and stores a vector alongside their name, description, and stats.
Each row describes a frame from a robot video clip and stores an embedding alongside its video, caption, and quality stats.
<Info>
These vectors are made up for this example. In your application, use an embedding model to generate vectors from your data.
These embeddings are made up for this example. In your application, use an embedding model to generate them from your data.
</Info>
<Accordion title="Character records">
<Accordion title="Frame records">
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartData}
@@ -187,7 +195,7 @@ These vectors are made up for this example. In your application, use an embeddin
</CodeGroup>
</Accordion>
Create a `characters` table from those records:
Create a `frames` table from those records:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
@@ -201,9 +209,9 @@ Create a `characters` table from those records:
</CodeBlock>
</CodeGroup>
This example uses overwrite mode so you can run it again. It replaces any existing `characters` table in the database you connect to.
This example uses overwrite mode so you can run it again. It replaces any existing `frames` table in the database you connect to.
Add one more character:
Add one more frame:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
@@ -219,11 +227,11 @@ Add one more character:
## 4\. Search the table {#4-semantic-search}
Find the two vectors closest to the query vector. This example uses a fixed vector; it does not call an embedding model.
Find the two embeddings closest to the query embedding. This example uses a fixed embedding; it does not call an embedding model.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartVectorSearch1}
{"# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install pandas` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(result)\n"}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartVectorSearch1}
@@ -233,84 +241,109 @@ Find the two vectors closest to the query vector. This example uses a fixed vect
</CodeBlock>
</CodeGroup>
Merlin and Morgana come back first. Python and Rust display a Polars DataFrame; TypeScript displays an array of results. The `_distance` column measures how far each result is from your query vector, with smaller values indicating a closer match.
Frames 2 and 4 come back first -- the gripper closing on the cup, then lowering it. Python displays a Pandas DataFrame, Rust displays a Polars DataFrame, and TypeScript displays an array of results. The `_distance` column measures how far each result is from your query embedding, with smaller values indicating a closer match.
## Complete examples {#code}
## 5\. Filter results {#5-curation}
Each example below creates the local database, adds four characters, and prints the two closest matches. Use the dependencies from step 1.
<Accordion title="Python: quickstart.py">
Save this as `quickstart.py` and run `python quickstart.py`. If you installed with uv, run `uv run quickstart.py`.
Combine vector search with a metadata filter, such as `stats.motion >= 4`, to curate results instead of just ranking them:
<CodeGroup>
<CodeBlock filename="quickstart.py" language="Python" icon="python">
{PyConnect}
{PyQuickstartData}
{PyQuickstartCreateTable}
{PyQuickstartAddData}
{PyQuickstartVectorSearch1}
<CodeBlock filename="Python" language="Python" icon="python">
{"curated = (\n table.search(query_vector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(curated)\n"}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartCurateWithMetadata}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartCurateWithMetadata}
</CodeBlock>
</CodeGroup>
</Accordion>
<Accordion title="TypeScript: quickstart.mts">
Save this as `quickstart.mts` and run `node quickstart.mts` from the directory where you installed LanceDB.
Frames 2 and 4 come back again here, since both already have `motion >= 4`; frames 1 and 3 are excluded from ranking entirely, rather than ranked and then dropped. See [SQL filtering](/search/filtering) for the full predicate syntax.
## 6\. Add a derived feature {#6-add-a-derived-feature}
Add a column calculated from existing fields with a SQL expression. This fills every existing row once, immediately:
<CodeGroup>
<CodeBlock filename="quickstart.mts" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnect}
{TsQuickstartData}
{TsQuickstartCreateTable}
{TsQuickstartAddData}
{TsQuickstartVectorSearch1}
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddFeature}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartAddFeature}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartAddFeature}
</CodeBlock>
</CodeGroup>
</Accordion>
<Accordion title="Rust: src/main.rs">
Replace `src/main.rs` with this program, then run `cargo run` from your project directory. The schema and Arrow conversion helpers are included.
Read it back like any other column:
<CodeGroup>
<CodeBlock filename="src/main.rs" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsQuickstartDefineStruct}
{"\n#[tokio::main]\nasync fn main() -> Result<(), Box<dyn std::error::Error>> {\n"}
{RsConnect}
{RsQuickstartData}
{RsQuickstartCreateTable}
{RsQuickstartAddData}
{RsQuickstartVectorSearch1}
{"\nOk(())\n}\n"}
<CodeBlock filename="Python" language="Python" icon="python">
{"features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).limit(10).to_pandas()\nprint(features)\n"}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartQueryFeature}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartQueryFeature}
</CodeBlock>
</CodeGroup>
</Accordion>
## Next steps {#next-steps}
See [Adding columns](/management/table#schema-add-new-columns) for more on this form.
You've created a table, added rows, and searched it. Continue with the guides below.
## 7\. Set up computed columns {#7-computed-columns}
### Filter results {#5-curation}
A computed column is declared up front and filled -- or refilled -- on demand, instead of being calculated once like step 6's. Declare `caption_length`, then fill it explicitly:
Combine vector search with metadata filters, such as `stats.magic >= 4`. See [SQL filtering](/search/filtering).
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartComputedFeature}
</CodeBlock>
</CodeGroup>
### Extend your table {#extensions}
A later `refresh_column` call fills only rows added since the last refresh and recomputes rows whose inputs changed, leaving unchanged rows alone. On LanceDB Enterprise, the equivalent refresh runs as a background job instead of blocking:
#### Add a derived feature {#6-add-a-derived-feature}
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartComputedFeatureEnterprise}
</CodeBlock>
</CodeGroup>
Add columns calculated from existing fields. See [Adding columns](/management/table#schema-add-new-columns).
TypeScript and Rust declare and refresh computed columns the same way, through the same `add_columns`/`refresh_column` calls. See [Computed columns](/feature-engineering/add-computed-column) for the full refresh semantics and the automatic-refresh option.
#### Computed columns {#computed-columns}
## 8\. Store multimodal data {#8-store-multimodal-data}
Declare a column and choose when to fill or refresh it. See [Computed columns](/feature-engineering/add-computed-column).
Store images, audio, or other binary data alongside vectors and metadata, in a column of raw bytes. This creates a new table from one image file -- replace the path with one of your own:
#### Store multimodal data {#7-store-multimodal-data}
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartMultimodalBytes}
</CodeBlock>
</CodeGroup>
Store images, audio, and other binary data alongside vectors and metadata. See [Multimodal data types](/management/table#multimodal).
See [Multimodal data types](/management/table#multimodal) for larger payloads and the Blob v2 column type, which avoids loading binary data on every scan.
#### Train on your data {#8-train}
## 9\. Train on your data {#9-train}
Read table data into PyTorch for training. See [Data loading](/training/data-loading).
Read table data into PyTorch, projecting and shuffling only the columns a training step needs:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartTraining}
</CodeBlock>
</CodeGroup>
`Permutation` is LanceDB's row-ordering primitive for training reads; `select_columns` projects before reading, so a training step touches only the columns it selects, not unused ones such as `caption` or `embedding`. See [Data loading](/training/data-loading) for the full `StreamingDataset` API, including resuming and multi-worker loading.
## Where to go next {#next-steps}
<Columns cols={2}>
<Card title="Basic table operations" icon="table" href="/management/table">
docs/web/reranking/index.mdx
+1 -3
View File
@@ -5,10 +5,8 @@ description: "Choose a reranker, apply it to search candidates and evaluate rele
icon: "sort-amount-down"
keywords: ["re-ranking", "reranker", "rerank"]
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Reranking is available with LanceDB Embedded and LanceDB Enterprise.
---
import {
docs/web/search/explain.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Query Plans"
description: "Inspect query execution, identify expensive work and compare changes."
icon: "map"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Query plans are available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyQueryPlan, TsQueryPlanBody, TsQueryPlanTail } from '/snippets/search_examples.mdx';
docs/web/search/filtering.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: Filtering
description: "Use predicates to select rows and control when filters apply to search."
icon: "filter"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Metadata filtering is available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyFilterSearch, TsFilterSearchBody, TsFilterSearchTail } from '/snippets/search_examples.mdx';
docs/web/search/full-text-search.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: Full-Text Search (FTS)
description: "Find records by words, phrases or partial matches using a full-text index."
icon: "book"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Full-text search is available in LanceDB Embedded and LanceDB Enterprise.
---
import {
docs/web/search/hybrid-search.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Hybrid Search"
description: "Combine vector similarity and keyword matches in a single ranked result set."
icon: "search"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Hybrid search is available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
docs/web/search/index.mdx
+2 -4
View File
@@ -4,10 +4,8 @@ sidebarTitle: Overview
description: "Choose vector, full-text or hybrid search and return the records your application needs."
icon: "list"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Vector, full-text and hybrid search are available in LanceDB Embedded and LanceDB Enterprise.
---
import { PyVectorSearch, TsVectorSearchBody, TsVectorSearchTail } from '/snippets/search_examples.mdx';
@@ -68,7 +66,7 @@ and removes it when finished.
This query returns the puppy and dog records, ordered by vector distance.
It searches the three-row table directly, without an ANN index or an embedding
service. For connection options, see [Connect](/basics/connect).
service. For connection options, see [Connect](/basics/catalog).
</Accordion>
docs/web/search/realtime-read.mdx
+74
View File
@@ -0,0 +1,74 @@
---
title: "Real Time reads"
sidebarTitle: "Real Time Read"
description: "Query a table with the real time engine enabled and get writes back within milliseconds of being written, merged transparently with the base table."
icon: "magnifying-glass"
keywords: ["rte", "real time engine", "real time search", "real time query", "fresh reads", "memwal", "upsert"]
availability:
embedded: unavailable
enterprise: available
---
A table with the [real time engine](/tables/realtime-engine) enabled queries
the same way as any other table — nothing about `search`, filtering, or SQL
changes. This page covers what you get back and how it performs; for enabling
RTE and writing to a table, see [Real Time writes](/ingestion/realtime-write).
## Query a real time table {#query}
Every query runs against the base table and the fresh tier (unflushed
memtables and SSTables) in parallel, and the two are merged into one result.
Fresh rows supersede base rows with the same primary key, so a row written a
moment ago is already reflected:
<CodeGroup>
```python Python icon="python"
# continuing the `frames` table connected and written to in Real Time writes
import numpy as np
query_vector = np.random.rand(frames.schema.field("embedding").type.list_size).astype("float32")
frames.search(query_vector, vector_column_name="embedding").limit(10).to_list()
frames.search("gripper closing on a cup", fts_columns="caption").limit(10).to_list()
```
</CodeGroup>
Vector, full-text, BTree, bitmap, and most other scalar indexes are maintained
on the memtable, so new rows are searchable through the index as soon as
they're written — not just after the next compaction.
## Read performance {#read-performance}
Reads on an RTE table stay close to those on a fully compacted table, even
with the fresh tier at its fullest before compaction.
The table below compares a 5M-row table with a 128-dimension IVF-PQ index plus
BTree and FTS indexes, with and without RTE. The RTE table holds one
unflushed memtable and three SSTables, each from a 100k-row write wave, half of
which are updates to existing rows. Latencies are p50 in milliseconds.
| Query | Without RTE | With RTE |
| --- | --: | --: |
| Primary-key lookup | 62 | 63 |
| Vector search | 68 | 71 |
| Vector search with prefilter | 90 | 91 |
| Full-text term | 63 | 64 |
| Full-text phrase | 68 | 82 |
| Hybrid search (RRF) | 68 | 83 |
| SQL point lookup | 189 | 187 |
Most queries are within a few milliseconds of the base table. Hybrid and phrase
queries add roughly 15 ms to merge the two tiers.
<Note>
Support for all table operations on RTE tables is coming soon. Time travel,
branching, `count_rows`, row IDs (`with_row_id`), and `TABLESAMPLE` are not yet
available.
</Note>
## Next steps {#next-steps}
- [Real Time writes](/ingestion/realtime-write) — enable RTE and write to a table
- [How the real time engine works](/tables/realtime-engine#how-it-works)
- [Hybrid search](/search/hybrid-search)
- [Vector search](/search/vector-search)
docs/web/search/vector-search.mdx
+1 -3
View File
@@ -4,10 +4,8 @@ sidebarTitle: "Vector Search"
description: "Search by vector similarity, choose a distance metric and tune your queries."
icon: "arrow-up-right-dots"
availability:
oss: available
embedded: available
enterprise: available
summary: >-
Vector search is available in LanceDB Embedded and LanceDB Enterprise.
---
import {
docs/web/snippets/quickstart.mdx
+31 -31
View File
@@ -1,66 +1,66 @@
{/* Auto-generated by scripts/mdx_snippets_gen.py. Do not edit manually. */}
export const PyQuickstartAddData = "table.add(\n [\n {\n \"id\": \"4\",\n \"name\": \"Morgana\",\n \"role\": \"Sorceress\",\n \"description\": \"Powerful sorceress of Avalon.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 4},\n \"vector\": [0.3, 0.9, 0.6, 0.8],\n }\n ]\n)\n";
export const PyQuickstartAddData = "table.add(\n [\n {\n \"frame_id\": \"4\",\n \"video_id\": \"clip-2\",\n \"caption\": \"a gripper lowering a cup onto the table\",\n \"stats\": {\"motion\": 5, \"sharpness\": 4, \"exposure\": 4, \"framing\": 2},\n \"embedding\": [0.3, 0.9, 0.6, 0.8],\n }\n ]\n)\n";
export const PyQuickstartAddFeature = "table.add_columns(\n {\n \"power_score\": \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n }\n)\n";
export const PyQuickstartAddFeature = "table.add_columns(\n {\n \"quality_score\": \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\"\n }\n)\n";
export const PyQuickstartComputedFeature = "table.add_columns(computed={\"description_length\": \"length(description)\"})\ntable.refresh_column(\"description_length\")\n";
export const PyQuickstartComputedFeature = "table.add_columns(computed={\"caption_length\": \"length(caption)\"})\ntable.refresh_column(\"caption_length\")\n";
export const PyQuickstartComputedFeatureEnterprise = "table.add_columns(computed={\"description_length\": \"length(description)\"})\njob = table.refresh_column_async(\"description_length\")\njob.wait()\n";
export const PyQuickstartComputedFeatureEnterprise = "table.add_columns(computed={\"caption_length\": \"length(caption)\"})\njob = table.refresh_column_async(\"caption_length\")\njob.wait()\n";
export const PyQuickstartCreateTable = "table = db.create_table(\"characters\", data=data, mode=\"overwrite\")\n";
export const PyQuickstartCreateTable = "table = db.create_table(\"frames\", data=data, mode=\"overwrite\")\n";
export const PyQuickstartCreateTableAsync = "async_table = await async_db.create_table(\n \"characters\",\n data=data,\n mode=\"overwrite\",\n)\n";
export const PyQuickstartCreateTableAsync = "async_table = await async_db.create_table(\n \"frames\",\n data=data,\n mode=\"overwrite\",\n)\n";
export const PyQuickstartCurateWithMetadata = "curated = (\n table.search(query_vector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(curated)\n";
export const PyQuickstartCurateWithMetadata = "curated = (\n table.search(query_vector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(curated)\n";
export const PyQuickstartData = "data = [\n {\n \"id\": \"1\",\n \"name\": \"King Arthur\",\n \"role\": \"King\",\n \"description\": \"Leader of Camelot and wielder of Excalibur.\",\n \"stats\": {\"strength\": 4, \"magic\": 1, \"leadership\": 5, \"wisdom\": 4},\n \"vector\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"id\": \"2\",\n \"name\": \"Merlin\",\n \"role\": \"Wizard\",\n \"description\": \"Advisor and prophet with deep magical knowledge.\",\n \"stats\": {\"strength\": 2, \"magic\": 5, \"leadership\": 4, \"wisdom\": 5},\n \"vector\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"id\": \"3\",\n \"name\": \"Sir Lancelot\",\n \"role\": \"Knight\",\n \"description\": \"Legendary knight known for courage and combat skill.\",\n \"stats\": {\"strength\": 5, \"magic\": 1, \"leadership\": 3, \"wisdom\": 3},\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n";
export const PyQuickstartData = "data = [\n {\n \"frame_id\": \"1\",\n \"video_id\": \"clip-1\",\n \"caption\": \"a robot arm reaching toward a cup\",\n \"stats\": {\"motion\": 3, \"sharpness\": 4, \"exposure\": 4, \"framing\": 3},\n \"embedding\": [0.7, 0.1, 0.9, 0.7],\n },\n {\n \"frame_id\": \"2\",\n \"video_id\": \"clip-1\",\n \"caption\": \"a two-finger gripper closing on a cup\",\n \"stats\": {\"motion\": 5, \"sharpness\": 4, \"exposure\": 5, \"framing\": 2},\n \"embedding\": [0.2, 0.9, 0.4, 0.9],\n },\n {\n \"frame_id\": \"3\",\n \"video_id\": \"clip-2\",\n \"caption\": \"a robot arm holding a cup steady\",\n \"stats\": {\"motion\": 2, \"sharpness\": 3, \"exposure\": 3, \"framing\": 4},\n \"embedding\": [0.9, 0.1, 0.5, 0.4],\n },\n]\n";
export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_bytes = Path(\"data/sir-lancelot.jpg\").read_bytes()\n\nmultimodal_table = db.create_table(\n \"character_images\",\n data=[\n {\n \"id\": \"lancelot\",\n \"description\": \"Portrait of Sir Lancelot\",\n \"image\": image_bytes,\n \"vector\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n";
export const PyQuickstartMultimodalBytes = "from pathlib import Path\n\nimage_bytes = Path(\"data/sample-frame.jpg\").read_bytes()\n\nmultimodal_table = db.create_table(\n \"frame_images\",\n data=[\n {\n \"frame_id\": \"thumb-1\",\n \"caption\": \"Sample frame thumbnail\",\n \"image\": image_bytes,\n \"embedding\": [0.9, 0.1, 0.5, 0.4],\n }\n ],\n mode=\"overwrite\",\n)\n";
export const PyQuickstartOutputPandas = "# Ensure you run `pip install pandas` beforehand\nresult = table.search(query_vector).limit(2).to_pandas()\nprint(result)\n";
export const PyQuickstartQueryFeature = "features = table.search().select([\"name\", \"role\", \"power_score\"]).to_polars()\nprint(features)\n";
export const PyQuickstartQueryFeature = "features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).to_polars()\nprint(features)\n";
export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ntable.add_columns(\n {\n \"magic\": \"cast(stats.magic as float)\",\n \"strength\": \"cast(stats.strength as float)\",\n }\n)\ndataset = Permutation.identity(table).select_columns([\"magic\", \"strength\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n";
export const PyQuickstartTraining = "import torch\nfrom lancedb.permutation import Permutation\n\ntable.add_columns(\n {\n \"motion\": \"cast(stats.motion as float)\",\n \"sharpness\": \"cast(stats.sharpness as float)\",\n }\n)\ndataset = Permutation.identity(table).select_columns([\"motion\", \"sharpness\"])\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=1024, shuffle=True)\n";
export const PyQuickstartVectorSearch1 = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nresult = (\n table.search(query_vector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(result)\n";
export const PyQuickstartVectorSearch1 = "# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_polars()\n)\nprint(result)\n";
export const PyQuickstartVectorSearch1Async = "# Search for examples similar to a \"wise magical advisor\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nasync_result = await (\n await async_table.search(query_vector)\n).select([\"name\", \"role\", \"description\", \"_distance\"]).limit(2).to_polars()\nprint(async_result)\n";
export const PyQuickstartVectorSearch1Async = "# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install polars` beforehand\nasync_result = await (\n await async_table.search(query_vector)\n).select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"]).limit(2).to_polars()\nprint(async_result)\n";
export const TsQuickstartAddData = "await table.add([\n {\n id: \"4\",\n name: \"Morgana\",\n role: \"Sorceress\",\n description: \"Powerful sorceress of Avalon.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 4 },\n vector: [0.3, 0.9, 0.6, 0.8],\n },\n]);\n";
export const TsQuickstartAddData = "await table.add([\n {\n frame_id: \"4\",\n video_id: \"clip-2\",\n caption: \"a gripper lowering a cup onto the table\",\n stats: { motion: 5, sharpness: 4, exposure: 4, framing: 2 },\n embedding: [0.3, 0.9, 0.6, 0.8],\n },\n]);\n";
export const TsQuickstartAddFeature = "await table.addColumns([\n {\n name: \"power_score\",\n valueSql:\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\",\n },\n]);\n";
export const TsQuickstartAddFeature = "await table.addColumns([\n {\n name: \"quality_score\",\n valueSql:\n \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\",\n },\n]);\n";
export const TsQuickstartCreateTable = "const table = await db.createTable(\"characters\", data, { mode: \"overwrite\" });\n";
export const TsQuickstartCreateTable = "const table = await db.createTable(\"frames\", data, { mode: \"overwrite\" });\n";
export const TsQuickstartCurateWithMetadata = "const curated = await table\n .search(queryVector)\n .where(\"stats.magic >= 4\")\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(curated);\n";
export const TsQuickstartCurateWithMetadata = "const curated = await table\n .search(queryVector)\n .where(\"stats.motion >= 4\")\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(curated);\n";
export const TsQuickstartData = "const data = [\n {\n id: \"1\",\n name: \"King Arthur\",\n role: \"King\",\n description: \"Leader of Camelot and wielder of Excalibur.\",\n stats: { strength: 4, magic: 1, leadership: 5, wisdom: 4 },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n {\n id: \"2\",\n name: \"Merlin\",\n role: \"Wizard\",\n description: \"Advisor and prophet with deep magical knowledge.\",\n stats: { strength: 2, magic: 5, leadership: 4, wisdom: 5 },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n {\n id: \"3\",\n name: \"Sir Lancelot\",\n role: \"Knight\",\n description: \"Legendary knight known for courage and combat skill.\",\n stats: { strength: 5, magic: 1, leadership: 3, wisdom: 3 },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const TsQuickstartData = "const data = [\n {\n frame_id: \"1\",\n video_id: \"clip-1\",\n caption: \"a robot arm reaching toward a cup\",\n stats: { motion: 3, sharpness: 4, exposure: 4, framing: 3 },\n embedding: [0.7, 0.1, 0.9, 0.7],\n },\n {\n frame_id: \"2\",\n video_id: \"clip-1\",\n caption: \"a two-finger gripper closing on a cup\",\n stats: { motion: 5, sharpness: 4, exposure: 5, framing: 2 },\n embedding: [0.2, 0.9, 0.4, 0.9],\n },\n {\n frame_id: \"3\",\n video_id: \"clip-2\",\n caption: \"a robot arm holding a cup steady\",\n stats: { motion: 2, sharpness: 3, exposure: 3, framing: 4 },\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imageBytes = await readFile(\"data/sir-lancelot.jpg\");\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"id\", new arrow.Utf8()),\n new arrow.Field(\"description\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"vector\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n id: \"lancelot\",\n description: \"Portrait of Sir Lancelot\",\n image: imageBytes,\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"character_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n";
export const TsQuickstartMultimodalBytes = "const arrow = await import(\"apache-arrow\");\nconst { readFile } = await import(\"node:fs/promises\");\n\nconst imageBytes = await readFile(\"data/sample-frame.jpg\");\nconst imageSchema = new arrow.Schema([\n new arrow.Field(\"frame_id\", new arrow.Utf8()),\n new arrow.Field(\"caption\", new arrow.Utf8()),\n new arrow.Field(\"image\", new arrow.Binary()),\n new arrow.Field(\n \"embedding\",\n new arrow.FixedSizeList(\n 4,\n new arrow.Field(\"item\", new arrow.Float32(), true),\n ),\n ),\n]);\nconst imageData = lancedb.makeArrowTable(\n [\n {\n frame_id: \"thumb-1\",\n caption: \"Sample frame thumbnail\",\n image: imageBytes,\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n ],\n { schema: imageSchema },\n);\nconst multimodalTable = await db.createTable(\n \"frame_images\",\n imageData,\n { mode: \"overwrite\" },\n);\n";
export const TsQuickstartQueryFeature = "const features = await table\n .query()\n .select([\"name\", \"role\", \"power_score\"])\n .toArray();\nconsole.table(features);\n";
export const TsQuickstartQueryFeature = "const features = await table\n .query()\n .select([\"frame_id\", \"video_id\", \"quality_score\"])\n .toArray();\nconsole.table(features);\n";
export const TsQuickstartVectorSearch1 = "// Search for examples similar to a \"wise magical advisor\"\nconst queryVector = [0.2, 0.8, 0.4, 0.9];\n\nconst result = await table\n .search(queryVector)\n .select([\"name\", \"role\", \"description\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n";
export const TsQuickstartVectorSearch1 = "// Search for frames similar to \"a gripper closing on an object\"\nconst queryVector = [0.2, 0.8, 0.4, 0.9];\n\nconst result = await table\n .search(queryVector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .toArray();\nconsole.table(result);\n";
export const RsQuickstartAddData = "let more_data = vec![Character {\n id: \"4\".to_string(),\n name: \"Morgana\".to_string(),\n role: \"Sorceress\".to_string(),\n description: \"Powerful sorceress of Avalon.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 4,\n },\n vector: [0.3, 0.9, 0.6, 0.8],\n}];\ntable\n .add(characters_to_reader(schema.clone(), &more_data))\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartAddData = "let more_data = vec![Frame {\n frame_id: \"4\".to_string(),\n video_id: \"clip-2\".to_string(),\n caption: \"a gripper lowering a cup onto the table\".to_string(),\n stats: Stats {\n motion: 5,\n sharpness: 4,\n exposure: 4,\n framing: 2,\n },\n embedding: [0.3, 0.9, 0.6, 0.8],\n}];\ntable\n .add(frames_to_reader(schema.clone(), &more_data))\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartAddFeature = "table\n .add_columns(\n NewColumnTransform::SqlExpressions(vec![(\n \"power_score\".to_string(),\n \"cast(((stats.strength + stats.magic + stats.leadership + stats.wisdom) / 4.0) as float)\"\n .to_string(),\n )]),\n None,\n )\n .await\n .unwrap();\n";
export const RsQuickstartAddFeature = "table\n .add_columns(\n NewColumnTransform::SqlExpressions(vec![(\n \"quality_score\".to_string(),\n \"cast(((stats.motion + stats.sharpness + stats.exposure + stats.framing) / 4.0) as float)\"\n .to_string(),\n )]),\n None,\n )\n .await\n .unwrap();\n";
export const RsQuickstartCreateTable = "use lancedb::database::CreateTableMode;\n\nlet schema = characters_schema();\nlet table = db\n .create_table(\"characters\", characters_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartCreateTable = "use lancedb::database::CreateTableMode;\n\nlet schema = frames_schema();\nlet table = db\n .create_table(\"frames\", frames_to_reader(schema.clone(), &data))\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartCurateWithMetadata = "let curated = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.magic >= 4\")\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n";
export const RsQuickstartCurateWithMetadata = "let curated = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .only_if(\"stats.motion >= 4\")\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{curated:?}\");\n";
export const RsQuickstartData = "let data = vec![\n Character {\n id: \"1\".to_string(),\n name: \"King Arthur\".to_string(),\n role: \"King\".to_string(),\n description: \"Leader of Camelot and wielder of Excalibur.\".to_string(),\n stats: Stats {\n strength: 4,\n magic: 1,\n leadership: 5,\n wisdom: 4,\n },\n vector: [0.7, 0.1, 0.9, 0.7],\n },\n Character {\n id: \"2\".to_string(),\n name: \"Merlin\".to_string(),\n role: \"Wizard\".to_string(),\n description: \"Advisor and prophet with deep magical knowledge.\".to_string(),\n stats: Stats {\n strength: 2,\n magic: 5,\n leadership: 4,\n wisdom: 5,\n },\n vector: [0.2, 0.9, 0.4, 0.9],\n },\n Character {\n id: \"3\".to_string(),\n name: \"Sir Lancelot\".to_string(),\n role: \"Knight\".to_string(),\n description: \"Legendary knight known for courage and combat skill.\".to_string(),\n stats: Stats {\n strength: 5,\n magic: 1,\n leadership: 3,\n wisdom: 3,\n },\n vector: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const RsQuickstartData = "let data = vec![\n Frame {\n frame_id: \"1\".to_string(),\n video_id: \"clip-1\".to_string(),\n caption: \"a robot arm reaching toward a cup\".to_string(),\n stats: Stats {\n motion: 3,\n sharpness: 4,\n exposure: 4,\n framing: 3,\n },\n embedding: [0.7, 0.1, 0.9, 0.7],\n },\n Frame {\n frame_id: \"2\".to_string(),\n video_id: \"clip-1\".to_string(),\n caption: \"a two-finger gripper closing on a cup\".to_string(),\n stats: Stats {\n motion: 5,\n sharpness: 4,\n exposure: 5,\n framing: 2,\n },\n embedding: [0.2, 0.9, 0.4, 0.9],\n },\n Frame {\n frame_id: \"3\".to_string(),\n video_id: \"clip-2\".to_string(),\n caption: \"a robot arm holding a cup steady\".to_string(),\n stats: Stats {\n motion: 2,\n sharpness: 3,\n exposure: 3,\n framing: 4,\n },\n embedding: [0.9, 0.1, 0.5, 0.4],\n },\n];\n";
export const RsQuickstartDefineStruct = "use std::sync::Arc;\n\nuse lancedb::arrow::arrow_array::types::Float32Type;\nuse lancedb::arrow::arrow_array::{\n FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,\n RecordBatchReader, StructArray,\n};\nuse lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};\n\n// Define structs representing the data schema\n#[derive(Debug, Clone)]\nstruct Stats {\n strength: i8,\n magic: i8,\n leadership: i8,\n wisdom: i8,\n}\n\n#[derive(Debug, Clone)]\nstruct Character {\n id: String,\n name: String,\n role: String,\n description: String,\n stats: Stats,\n vector: [f32; 4],\n}\n\nfn characters_schema() -> Arc<Schema> {\n Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"name\", DataType::LargeUtf8, false),\n Field::new(\"role\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(Fields::from(vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n\nfn characters_to_reader(\n schema: Arc<Schema>,\n rows: &[Character],\n) -> Box<dyn RecordBatchReader + Send> {\n let ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.id.as_str()));\n let names = LargeStringArray::from_iter_values(rows.iter().map(|row| row.name.as_str()));\n let roles = LargeStringArray::from_iter_values(rows.iter().map(|row| row.role.as_str()));\n let descriptions =\n LargeStringArray::from_iter_values(rows.iter().map(|row| row.description.as_str()));\n\n let strength = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.strength));\n let magic = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.magic));\n let leadership = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.leadership));\n let wisdom = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.wisdom));\n let stats_fields: Vec<FieldRef> = vec![\n Arc::new(Field::new(\"strength\", DataType::Int8, false)),\n Arc::new(Field::new(\"magic\", DataType::Int8, false)),\n Arc::new(Field::new(\"leadership\", DataType::Int8, false)),\n Arc::new(Field::new(\"wisdom\", DataType::Int8, false)),\n ];\n let stats = StructArray::new(\n stats_fields.into(),\n vec![\n Arc::new(strength),\n Arc::new(magic),\n Arc::new(leadership),\n Arc::new(wisdom),\n ],\n None,\n );\n\n let vectors = FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n rows.iter()\n .map(|row| Some(row.vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n );\n\n let batch = RecordBatch::try_new(\n schema.clone(),\n vec![\n Arc::new(ids),\n Arc::new(names),\n Arc::new(roles),\n Arc::new(descriptions),\n Arc::new(stats),\n Arc::new(vectors),\n ],\n )\n .unwrap();\n\n Box::new(RecordBatchIterator::new(\n vec![Ok(batch)].into_iter(),\n schema,\n ))\n}\n";
export const RsQuickstartDefineStruct = "use std::sync::Arc;\n\nuse lancedb::arrow::arrow_array::types::Float32Type;\nuse lancedb::arrow::arrow_array::{\n FixedSizeListArray, Int8Array, LargeStringArray, RecordBatch, RecordBatchIterator,\n RecordBatchReader, StructArray,\n};\nuse lancedb::arrow::arrow_schema::{DataType, Field, FieldRef, Fields, Schema};\n\n// Define structs representing the data schema\n#[derive(Debug, Clone)]\nstruct Stats {\n motion: i8,\n sharpness: i8,\n exposure: i8,\n framing: i8,\n}\n\n#[derive(Debug, Clone)]\nstruct Frame {\n frame_id: String,\n video_id: String,\n caption: String,\n stats: Stats,\n embedding: [f32; 4],\n}\n\nfn frames_schema() -> Arc<Schema> {\n Arc::new(Schema::new(vec![\n Field::new(\"frame_id\", DataType::LargeUtf8, false),\n Field::new(\"video_id\", DataType::LargeUtf8, false),\n Field::new(\"caption\", DataType::LargeUtf8, false),\n Field::new(\n \"stats\",\n DataType::Struct(Fields::from(vec![\n Arc::new(Field::new(\"motion\", DataType::Int8, false)),\n Arc::new(Field::new(\"sharpness\", DataType::Int8, false)),\n Arc::new(Field::new(\"exposure\", DataType::Int8, false)),\n Arc::new(Field::new(\"framing\", DataType::Int8, false)),\n ])),\n false,\n ),\n Field::new(\n \"embedding\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n ]))\n}\n\nfn frames_to_reader(\n schema: Arc<Schema>,\n rows: &[Frame],\n) -> Box<dyn RecordBatchReader + Send> {\n let frame_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.frame_id.as_str()));\n let video_ids = LargeStringArray::from_iter_values(rows.iter().map(|row| row.video_id.as_str()));\n let captions =\n LargeStringArray::from_iter_values(rows.iter().map(|row| row.caption.as_str()));\n\n let motion = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.motion));\n let sharpness = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.sharpness));\n let exposure = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.exposure));\n let framing = Int8Array::from_iter_values(rows.iter().map(|row| row.stats.framing));\n let stats_fields: Vec<FieldRef> = vec![\n Arc::new(Field::new(\"motion\", DataType::Int8, false)),\n Arc::new(Field::new(\"sharpness\", DataType::Int8, false)),\n Arc::new(Field::new(\"exposure\", DataType::Int8, false)),\n Arc::new(Field::new(\"framing\", DataType::Int8, false)),\n ];\n let stats = StructArray::new(\n stats_fields.into(),\n vec![\n Arc::new(motion),\n Arc::new(sharpness),\n Arc::new(exposure),\n Arc::new(framing),\n ],\n None,\n );\n\n let embeddings = FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n rows.iter()\n .map(|row| Some(row.embedding.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n );\n\n let batch = RecordBatch::try_new(\n schema.clone(),\n vec![\n Arc::new(frame_ids),\n Arc::new(video_ids),\n Arc::new(captions),\n Arc::new(stats),\n Arc::new(embeddings),\n ],\n )\n .unwrap();\n\n Box::new(RecordBatchIterator::new(\n vec![Ok(batch)].into_iter(),\n schema,\n ))\n}\n";
export const RsQuickstartMultimodalBytes = "use lancedb::arrow::arrow_array::BinaryArray;\n\nlet image_bytes = std::fs::read(\"data/sir-lancelot.jpg\").unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"id\", DataType::LargeUtf8, false),\n Field::new(\"description\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"vector\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_vectors = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"lancelot\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Portrait of Sir Lancelot\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n image_vectors\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(\n vec![Ok(image_batch)].into_iter(),\n image_schema,\n));\nlet multimodal_table = db\n .create_table(\"character_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartMultimodalBytes = "use lancedb::arrow::arrow_array::BinaryArray;\n\nlet image_bytes = std::fs::read(\"data/sample-frame.jpg\").unwrap();\n\nlet image_schema = Arc::new(Schema::new(vec![\n Field::new(\"frame_id\", DataType::LargeUtf8, false),\n Field::new(\"caption\", DataType::LargeUtf8, false),\n Field::new(\"image\", DataType::Binary, false),\n Field::new(\n \"embedding\",\n DataType::FixedSizeList(Arc::new(Field::new(\"item\", DataType::Float32, true)), 4),\n false,\n ),\n]));\nlet image_embeddings = [[0.9_f32, 0.1, 0.5, 0.4]];\nlet image_batch = RecordBatch::try_new(\n image_schema.clone(),\n vec![\n Arc::new(LargeStringArray::from_iter_values([\"thumb-1\"])),\n Arc::new(LargeStringArray::from_iter_values([\n \"Sample frame thumbnail\",\n ])),\n Arc::new(BinaryArray::from_iter_values([image_bytes.as_slice()])),\n Arc::new(\n FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(\n image_embeddings\n .iter()\n .map(|vector| Some(vector.iter().copied().map(Some).collect::<Vec<_>>())),\n 4,\n ),\n ),\n ],\n)\n.unwrap();\nlet image_reader: Box<dyn RecordBatchReader + Send> = Box::new(RecordBatchIterator::new(\n vec![Ok(image_batch)].into_iter(),\n image_schema,\n));\nlet multimodal_table = db\n .create_table(\"frame_images\", image_reader)\n .mode(CreateTableMode::Overwrite)\n .execute()\n .await\n .unwrap();\n";
export const RsQuickstartOutputArray = "let result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartOutputArray = "let result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartQueryFeature = "let features = table\n .query()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"power_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n";
export const RsQuickstartQueryFeature = "let features = table\n .query()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"quality_score\".to_string(),\n ]))\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{features:?}\");\n";
export const RsQuickstartVectorSearch1 = "use lancedb::arrow::IntoPolars;\nuse lancedb::query::{ExecutableQuery, QueryBase, Select};\n\n// Search for examples similar to a \"wise magical advisor\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"name\".to_string(),\n \"role\".to_string(),\n \"description\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
export const RsQuickstartVectorSearch1 = "use lancedb::arrow::IntoPolars;\nuse lancedb::query::{ExecutableQuery, QueryBase, Select};\n\n// Search for frames similar to \"a gripper closing on an object\"\nlet query_vector = [0.2, 0.8, 0.4, 0.9];\n\nlet result = table\n .query()\n .nearest_to(&query_vector)\n .unwrap()\n .select(Select::Columns(vec![\n \"frame_id\".to_string(),\n \"video_id\".to_string(),\n \"caption\".to_string(),\n \"_distance\".to_string(),\n ]))\n .limit(2)\n .execute()\n .await\n .unwrap()\n .into_polars()\n .await\n .unwrap();\nprintln!(\"{result:?}\");\n";
docs/web/sql/data-types.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Data Types"
description: "How Lance columns appear to SQL, including the types SQL has no name for."
icon: shapes
keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "json"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
There is no SQL type layer over Lance storage. The table provider hands
DataFusion the dataset's Arrow schema directly, so a column is the same object
to the query engine as it is to the format.
docs/web/sql/ddl/database.mdx
+50
View File
@@ -0,0 +1,50 @@
---
title: "Database"
sidebarTitle: "Database"
description: "ATTACH DATABASE: bind a database to a storage location."
icon: database
keywords: ["attach database", "database", "sql", "ddl"]
availability:
embedded: unavailable
enterprise: available
---
A database is the catalog's top-level container: it holds
[namespaces](/sql/ddl/namespace), which hold tables, views and other objects.
`ATTACH DATABASE` binds a name to a storage location so its objects become
reachable.
## ATTACH DATABASE {#attach}
```sql
ATTACH DATABASE analytics WITH (location = 'analytics')
ATTACH DATABASE IF NOT EXISTS analytics WITH (location = 'analytics')
```
`location` is the only supported option, and the only way to say where a
database lives: a relative subdirectory, never an absolute path or a URI.
<Warning>
An absolute path (`/tmp/db`), an `s3://` or `file://` URI, a Windows path
(`C:/db`), a `..` segment, an empty string and a doubled slash (`sub//db`)
are all refused.
</Warning>
`IF NOT EXISTS` makes the statement a no-op rather than an error when the
database is already attached.
<Note>
A quoted name reaches characters a bare identifier cannot, such as
`` ATTACH DATABASE "team/search" WITH (location = 'team-search') ``.
</Note>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Namespace" icon="folder-tree" href="/sql/ddl/namespace">
CREATE SCHEMA inside an attached database.
</Card>
<Card title="Table" icon="table" href="/sql/ddl/table">
Creating tables once a database and namespace exist.
</Card>
</CardGroup>
docs/web/sql/ddl/function.mdx
+23
View File
@@ -17,6 +17,15 @@ AS 'def f(x): return x * 2'
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
```
A function takes more than one parameter; each one binds a separate column
or literal where the function is called:
```sql
CREATE FUNCTION ratio(numerator float64, denominator float64) RETURNS float64
LANGUAGE python
AS 'def ratio(numerator, denominator): return numerator / denominator'
```
Parameter and result types are Arrow type names, not SQL ones: `int64`,
`float64`, `utf8`, `bool`, `date32`, and lists such as `list<float32>` or
`fixed_size_list<float32, 768>`. `BIGINT` is refused. A parameter accepts nulls
@@ -51,6 +60,20 @@ version is unaffected.
the function is baked into an image before the version becomes usable.
</Note>
## Call it {#call}
Once registered, a function is callable by name in any query, the same as a
built-in scalar function:
```sql
SELECT id, f(x) AS doubled FROM my_table
```
This is also the fastest way to check a new registration before binding it to
a column: call it over a few rows, or over literal arguments, before
committing to [a computed column](/sql/ddl/table#computed-column) that fills
from it.
## SHOW FUNCTIONS {#show}
```sql
docs/web/sql/ddl/grant.mdx
+77
View File
@@ -0,0 +1,77 @@
---
title: "Grant"
sidebarTitle: "Grant"
description: "GRANT and REVOKE: privileges on Secrets and Functions."
icon: user-lock
keywords: ["grant", "revoke", "privileges", "authorization", "sql", "ownership"]
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
```sql
GRANT <privilege>[, ...] ON { SECRET | FUNCTION } <name> TO <grantee>[, ...]
GRANT <privilege>[, ...] ON SCHEMA <namespace> TO <grantee>[, ...]
REVOKE <privilege>[, ...] ON ... FROM <grantee>[, ...]
```
```sql
GRANT USAGE, UPDATE ON SECRET api_key TO ROLE readers, writers
GRANT CREATE FUNCTION ON SCHEMA analytics TO ROLE data_science
REVOKE USAGE ON FUNCTION embed FROM ROLE readers
```
## Privileges {#privileges}
| Privilege | Allows |
| --- | --- |
| `OWNERSHIP` | everything on the object |
| `USAGE` | using it: binding a Secret's value, calling a Function |
| `UPDATE` | changing it |
| `CREATE SECRET` | creating a Secret inside a `SCHEMA` |
| `CREATE FUNCTION` | creating a Function inside a `SCHEMA` |
<Warning>
`OWNERSHIP` cannot be revoked, only granted to someone else, which transfers
it: an object always has exactly one owner. `GRANT OWNERSHIP` is refused with
more than one grantee for the same reason.
</Warning>
These are a subset of the full privilege model; see
[Authorization](/governance/authorization#privileges) for privileges on
tables, views and namespaces, which this statement does not grant.
## Grantees {#grantees}
```sql
<grantee> ::= [ ROLE ] <role> | [ { USER | GROUP } ] { <name> | ID '<id>' }
```
```sql
GRANT USAGE ON SECRET api_key TO ROLE readers, USER alice, GROUP ID 'g1'
```
A grantee is a role, or a user or group named by its identity-directory name
or its directory ID. A grantee with no leading `ROLE`/`USER`/`GROUP` keyword
takes the kind of the one before it, so `TO ROLE a, b` names two roles --
but the first grantee in the list always needs its keyword. An unquoted name
is folded to lowercase before it is looked up; a quoted name is looked up
exactly as written.
## Naming the target {#targets}
A `SECRET` or `FUNCTION` target can be qualified with its namespace and
database, the same as anywhere else: `SECRET analytics.api_key`,
`SECRET db1.analytics.api_key`. A `SCHEMA` target is the namespace itself.
There is no `TABLE` target: table privileges are granted through
[Authorization](/governance/authorization), not this statement.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Authorization" icon="user-shield" href="/governance/authorization">
The full privilege model, and how creation privileges resolve upward.
</Card>
<Card title="Namespace" icon="folder-tree" href="/sql/ddl/namespace">
Creating the schemas these privileges apply to.
</Card>
</CardGroup>
docs/web/sql/ddl/index.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Index"
description: "CREATE INDEX and DROP INDEX: one column, with the algorithm chosen by USING."
icon: bolt
keywords: ["create index", "drop index", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE INDEX {#create}
```sql
docs/web/sql/ddl/job.mdx
+4 -3
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Job"
description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL."
icon: hourglass-half
keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A [job](/basics/jobs) is not a catalog object (it has no schema and is not
namespaced), but it has its own SQL statements, grouped here the same way the
catalog objects are.
@@ -32,7 +33,7 @@ SHOW ERRORS ON documents
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
```
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports
fewer filled rows than you expected.
**Errors and limits**
docs/web/sql/ddl/materialized-view.mdx
+59 -8
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Materialized View"
description: "CREATE MATERIALIZED VIEW and SHOW MATERIALIZED VIEWS; refreshing is a REFRESH statement."
icon: table
keywords: ["materialized view", "create materialized view", "show materialized views", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A materialized view stores a query's result as rows, kept current by a refresh.
It is the counterpart to a [view](/sql/ddl/view), which stores only the query.
@@ -28,6 +29,29 @@ Once created, a materialized view can be queried like a table. Use the
statements below to list, refresh and drop it. For SDK methods, see
[Materialized views](/management/materialized-view#create).
{/*
Draft for automatic materialized-view refresh. Enable this section, and
"ALTER MATERIALIZED VIEW" below, when the server feature is released.
Set scheduling options at creation, continuing the join and chained
Functions from [Functions](/management/function#chain):
```sql
CREATE MATERIALIZED VIEW frames_captioned AS
SELECT f.frame_id, extract_frame(v.video, f.ts) AS jpeg,
caption(extract_frame(v.video, f.ts)) AS caption
FROM videos v JOIN frames f ON v.id = f.video_id
WHERE f.selected = true
WITH (auto_refresh_enabled = false, auto_refresh_interval_seconds = 5)
```
`auto_refresh_enabled` and `auto_refresh_interval_seconds` work the same way
here as for a [computed column](/sql/ddl/table#automatic-refresh-options):
the interval is a minimum pacing after the last successful refresh, and
automatic execution needs the same server-side switch enabled. Omitting
`WITH` leaves automatic refresh at the server's default.
*/}
## SHOW MATERIALIZED VIEWS {#show}
```sql
@@ -45,9 +69,37 @@ on a `db://` connection.
REFRESH MATERIALIZED VIEW chunks
```
See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax,
See [REFRESH MATERIALIZED VIEW](/sql/dml/refresh-materialized-view) for the full syntax,
and compatibility notes for older server versions.
{/*
Draft for automatic materialized-view refresh. Enable this section when the
server feature is released; see the note on "CREATE MATERIALIZED VIEW" above.
## ALTER MATERIALIZED VIEW {#alter}
```sql
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = true;
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_interval_seconds = 300;
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = false;
```
The whole view toggles together; there is no per-output granularity the way
a multi-output Function column has one setting per output. SET takes effect
immediately and does not itself run a refresh:
- **Disabling** prevents new automatic refreshes. An inflight one may still
finish, and `REFRESH MATERIALIZED VIEW` keeps working manually.
- **Re-enabling catches up the entire backlog in one pass**, the same as a
computed column's [automatic refresh](/sql/ddl/table#automatic-refresh-options):
the next scheduler tick picks up every source change accumulated since the
view was disabled, immediately, with no further write needed to trigger it.
A materialized view's *initial* population, at `CREATE` or when the server
runs it, happens regardless of `auto_refresh_enabled`; only refreshes after
that are gated by the setting.
*/}
## Dropping one {#drop}
A materialized view **is** a table, so it is dropped with
@@ -55,9 +107,8 @@ A materialized view **is** a table, so it is dropped with
stored rows, unlike dropping a [view](/sql/ddl/view#drop).
<Note>
**Enterprise job type:** `mat_view_refresh` for a refresh, or `sql` on a
deployment that rebuilds views through SQL. Dropping one runs the same job as
dropping any table, `drop_table` or `drop_object`.
**Enterprise job type for a refresh:** `sql`. Dropping, by contrast, runs the
same job as dropping any table, `drop_table` or `drop_object`.
</Note>
## What to read next {#next}
@@ -66,7 +117,7 @@ dropping any table, `drop_table` or `drop_object`.
<Card title="Materialized View" icon="table" href="/management/materialized-view">
Creating one from the SDK.
</Card>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
How a refresh works, locally and in LanceDB Enterprise.
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
How a refresh works, in LanceDB Enterprise.
</Card>
</CardGroup>
docs/web/sql/ddl/namespace.mdx
+6 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Namespace"
description: "CREATE SCHEMA and DROP SCHEMA: the SQL spelling of namespace DDL."
icon: folder-tree
keywords: ["namespace", "schema", "create schema", "drop schema", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA`, the
standard's name for the same concept.
@@ -46,6 +47,9 @@ default) states that requirement rather than changing it.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Database" icon="database" href="/sql/ddl/database">
Attaching the database a namespace lives in.
</Card>
<Card title="Namespace" icon="folder-tree" href="/management/namespace">
The SDK equivalent, and how a namespace fits the catalog hierarchy.
</Card>
docs/web/sql/ddl/table.mdx
+27 -10
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Table"
description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE."
icon: table
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE TABLE {#create}
```sql
@@ -50,7 +51,7 @@ ALTER TABLE t ADD COLUMN doubled AS value * 2
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
The column is committed empty and the expression is stored, not evaluated:
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
costs. [`REFRESH COLUMN`](/sql/dml/refresh-column) fills it.
{/*
Draft for automatic computed-column refresh. Enable this section when the
@@ -74,19 +75,35 @@ and is minimum pacing after a successful refresh of that output; unchanged
inputs do not force work. A new column is immediately eligible for its first
refresh. The server's separate automatic-refresh switch must also be enabled.
These options also work beside a Function binding's `version` and `branch`:
These options also work beside a Function binding's `version` and `branch`,
continuing the [`caption` Function](/management/function#chain) from Functions:
```sql
ALTER TABLE t ADD COLUMN (label, score) AS classify(text)
ALTER TABLE frames ADD COLUMN caption AS caption(extract_frame(video, ts))
WITH (version = 'fv_1', auto_refresh_enabled = true,
auto_refresh_interval_seconds = 300);
ALTER TABLE t ALTER COLUMN score SET auto_refresh_enabled = false;
ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_enabled = false;
```
Declaration settings initialize every output in the group; later SET commands
change one output. Disabling automatic refresh prevents new submissions while
allowing inflight jobs to finish and explicit REFRESH COLUMN to run. Column
SUSPEND/RESUME syntax is not supported.
change one output. SET takes effect immediately and does not itself read or
write any row:
- **Disabling** prevents new automatic submissions. Inflight jobs may still
finish, explicit `REFRESH COLUMN` keeps working, and rows written while
disabled commit with no computed value -- they do not queue anywhere
separate.
- **Re-enabling catches up the entire backlog in one pass.** The next
scheduler tick processes every row left unfilled since the column was
disabled, immediately, with no further write required to trigger it: a
column disabled for an hour and re-enabled fills everything that
accumulated during that hour, not just rows written after the moment it
is re-enabled.
Column SUSPEND/RESUME syntax is not supported; `SET auto_refresh_enabled` is
the only toggle. The current setting is in the column's own field metadata,
`computed_column.auto_refresh_enabled`, readable from `DESCRIBE` or the
SDK's `table.schema` without submitting a job to ask.
*/}
{/*
@@ -148,7 +165,7 @@ ALTER TABLE t SET RTE WITH (maintained_indexes = '')
ALTER TABLE t UNSET RTE
```
The [realtime engine](/tables/realtime-engine) puts a write-ahead log in front
The [real time engine](/tables/realtime-engine) puts a write-ahead log in front
of the table, making small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `bucket` sharding names its `column`, which must be
docs/web/sql/ddl/view.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "View"
description: "CREATE VIEW and DROP VIEW: a saved query, evaluated when read."
icon: eye
keywords: ["create view", "drop view", "sql", "catalog"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A view stores a **query**, not rows. Reading it runs the query against current
data, so a view is never stale, and never faster than the query it wraps. It
is the counterpart to a [materialized view](/sql/ddl/materialized-view), which
docs/web/sql/dml/delete.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Delete"
description: "DELETE FROM ... WHERE: rows are marked, not erased."
icon: trash
keywords: ["delete", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
```sql
DELETE FROM documents WHERE created_at < TIMESTAMP '2025-01-01 00:00:00'
```
docs/web/sql/dml/insert.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Insert"
description: "INSERT INTO, including INSERT ... SELECT to move rows server-side."
icon: plus
keywords: ["insert", "insert into", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
```sql
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
docs/web/sql/dml/refresh.mdx → docs/web/sql/dml/refresh-column.mdx
+14 -32
View File
@@ -1,19 +1,18 @@
---
title: "REFRESH"
sidebarTitle: "REFRESH"
description: "Filling computed columns and bringing materialized views up to date."
title: "REFRESH COLUMN"
sidebarTitle: "REFRESH COLUMN"
description: "Filling a computed column: idempotent, scoped, and resumable."
icon: arrows-rotate
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
keywords: ["refresh column", "backfill", "computed column"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
LanceDB extensions with no DataFusion equivalent. They are what makes
A LanceDB extension with no DataFusion equivalent. It is what makes
[feature engineering](/feature-engineering) reachable from SQL: a column
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
## REFRESH COLUMN {#refresh-column}
```sql
REFRESH COLUMN <column> ON [TABLE] <table>
[WHERE <predicate>]
@@ -33,7 +32,7 @@ can recompute values the failed run had already written.
`WHERE` scopes the refresh, which is how you fill a slice and check it before
committing to the whole table.
### Options {#refresh-column-options}
## Options {#options}
| Option | Meaning |
| --- | --- |
@@ -62,39 +61,22 @@ memory and GPU options there.
</Note>
*/}
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
```sql
REFRESH MATERIALIZED VIEW <name>
```
```sql
REFRESH MATERIALIZED VIEW chunks
```
A refresh updates the view's stored result. Refresh views in dependency order,
starting with their source views.
<Note>
Some older server versions accepted `WITH (full = true)` and `CASCADE`.
Those options are not accepted by the SQL implementation described here.
Use the basic statement above; do not assume that the options accepted by a
local SDK's `refresh()` method are also SQL options.
</Note>
## Watching a refresh {#watching}
Both statements submit work that outlives the call. Track it with
A refresh submits work that outlives the call. Track it with
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
[`SHOW ERRORS`](/sql/ddl/job#errors).
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
The same operation from the SDKs, and what a refresh guarantees.
</Card>
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
Declaring the columns this fills.
</Card>
<Card title="REFRESH MATERIALIZED VIEW" icon="arrows-rotate" href="/sql/dml/refresh-materialized-view">
Bringing a view's stored result up to date, instead of a column.
</Card>
</CardGroup>
docs/web/sql/dml/refresh-materialized-view.mdx
+50
View File
@@ -0,0 +1,50 @@
---
title: "REFRESH MATERIALIZED VIEW"
sidebarTitle: "REFRESH MATERIALIZED VIEW"
description: "Bringing a materialized view's stored result up to date."
icon: arrows-rotate
keywords: ["refresh materialized view", "cascade"]
availability:
embedded: unavailable
enterprise: available
---
A LanceDB extension with no DataFusion equivalent.
```sql
REFRESH MATERIALIZED VIEW <name>
```
```sql
REFRESH MATERIALIZED VIEW chunks
```
A refresh updates the view's stored result. Refresh views in dependency order,
starting with their source views.
<Note>
Some older server versions accepted `WITH (full = true)` and `CASCADE`.
Those options are not accepted by the SQL implementation described here.
Use the basic statement above; do not assume that the options accepted by a
local SDK's `refresh()` method are also SQL options.
</Note>
## Watching a refresh {#watching}
A refresh submits work that outlives the call. Track it with
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
[`SHOW ERRORS`](/sql/ddl/job#errors).
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Materialized View" icon="table" href="/sql/ddl/materialized-view">
Creating the view this refreshes.
</Card>
<Card title="Refresh a materialized view" icon="layer-group" href="/feature-engineering/refresh-materialized-view">
The same operation from the SDKs, and automatic refresh.
</Card>
<Card title="REFRESH COLUMN" icon="arrows-rotate" href="/sql/dml/refresh-column">
Filling a computed column, instead of a view.
</Card>
</CardGroup>
docs/web/sql/dml/update.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Update"
description: "UPDATE ... SET ... WHERE, and why it does not recompute derived columns."
icon: pen
keywords: ["update", "sql"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
```sql
UPDATE documents SET title = upper(title) WHERE category = 'film'
```
docs/web/sql/dql/aggregation.mdx
+3 -2
View File
@@ -4,10 +4,11 @@ sidebarTitle: "Aggregation"
description: "GROUP BY, HAVING, and the aggregate function set."
icon: layer-group
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
availability:
embedded: unavailable
enterprise: available
---
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
Use SQL aggregates to summarize records, either across a whole table or
within groups. Common examples include counts by category, totals and averages.
docs/web/sql/dql/changes.mdx
+76
View File
@@ -0,0 +1,76 @@
---
title: "Changes since a version"
sidebarTitle: "Changes"
description: "CHANGES: read the rows appended to a table since an earlier version."
icon: clock-rotate-left
keywords: ["changes", "append", "incremental", "cdc", "version"]
availability:
embedded: unavailable
enterprise: available
---
`CHANGES` reads the rows appended to a table since an earlier
[version](/management/table#versioning), without rescanning rows that were
already there:
```sql
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
```
This returns the rows appended after version 42, up to the table's current
version at the time the query runs.
## Strictly append-only {#strict}
The clause is named for what it requires, not just what it does: every
change to the table between the starting version and the current one must
be an append. If any of them inserted with changes other than a pure append
-- an update, a delete, a merge -- the query is refused rather than silently
returning an incomplete or wrong result. Re-run it with a later starting
version that is itself after the non-append write, or read the table in full
instead of incrementally.
`INFORMATION => APPEND_ONLY` is the only supported value: it names what kind
of history the clause is willing to read, not a choice among several. `AT
(VERSION => ...)` takes a positive integer; there is no `AT (TIMESTAMP =>
...)` form for this clause, unlike other version qualifiers.
## Where the clause can go {#placement}
It attaches to a table in `FROM`, including inside a subquery, a CTE, a join,
a set operation, or `EXPLAIN`:
```sql
WITH recent AS (
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
)
SELECT count(*) FROM recent
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 10) a
JOIN events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 20) b
ON a.id = b.id
```
Each occurrence of a table keeps its own starting version, even when the same
table is referenced more than once in one statement.
## Unsupported forms {#refused}
| | |
| --- | --- |
| `CHANGES (INFORMATION => DEFAULT)` | only `APPEND_ONLY` is supported |
| `AT (TIMESTAMP => ...)` | only `AT (VERSION => ...)` is supported |
| `AT (VERSION => 0)` | the starting version must be positive |
| `... END (VERSION => ...)` | not supported; the current version is always the end |
| `CHANGES ... TABLESAMPLE (...)` | cannot be combined on the same table reference |
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Table" icon="table" href="/sql/ddl/table">
CREATE, ALTER, CLONE and versioning basics.
</Card>
<Card title="Select" icon="magnifying-glass" href="/sql/dql/select">
The rest of the query grammar this clause attaches to.
</Card>
</CardGroup>
docs/web/sql/dql/dedupe.mdx
+99
View File
@@ -0,0 +1,99 @@
---
title: "DEDUPE"
sidebarTitle: "DEDUPE"
description: "The dedupe() table function: one representative row per near-duplicate group, scanned over the index."
icon: layer-group
keywords: ["dedupe", "vector_dedup", "sql", "deduplication", "near-duplicate", "table function"]
availability:
embedded: unavailable
enterprise: available
---
`dedupe` scans a vector column's [index](/indexing/vector-index) partitions
for rows closer than a distance threshold, resolves the overlapping pairs it
finds into duplicate groups, and returns one representative row per group --
without a full pairwise scan or a shuffle. See
[`duplicate_pairs`](/sql/dql/duplicate-pairs) to inspect the pairs themselves
before committing to this resolution.
```sql
SELECT * FROM dedupe(<table>, <vector_column>,
distance_threshold => <float>
[, resolution => 'connected_components' | 'direct_representative']
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
)
```
```sql
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
```
<Note>
Earlier releases spelled this `vector_dedup`; that name is rejected on
current deployments.
</Note>
## Input {#input}
| Argument | Type | Required | Meaning |
| --- | --- | --- | --- |
| table name | string literal | yes | the table to scan |
| vector column | string literal | yes | the indexed vector column to compare |
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
| `resolution` | `'connected_components'` (default) \| `'direct_representative'` | no | how overlapping pairs group together -- see [Resolution](#resolution) |
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
| `seed` | int | no | random seed when `selection => 'random'` |
`id_column` is refused on `dedupe`: it has no separate pair-identifier output
to project onto, only [`duplicate_pairs`](/sql/dql/duplicate-pairs#input)
accepts it.
## Output {#output}
One row per duplicate group, **in the table's own schema** -- every column of
`<table>`, including a BlobV2 column, comes through unchanged on the
representative row Lance selected for that group. Unlike
[`duplicate_pairs`](/sql/dql/duplicate-pairs#output), there is no fixed
`dedupe`-specific schema to document: the output schema is the source
table's schema.
Because the result is already shaped like the table, it composes directly
into a new table or a materialized view:
```sql
CREATE MATERIALIZED VIEW frames_deduped AS
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
```
## Resolution {#resolution}
`resolution` chooses how overlapping pairs group together:
| `resolution` | |
| --- | --- |
| `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together |
| `direct_representative` | only pairs within the threshold of each other resolve together |
## Sampling partitions {#sampling}
`selection` scopes the scan to a sample of partitions instead of all of them,
the same `all` (default) / `first` / `random` choice as
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
duplicate rate on a large table before scanning it in full:
```sql
SELECT * FROM dedupe('frames', 'vec',
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
```
## What to read next {#next}
<CardGroup cols={2}>
<Card title="DUPLICATE_PAIRS" icon="clone" href="/sql/dql/duplicate-pairs">
Inspect the close pairs themselves instead of the resolved groups.
</Card>
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
When to reach for this instead of exact-key deduplication.
</Card>
</CardGroup>
docs/web/sql/dql/duplicate-pairs.mdx
+97
View File
@@ -0,0 +1,97 @@
---
title: "DUPLICATE_PAIRS"
sidebarTitle: "DUPLICATE_PAIRS"
description: "The duplicate_pairs() table function: every close pair in a vector column, scanned over the index."
icon: clone
keywords: ["duplicate_pairs", "vector_duplicate_pairs", "sql", "deduplication", "near-duplicate", "table function"]
availability:
embedded: unavailable
enterprise: available
---
`duplicate_pairs` scans a vector column's [index](/indexing/vector-index)
partitions for every pair of rows closer than a distance threshold, without a
full pairwise scan or a shuffle. It returns one row per close pair, not one
row per source row -- use it to inspect what would be collapsed before
committing to [`dedupe`](/sql/dql/dedupe), which resolves these same pairs
into groups.
```sql
SELECT * FROM duplicate_pairs(<table>, <vector_column>,
distance_threshold => <float>
[, id_column => <column>]
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
)
```
```sql
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02)
```
<Note>
Earlier releases spelled this `vector_duplicate_pairs`; that name is rejected
on current deployments.
</Note>
## Input {#input}
| Argument | Type | Required | Meaning |
| --- | --- | --- | --- |
| table name | string literal | yes | the table to scan |
| vector column | string literal | yes | the indexed vector column to compare |
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
| `id_column` | column name | no | project this column's value instead of the internal row ID -- see [Application-level identifiers](#id-column) |
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
| `seed` | int | no | random seed when `selection => 'random'` |
`resolution` is refused on `duplicate_pairs`: a pair has no notion of a group,
only [`dedupe`](/sql/dql/dedupe#input) accepts it.
## Output {#output}
One row per close pair found, in a fixed schema independent of the source
table:
| Column | Type | Meaning |
| --- | --- | --- |
| `row_id_a` | uint64 | internal row ID of the first row in the pair, the same identifier as the [`_rowid`](/sql/data-types#pseudo-columns) pseudo-column |
| `row_id_b` | uint64 | internal row ID of the second row in the pair |
| `distance` | float32 | distance between the pair's vectors, at or below `distance_threshold` |
| `segment_id` | int64 | which index segment the pair was found in |
| `partition_id` | int64 | which index partition the pair was found in |
| `dataset_version` | int64 | the dataset version the scan ran against |
### Application-level identifiers {#id-column}
A row ID is internal and not stable across compaction. Pass `id_column`, a
top-level scalar column such as a primary key, to project its value instead:
`row_id_a`/`row_id_b` are replaced by `id_a`/`id_b`, typed like `id_column`.
```sql
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02, id_column => 'frame_id')
```
## Sampling partitions {#sampling}
`selection` scopes the scan to a sample of partitions instead of all of them,
the same `all` (default) / `first` / `random` choice as
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
distance threshold or duplicate rate on a large table before scanning it in
full:
```sql
SELECT * FROM duplicate_pairs('frames', 'vec',
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
```
## What to read next {#next}
<CardGroup cols={2}>
<Card title="DEDUPE" icon="layer-group" href="/sql/dql/dedupe">
Resolve these same pairs into one representative row per group.
</Card>
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
When to reach for this instead of exact-key deduplication.
</Card>
</CardGroup>

Some files were not shown because too many files have changed in this diff Show More