rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 14:52:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: declare where management, jobs, feature engineering and curation apply

Browse Source 
Databases, namespaces, jobs and data evolution work in both offerings.
Materialized views, computed columns and column refresh work in the
embedded library and vary on a deployment, which has to enable them.
Views and functions exist only on a deployment that enables them, and
aggregation, joins, windowing and sampling are SQL, so they need one
with SQL enabled.

Sections now say where they differ: views have SDK methods on a
deployment, dropping a function needs a further setting, a local job
handle runs in-process and has no history, create_index and
refresh_column behave differently against a deployment, and the
overviews no longer call computed columns and materialized views
available everywhere.
This commit is contained in:
Brendan Clement
2026-10-08 21:31:42 -07:00
parent 6a1645116e
commit 6398ff3629
17 changed files with 161 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/basics/catalog.mdx
+6 -1
View File
@@ -27,6 +27,10 @@ resolves the same names to the same tables.
| [View](/management/view) | A saved query. Computed when read, not stored. |
| [Function](/management/function) | Registered code, applied to produce column values. |
Views and functions exist only on an Enterprise deployment that enables them.
Materialized views also exist in the embedded library, and a deployment has to
enable them too.
```
database
└── namespace
@@ -63,7 +67,8 @@ and it inherits down the path: reaching a table requires `USAGE` on the
namespace containing it and on the database containing that.
That is the practical reason to put a boundary in the hierarchy -- a namespace
per team gives you something to grant on. See
per team gives you something to grant on. A deployment checks these grants only
when its administrator has configured it to. See
[Authorization](/governance/authorization).
Embedded LanceDB has no such layer. Access is whatever the object store enforces
docs/web/basics/jobs.mdx
+17 -5
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Job"
description: "Work that outlives the call that started it: submitting, waiting, inspecting, cancelling."
icon: hourglass-half
keywords: ["job", "async", "background", "events", "cancel", "refresh"]
availability:
oss: available
enterprise: available
summary: >-
Long-running operations return a job handle in the embedded library and on
a deployment. Listing jobs, reopening them by id and reading their events
need a deployment, where the work runs on the server.
---
Some operations take longer than a request should. Building an index over a
@@ -35,7 +42,9 @@ a **job** immediately.
<Note>
The blocking form is not a different operation -- it submits the same job and
waits. Reach for the `_async` form when you want to do something else meanwhile,
run several at once, or survive the client going away.
run several at once, or survive the client going away. On a deployment,
`create_index` returns once the build is submitted unless you pass
`wait_timeout`, and `refresh_column` has no blocking form.
</Note>
## Submitting and waiting {#waiting}
@@ -155,8 +164,9 @@ in-process.
## Finding and cancelling jobs {#managing}
The connection lists and reopens jobs, so a handle lost to a restarted process
is recoverable from its id.
On a deployment, the connection lists and reopens jobs, so a handle lost to a
restarted process is recoverable from its id. A local database keeps no jobs to
list, and these calls raise there.
<CodeGroup>
```python Python icon="python"
@@ -189,8 +199,10 @@ parsing it or storing it to resume the job later is not supported.
A local database has no server to run anything, so its operations run in your
process. They still return a handle, so the same code works against both -- but
one with nothing behind it: `id` is `None`, `status()` reports `"finished"`,
`wait()` returns immediately, `cancel()` does nothing, and `events()` raises.
one with no server behind it: `id` is `None` and `events()` raises. An operation
still running in your process reports `"running"`, `wait()` waits for it and
`cancel()` stops it; one that finished before the call returned, such as a local
`drop_table_async`, reports `"finished"` at once.
This is the difference to expect when moving between embedded and deployed --
see [SDKs](/basics/sdk).
docs/web/curation/aggregate.mdx
+6 -3
View File
@@ -4,11 +4,14 @@ sidebarTitle: "Aggregate"
description: "Counting, grouping and summarizing — finding out what your data actually contains."
icon: chart-column
keywords: ["aggregate", "group by", "class balance", "count", "profiling", "distinct"]
availability:
oss: unavailable
enterprise: varies
summary: >-
Aggregation is SQL, which runs on an Enterprise deployment with SQL
enabled; the embedded library does not run SQL.
---
<Badge color="red">Enterprise</Badge> Aggregation is SQL, and
[SQL needs a deployment](/basics/sql#sql-is-remote).
Before curating anything, find out what you have. Most of the surprises in a
training set — a class that is 90% of the rows, a column that is half null, a
"unique" id that is not — are one `GROUP BY` away from being visible.
docs/web/curation/index.mdx
+5 -2
View File
@@ -15,8 +15,8 @@ reach for it, and links to the [SQL reference](/sql) for the complete syntax.
<Note>
Most curation is analytical -- grouping, joining, ranking -- which the query
builders do not do. It is therefore SQL, and SQL needs a deployment. See
[SQL](/basics/sql#sql-is-remote).
builders do not do. It is therefore SQL, and SQL needs a deployment with its
SQL endpoint enabled. See [SQL](/basics/sql#sql-is-remote).
</Note>
## The four questions {#questions}
@@ -52,3 +52,6 @@ you ended up with.
A curated result you want to keep is a
[materialized view](/management/materialized-view) -- stored, refreshed, and
readable by a [data loader](/training/data-loading) like any other table.
Keeping a joined or windowed result this way takes SQL's
`CREATE MATERIALIZED VIEW` on a deployment that enables materialized views; the
SDK's `create_materialized_view` reads a single table.
docs/web/curation/join.mdx
+10 -6
View File
@@ -4,11 +4,14 @@ sidebarTitle: "Join"
description: "Bringing labels, metadata and scores together with the rows they describe."
icon: link
keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
availability:
oss: unavailable
enterprise: varies
summary: >-
Joins are SQL, which runs on an Enterprise deployment with SQL enabled;
the embedded library does not run SQL.
---
<Badge color="red">Enterprise</Badge> Joins are SQL, and
[SQL needs a deployment](/basics/sql#sql-is-remote).
Curation usually means combining things that were produced separately — images
in one table, human labels in another, model scores in a third. A join is how
they become one dataset.
@@ -62,9 +65,10 @@ ORDER BY hits DESC
```
<Note>
Vector search has no SQL form, so a similarity result cannot be joined this way.
Run the search in an SDK, then join over its output — or write the result to a
table first.
Whether SQL can express vector search
[depends on the deployment](/basics/sql#no-vector-sql). Where it cannot, run the
search in an SDK, then join over its output — or write the result to a table
first.
</Note>
## Making it fast {#performance}
docs/web/curation/sampling.mdx
+8 -1
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Sampling"
description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible."
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
availability:
oss: unavailable
enterprise: varies
summary: >-
`TABLESAMPLE` is SQL, which runs on an Enterprise deployment with SQL
enabled. Shuffled reads for training work in both through the SDK; see
[Shuffling for training](#training).
---
Most curation starts with a subset. A question you can answer in ten seconds on
@@ -79,7 +86,7 @@ smaller result.
## Shuffling for training {#training}
For a training loop rather than a query, the SDKs read a table in a shuffled
order without SQL at all:
order without SQL at all, in the embedded library and on a deployment alike:
```python
from lancedb.permutation import Permutation
docs/web/curation/windowing.mdx
+6 -3
View File
@@ -4,11 +4,14 @@ sidebarTitle: "Windowing"
description: "Picking one row per thing, ranking within a group, and splitting a dataset."
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
availability:
oss: unavailable
enterprise: varies
summary: >-
Window functions are SQL, which runs on an Enterprise deployment with
SQL enabled; the embedded library does not run SQL.
---
<Badge color="red">Enterprise</Badge> Window functions are SQL, and
[SQL needs a deployment](/basics/sql#sql-is-remote).
A window function computes across a set of related rows and returns a value
**per row** — nothing is collapsed. That is what makes it the tool for "which of
these do I keep", where an aggregate would throw away the rows you are choosing
docs/web/feature-engineering/add-computed-column.mdx
+10 -2
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Computed Columns"
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
icon: table-columns
keywords: ["computed columns", "add_columns", "transforms", "schema"]
availability:
oss: available
enterprise: varies
summary: >-
Computed columns work in the embedded library. On a deployment they are off
by default and have to be enabled there.
---
A computed column is declared with an expression and starts with no values.
@@ -33,7 +39,8 @@ including that you can travel back to before the column existed.
- **SQL expressions in `transforms`** are evaluated *now*, for every row, as
part of the call. Good for a small table, or a column you want populated
immediately. A `FunctionApplication` passed through `transforms` instead
declares empty Function-bound columns; a refresh fills them.
declares empty Function-bound columns, which a refresh fills; that needs a
deployment with Functions enabled, and the embedded library refuses it.
- **`computed`** stores the expression and evaluates nothing. The column is
committed empty and filled later by
[`refresh_column`](/feature-engineering/refresh-column).
@@ -77,7 +84,8 @@ reshape later. Dropping the computed column releases the constraint.
When the value cannot be expressed in SQL -- an embedding, a model call, an
image decode -- the column is filled by a registered
[function](/management/function) instead of an expression. Declaring it looks
the same; what differs is what fills it.
the same; what differs is what fills it. It needs a deployment with Functions
enabled.
## What to read next {#next}
docs/web/feature-engineering/data-evolution.mdx
+9 -1
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Data Evolution"
description: "Derive new columns from data already in the table, without rewriting what is already there."
icon: wand-magic-sparkles
keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
availability:
oss: available
enterprise: available
summary: >-
Adding columns computed from SQL expressions works in the embedded library
and on a deployment.
---
import {
@@ -27,7 +33,7 @@ column -- the existing data, which is the expensive part on a multimodal table,
is never rewritten. See [why that works](/basics/lance#evolution).
<Note>
In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
In SQL, on a deployment that enables computed columns, this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
</Note>
## Set up the example table {#set-up-the-example-table}
@@ -105,6 +111,8 @@ exist when that does not fit:
- A [function](/management/function) fills a column with Python, for values SQL
cannot express -- an embedding, a model call, an image decode.
On a deployment, both have to be enabled there.
## What to read next {#next}
<CardGroup cols={2}>
docs/web/feature-engineering/index.mdx
+4 -4
View File
@@ -17,7 +17,7 @@ versioned, and refreshed like anything else, not a separate system bolted
alongside.
<Note>
Every operation here has a SQL form -- see [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it.
On a deployment, every operation here also has a SQL form -- see [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it.
</Note>
## The shape everything shares {#the-shape}
@@ -46,9 +46,9 @@ idempotent and cheap -- but it will surprise you if you expect a spreadsheet.
| | What it produces | Filled by | Available |
| --- | --- | --- | --- |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | everywhere |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | everywhere |
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | embedded, and deployments that enable it |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | embedded, and deployments that enable it |
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> deployments that enable it |
**Computed columns** are the common case: a column declared with a SQL
expression, filled by [a refresh](/feature-engineering/refresh-column).
docs/web/feature-engineering/refresh-column.mdx
+8 -1
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Column Refresh"
description: "Fill the rows of a computed column that lack values — in process, or as a server-side job."
icon: arrows-rotate
keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"]
availability:
oss: available
enterprise: varies
summary: >-
In the embedded library a refresh runs in your process. On a deployment it
runs as a server job, which has to be enabled there.
---
A [declared column](/feature-engineering/add-computed-column) starts empty. A
@@ -72,7 +78,8 @@ REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
<Badge color="red">Enterprise</Badge>
Against a deployment the refresh runs on the cluster. `refresh_column` raises
Against a deployment that enables computed columns, the refresh runs on the
cluster. `refresh_column` raises
there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can
wait on or poll instead of holding a connection open:
docs/web/management/database.mdx
+17 -8
View File
@@ -4,6 +4,13 @@ sidebarTitle: "Database"
description: "Connecting to LanceDB: the URI forms, their options, and what each one changes."
icon: database
keywords: ["connect", "database", "uri", "api key", "storage options", "consistency"]
availability:
oss: available
enterprise: available
summary: >-
The embedded library connects to a local directory or object storage, and
a `db://` URI connects to a deployment. Which operations exist depends on
the connection, as the table below shows.
---
A database is the root of a [catalog](/basics/catalog). A connection opens
@@ -40,10 +47,10 @@ storage and does the work itself. The third talks to a **deployment**.
| --- | --- | --- |
| Where queries run | in your process | on the server |
| Table handle | `LanceTable` | `RemoteTable` |
| [SQL](/basics/sql) | -- | yes |
| [SQL](/basics/sql) | -- | where enabled |
| Server-side [jobs](/basics/jobs) | -- | yes |
| [Functions](/management/function) | -- | yes |
| [Authorization](/governance/authorization) | storage permissions | granted per object |
| [Functions](/management/function) | -- | where enabled |
| [Authorization](/governance/authorization) | storage permissions | granted per object, where configured |
![](/static/assets/images/overview/understanding-tables.png)
@@ -132,9 +139,11 @@ const db = await lancedb.connect(uri, { readConsistencyInterval: 5 });
```
</CodeGroup>
The default reads a snapshot taken when the table was opened, so a long-lived
connection **will not see other processes' writes** until it reopens the table.
That is the right default for a batch job and the wrong one for a service.
In the embedded library, the default reads a snapshot taken when the table was
opened, so a long-lived connection **will not see other processes' writes**
until it reopens the table. That is the right default for a batch job and the
wrong one for a service. On a deployment, reads follow the deployment's own
freshness setting, which the interval can only tighten.
Set it when several processes write the same table and readers must observe
those writes. The cost is a metadata check on that interval. See
@@ -146,8 +155,8 @@ those writes. The cost is a metadata check on that interval. See
| --- | --- | --- | --- |
| list tables | `table_names`, `list_tables` | `tableNames`, `listTables` | -- |
| the URI it opened | `uri` | `display()` | -- |
| drop every table | `drop_all_tables` | `dropAllTables` | -- |
| drop the database | `drop_database` | -- | -- |
| drop every table, embedded only | `drop_all_tables` | `dropAllTables` | -- |
| drop the database, embedded only | `drop_database` | -- | -- |
| close | -- | `close`, `isOpen` | -- |
<CodeGroup>
docs/web/management/function.mdx
+12 -3
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Function"
description: "Register a Python function once and apply it as a column, for work SQL cannot express."
icon: code
keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"]
availability:
oss: unavailable
enterprise: varies
summary: >-
Functions run on a LanceDB Enterprise deployment that has them enabled; the
embedded library cannot register or run them.
---
import {
@@ -17,14 +23,13 @@ import {
PyFunctionInspect as FunctionInspect,
} from '/snippets/function.mdx';
<Badge color="red">Enterprise</Badge>
Some columns cannot be written as a SQL expression: an embedding, a model call,
an image decode. A function is a Python callable registered with the database,
versioned, and then applied as a column — the code runs where the data is
rather than pulling rows to your process.
Registering and running a function needs a LanceDB Enterprise connection. On a
Registering and running a function needs a LanceDB Enterprise deployment with
Functions enabled, which they are not by default. On a
local connection `create_function` raises `NotImplementedError`; to derive a
column locally, declare a
[SQL computed column](/feature-engineering/add-computed-column) instead.
@@ -212,6 +217,10 @@ the name was already gone. Deleting the object's content can finish after the
call returns, as a [background job](/maintenance/background-jobs).
`drop_function_async` returns that job with the result, so you can wait for it.
A deployment refuses the drop unless it also allows changes to a function's
lifecycle, which is a separate setting, off by default even where Functions are
enabled.
## What to read next {#next}
<CardGroup cols={2}>
docs/web/management/index.mdx
+9 -7
View File
@@ -16,7 +16,7 @@ Operations that can take a while have an `_async` form returning a
## The surface at a glance {#surface}
<Note>
SQL requires a deployment -- a local database has no engine to plan a statement.
SQL requires a deployment that enables its SQL endpoint -- a local database has none.
See [SQL](/basics/sql#sql-is-remote).
</Note>
@@ -71,11 +71,12 @@ it current.
| | Python | TypeScript | SQL |
| --- | --- | --- | --- |
| create | -- | -- | `CREATE VIEW` |
| drop | -- | -- | `DROP VIEW` |
| create | `create_view` | `createView` | `CREATE VIEW` |
| list | `list_views` | `listViews` | -- |
| drop | `drop_view`, `drop_view_async` | `dropView`, `dropViewAsync` | `DROP VIEW` |
<Badge color="red">Enterprise</Badge> Views exist only in SQL -- there is no SDK
method for them, because the thing a view stores *is* a SQL statement. See
<Badge color="red">Enterprise</Badge> Views exist only on a deployment that
enables them; the SDK methods send the view's SQL query to the deployment. See
[Views](/management/view).
### Functions {#functions}
@@ -87,8 +88,9 @@ method for them, because the thing a view stores *is* a SQL statement. See
| list | `list_functions` | -- | `SHOW FUNCTIONS` |
| drop | `drop_function` | -- | -- |
<Badge color="red">Enterprise</Badge> A function runs on the server, so there is
no local form. The TypeScript SDK has no function methods; use Python or SQL.
<Badge color="red">Enterprise</Badge> A function runs on the server, and only on
a deployment that enables functions, so there is no local form. The TypeScript
SDK has no function methods; use Python or SQL.
See [Functions](/management/function).
### Jobs {#jobs}
docs/web/management/materialized-view.mdx
+14 -5
View File
@@ -4,6 +4,14 @@ sidebarTitle: "Materialized View"
description: "Create, open, list and drop a table whose contents are defined by a query."
icon: table
keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"]
availability:
oss: available
enterprise: varies
summary: >-
Materialized views are created and managed in the embedded library. On a
deployment they are off by default and have to be enabled, and the SDK
creates them through the deployment's SQL endpoint; see
[Against a deployment](#deployment).
---
A materialized view is a table whose contents are defined by a query over
@@ -41,8 +49,8 @@ const view = await db.createMaterializedView("recent_docs", "documents", {
</CodeGroup>
`source` names the table it derives from. A local source does not need stable
row ids or any other storage option; a deployment may require stable row ids
(see [Against a deployment](#deployment)). `select`, `where` and `limit` record
row ids or any other storage option; an earlier deployment version may require
stable row ids (see [Against a deployment](#deployment)). `select`, `where` and `limit` record
the query -- there is no separate definition step.
## Opening, listing and dropping {#manage}
@@ -78,9 +86,10 @@ rows. Both are properties; on an async view, the definition is read with
<Badge color="red">Enterprise</Badge> `create_materialized_view` works on a
`db://` connection as well as a local one. There the SDK creates and refreshes
the view through SQL, and the work runs on the cluster rather than in your
process. Materialized views are off by default on a deployment and have to be
enabled there, and a deployment may require a view's source to have stable row
ids; see
process, so the deployment's [SQL endpoint](/basics/sql#sql-is-remote) has to be
reachable. Materialized views are off by default on a deployment and have to be
enabled there, and an earlier deployment version may require a view's source to
have stable row ids; see
[Against a deployment](/feature-engineering/materialized-views#availability).
## What to read next {#next}
docs/web/management/namespace.mdx
+9 -2
View File
@@ -4,6 +4,12 @@ sidebarTitle: "Namespace"
description: "Grouping tables into a hierarchy, and addressing them through it."
icon: folder-tree
keywords: ["namespace", "catalog", "hierarchy", "directory namespace", "rest namespace"]
availability:
oss: available
enterprise: available
summary: >-
Namespaces work in the embedded library, through directory or REST
catalogs, and on a deployment.
---
import {
@@ -29,7 +35,7 @@ language: Python uses `namespace_path`, Rust uses builder methods like `.namespa
TypeScript uses `namespacePath` arguments.
<Note>
In SQL a namespaced table is addressed as a [dotted path](/sql/dql/select#namespaces).
In SQL on a deployment, a namespaced table is addressed as a [dotted path](/sql/dql/select#namespaces).
</Note>
## Namespace hierarchy {#namespace-hierarchy}
@@ -227,7 +233,8 @@ let db = lancedb::connect_namespace("rest", properties)
```
</CodeGroup>
[LanceDB Enterprise](/enterprise) operates a REST namespace server on top of the Lance format, so any REST client that can speak the REST namespace API
contract can be used to interact with it. For authentication examples in LanceDB Enterprise, visit
contract can be used to interact with it. It does not serve the contract's `exists` routes, so checks such as `namespace_exists`
fail against it. For authentication examples in LanceDB Enterprise, visit
the [Namespaces in SDKs](#namespaces-in-lancedb-enterprise) page.
## Best practices {#best-practices}
docs/web/management/view.mdx
+11 -9
View File
@@ -4,14 +4,14 @@ sidebarTitle: "View"
description: "A saved query, evaluated when read. The non-materialized counterpart to a materialized view."
icon: eye
keywords: ["view", "create view", "sql", "catalog"]
availability:
oss: unavailable
enterprise: varies
summary: >-
Views need a LanceDB Enterprise deployment that has them enabled; the
embedded library cannot create them.
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it -- see
[Architecture](/architecture) for what differs between the two.
A view stores a **query**, not rows. Reading it runs the query against current
data, so a view is never stale -- and never faster than the query it wraps.
@@ -27,6 +27,8 @@ defined by a query; the difference is entirely when that query runs.
| Read cost | the full query, every read | a table scan |
| Write cost | none | a refresh |
Views are created in SQL, over the [Flight SQL endpoint](/sql#connecting) --
there is no SDK method, because the thing a view stores *is* a SQL statement.
See [View](/sql/ddl/view) for `CREATE VIEW`, `DROP VIEW`, and listing.
Views are created on a deployment, either in SQL over the
[Flight SQL endpoint](/sql#connecting) or with the SDKs' `create_view` /
`createView`, which send the query to the deployment. Either way, the deployment
must have views enabled. See [View](/sql/ddl/view) for `CREATE VIEW`,
`DROP VIEW`, and listing.