mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-27 14:06:14 +00:00
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
This commit is contained in:
@@ -4,6 +4,12 @@ sidebarTitle: "Data Evolution"
|
||||
description: "Derive new columns from data already in the table, without rewriting what is already there."
|
||||
icon: wand-magic-sparkles
|
||||
keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
|
||||
availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Adding columns computed from SQL expressions works in the embedded library
|
||||
and in LanceDB Enterprise.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -18,20 +24,23 @@ import {
|
||||
RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
|
||||
} from '/snippets/tables.mdx';
|
||||
|
||||
Data evolution is adding **values** derived from what a table already holds.
|
||||
[Schema evolution](/management/table#schema) changes the table's shape; this fills the
|
||||
shape with something new.
|
||||
Add derived columns to calculate features from data already in a table.
|
||||
For example, compute a product score from existing fields without reading
|
||||
and rewriting the table in your application.
|
||||
|
||||
The reason it is cheap is the format. Adding a derived column writes only that
|
||||
column -- the existing data, which is the expensive part on a multimodal table,
|
||||
is never rewritten. See [why that works](/basics/lance#evolution).
|
||||
Lance can write the new columns without rewriting the existing columns.
|
||||
This is useful when the table also contains large images, video or embeddings.
|
||||
See [Lance format](/basics/lance) for the storage model.
|
||||
|
||||
<Note>
|
||||
In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
|
||||
In SQL, on an Enterprise cluster that enables computed columns, this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
|
||||
</Note>
|
||||
|
||||
## Set up the example table {#set-up-the-example-table}
|
||||
|
||||
The examples use `db`, a connection opened as in
|
||||
[Connect to LanceDB](/basics/connect), and create a small product table in it:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{PySchemaAddSetup}
|
||||
@@ -96,16 +105,24 @@ If your transformation cannot be expressed in SQL, compute the values outside
|
||||
|
||||
## When SQL is not enough {#beyond-sql}
|
||||
|
||||
The expressions above are evaluated once, as part of the call. Two other shapes
|
||||
exist when that does not fit:
|
||||
The expressions above are evaluated once, as part of the call. For other workflows:
|
||||
|
||||
- A [computed column](/feature-engineering/add-computed-column) declares the
|
||||
expression and leaves the column empty, so you choose when the work happens.
|
||||
That is the right default on a large table.
|
||||
Use it when you want to schedule the computation separately.
|
||||
- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for
|
||||
values SQL cannot express -- an embedding, a model call, an image decode.
|
||||
|
||||
{/*
|
||||
Draft for the Functions release. Replaces the Geneva item in the list above:
|
||||
- A [function](/management/function) fills a column with Python, for values SQL
|
||||
cannot express -- an embedding, a model call, an image decode.
|
||||
*/}
|
||||
|
||||
## What to read next {#next}
|
||||
A deployment has to enable computed columns before you use them.
|
||||
{/* Draft for the Functions release: Functions need enabling there too. */}
|
||||
|
||||
## Related guides {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
|
||||
|
||||
Reference in New Issue
Block a user