mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
This commit is contained in:
Jack Ye
co-authored by
Brendan Clement
Brendan Clement
parent
de005f1d61
commit
47b7babc08
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Copy"
|
||||
sidebarTitle: "Copy"
|
||||
description: "COPY ... TO — writing a query result to object storage as files."
|
||||
icon: file-export
|
||||
keywords: ["copy", "export", "parquet", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
|
||||
```
|
||||
|
||||
Writes a query result to object storage. This is the export path when the
|
||||
consumer is another engine rather than an application -- results land as files
|
||||
rather than streaming to a client.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Insert" icon="plus" href="/sql/dml/insert">
|
||||
Bringing rows in, rather than exporting them.
|
||||
</Card>
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "Delete"
|
||||
sidebarTitle: "Delete"
|
||||
description: "DELETE FROM ... WHERE — rows are marked, not erased."
|
||||
icon: trash
|
||||
keywords: ["delete", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
DELETE FROM documents WHERE created_at < '2025-01-01'
|
||||
```
|
||||
|
||||
Deletion marks rows rather than erasing them, so the data remains recoverable
|
||||
for as long as the version holding it survives
|
||||
[retention](/maintenance/cleanup).
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Delete" icon="trash" href="/ingestion/delete">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: "Insert"
|
||||
sidebarTitle: "Insert"
|
||||
description: "INSERT INTO, including INSERT ... SELECT to move rows server-side."
|
||||
icon: plus
|
||||
keywords: ["insert", "insert into", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
|
||||
|
||||
INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
|
||||
```
|
||||
|
||||
`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
|
||||
server-side, without a round trip through your process.
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Insert" icon="plus" href="/ingestion/insert">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
<Card title="Merge Insert" icon="code-merge" href="/ingestion/merge-insert">
|
||||
Upsert -- matching incoming rows against existing keys -- has no SQL form.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "REFRESH"
|
||||
sidebarTitle: "REFRESH"
|
||||
description: "Filling computed columns and bringing materialized views up to date."
|
||||
icon: arrows-rotate
|
||||
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
LanceDB extensions with no DataFusion equivalent. They are what makes
|
||||
[feature engineering](/feature-engineering) reachable from SQL: a column
|
||||
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
||||
|
||||
## REFRESH COLUMN {#refresh-column}
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN <column> ON [TABLE] <table>
|
||||
[WHERE <predicate>]
|
||||
[WITH (<option> = <value> [, ...])]
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN doubled ON my_table
|
||||
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
||||
```
|
||||
|
||||
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
||||
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
||||
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
|
||||
can recompute values the failed run had already written.
|
||||
|
||||
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
||||
committing to the whole table.
|
||||
|
||||
### Options {#refresh-column-options}
|
||||
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `num_workers` | how many workers to use |
|
||||
| `cpu_millis` | CPU budget per worker |
|
||||
| `memory_bytes` | memory budget per worker |
|
||||
| `gpu_count` | GPUs per worker |
|
||||
| `batch_size` | rows a worker takes at a time |
|
||||
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN embedding ON documents
|
||||
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
||||
```
|
||||
|
||||
`num_workers` sets a count; the budget options size each worker. They are
|
||||
alternatives rather than a pair.
|
||||
|
||||
<Note>
|
||||
`error_policy` and `num_workers` are refused for a column filled by a registered
|
||||
[function](/management/function) with its own resource budget -- use the CPU,
|
||||
memory and GPU options there.
|
||||
</Note>
|
||||
|
||||
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW <name> [CASCADE] [WITH (<option> = <value>)]
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
||||
```
|
||||
|
||||
`full = true` asks for every row to be recomputed. Without it, whether a
|
||||
refresh rebuilds the whole view -- as every local SDK refresh does -- or
|
||||
reconciles only what changed in the source depends on the deployment's LanceDB
|
||||
Enterprise version and configuration.
|
||||
|
||||
<Warning>
|
||||
`CASCADE` asks for the views this one is built on to be refreshed first, in
|
||||
dependency order. Whether a deployment carries it out depends on its version;
|
||||
where it does not, refresh each view yourself, its sources first.
|
||||
</Warning>
|
||||
|
||||
## Watching a refresh {#watching}
|
||||
|
||||
Both statements submit work that outlives the call. Track it with
|
||||
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
||||
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
||||
|
||||
## See also {#see-also}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
The same operation from the SDKs, and what a refresh guarantees.
|
||||
</Card>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
||||
Declaring the columns this fills.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: "Update"
|
||||
sidebarTitle: "Update"
|
||||
description: "UPDATE ... SET ... WHERE — and why it does not recompute derived columns."
|
||||
icon: pen
|
||||
keywords: ["update", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
UPDATE documents SET title = upper(title) WHERE category = 'film'
|
||||
```
|
||||
|
||||
Assignments take any expression the [dialect](/sql/functions) supports,
|
||||
including references to other columns of the same row. Without a `WHERE`
|
||||
clause every row is updated.
|
||||
|
||||
<Warning>
|
||||
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
|
||||
that derive from the column you changed. Those rows keep the value computed from
|
||||
the old input until the next refresh recomputes them -- see
|
||||
[Guarantees](/basics/guarantees#jobs).
|
||||
</Warning>
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Update" icon="pen" href="/ingestion/update">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
Reference in New Issue
Block a user