mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 14:52:26 +00:00
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
92 lines
2.7 KiB
Plaintext
92 lines
2.7 KiB
Plaintext
---
|
|
title: "Joining datasets"
|
|
sidebarTitle: "Join"
|
|
description: "Bringing labels, metadata and scores together with the rows they describe."
|
|
icon: link
|
|
keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> Joins are SQL, and
|
|
[SQL needs a deployment](/basics/sql#sql-is-remote).
|
|
|
|
Curation usually means combining things that were produced separately — images
|
|
in one table, human labels in another, model scores in a third. A join is how
|
|
they become one dataset.
|
|
|
|
## Attaching labels {#labels}
|
|
|
|
```sql
|
|
SELECT i.id, i.embedding, l.label
|
|
FROM images AS i
|
|
JOIN labels AS l ON l.image_id = i.id
|
|
```
|
|
|
|
An inner join keeps only rows that matched, which for curation is often exactly
|
|
what you want: unlabelled rows are not training data.
|
|
|
|
When you need to *see* what did not match, use a left join and filter:
|
|
|
|
```sql
|
|
SELECT i.id
|
|
FROM images AS i
|
|
LEFT JOIN labels AS l ON l.image_id = i.id
|
|
WHERE l.image_id IS NULL
|
|
```
|
|
|
|
That is your unlabelled backlog, which is the thing worth counting before you
|
|
plan an annotation round.
|
|
|
|
## Across namespaces {#namespaces}
|
|
|
|
Tables in different [namespaces](/management/namespace) join by their dotted
|
|
paths — the catalog is one catalog, so there is nothing to federate:
|
|
|
|
```sql
|
|
SELECT i.id, u.team
|
|
FROM prod.media.images AS i
|
|
JOIN prod.reference.uploaders AS u ON u.id = i.uploader_id
|
|
```
|
|
|
|
## Joining a search result {#search}
|
|
|
|
Full-text search is a table function, so its result is a relation and joins like
|
|
any other. This is how you ask which categories a query matches, rather than
|
|
just which rows:
|
|
|
|
```sql
|
|
SELECT c.name, count(*) AS hits
|
|
FROM fts('documents', 'puppy') AS m
|
|
JOIN categories AS c ON c.id = m.category_id
|
|
GROUP BY c.name
|
|
ORDER BY hits DESC
|
|
```
|
|
|
|
<Note>
|
|
Vector search has no SQL form, so a similarity result cannot be joined this way.
|
|
Run the search in an SDK, then join over its output — or write the result to a
|
|
table first.
|
|
</Note>
|
|
|
|
## Making it fast {#performance}
|
|
|
|
Two things dominate join cost on a lakehouse table.
|
|
|
|
**Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets
|
|
the planner look rows up instead of scanning for them. Check with
|
|
[`EXPLAIN`](/sql/dql/explain): an indexed key shows a `ScalarIndexQuery`, an
|
|
unindexed one shows a `LanceScan` feeding the join.
|
|
|
|
**Do not carry wide columns through.** Select the columns you need before
|
|
joining, not after. Embeddings and blobs are the expensive part of a multimodal
|
|
table, and a join can materialize them many more times than the result has rows.
|
|
|
|
```sql
|
|
SELECT i.id, l.label
|
|
FROM (SELECT id, uploader_id FROM images) AS i
|
|
JOIN labels AS l ON l.image_id = i.id
|
|
```
|
|
|
|
## Reference {#reference}
|
|
|
|
The join types and their semantics are in [Joins](/sql/dql/join).
|