mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Brendan Clement
d0ed4d1a7a
- CREATE INDEX: the example left out USING, which the server requires, and the page said SQL cannot choose the algorithm or name the index. USING names the algorithm, the page lists them, and a name before ON is taken where the deployment's configuration allows it. - The search join passed fts() a bare string; fts() takes JSON. - Sampling a subquery's result is refused, so the sampling and subquery pages sample the table inside the subquery, and the refusals list it. - Two unaliased casts in one projection produce the same column name; the example aliases them. - A json_get nested in another json_get is refused. The page shows one level of nesting and json_extract with a JSONPath for deeper values, and corrects what json_extract and json_extract_with_type return.
99 lines
2.9 KiB
Plaintext
99 lines
2.9 KiB
Plaintext
---
|
|
title: "Joining datasets"
|
|
sidebarTitle: "Join"
|
|
description: "Bringing labels, metadata and scores together with the rows they describe."
|
|
icon: link
|
|
keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
|
|
availability:
|
|
oss: unavailable
|
|
enterprise: varies
|
|
summary: >-
|
|
Joins are SQL, which runs on an Enterprise deployment with SQL enabled;
|
|
the embedded library does not run SQL.
|
|
---
|
|
|
|
Curation usually means combining things that were produced separately — images
|
|
in one table, human labels in another, model scores in a third. A join is how
|
|
they become one dataset.
|
|
|
|
## Attaching labels {#labels}
|
|
|
|
```sql
|
|
SELECT i.id, i.embedding, l.label
|
|
FROM images AS i
|
|
JOIN labels AS l ON l.image_id = i.id
|
|
```
|
|
|
|
An inner join keeps only rows that matched, which for curation is often exactly
|
|
what you want: unlabelled rows are not training data.
|
|
|
|
When you need to *see* what did not match, use a left join and filter:
|
|
|
|
```sql
|
|
SELECT i.id
|
|
FROM images AS i
|
|
LEFT JOIN labels AS l ON l.image_id = i.id
|
|
WHERE l.image_id IS NULL
|
|
```
|
|
|
|
That is your unlabelled backlog, which is the thing worth counting before you
|
|
plan an annotation round.
|
|
|
|
## Across namespaces {#namespaces}
|
|
|
|
Tables in different [namespaces](/management/namespace) join by their dotted
|
|
paths — the catalog is one catalog, so there is nothing to federate:
|
|
|
|
```sql
|
|
SELECT i.id, u.team
|
|
FROM prod.media.images AS i
|
|
JOIN prod.reference.uploaders AS u ON u.id = i.uploader_id
|
|
```
|
|
|
|
## Joining a search result {#search}
|
|
|
|
Full-text search is a table function, so its result is a relation and joins like
|
|
any other. This is how you ask which categories a query matches, rather than
|
|
just which rows:
|
|
|
|
```sql
|
|
SELECT c.name, count(*) AS hits
|
|
FROM fts('documents', '{"match":{"column":"text","terms":"puppy"}}') AS m
|
|
JOIN categories AS c ON c.id = m.category_id
|
|
GROUP BY c.name
|
|
ORDER BY hits DESC
|
|
```
|
|
|
|
`fts` takes the query as JSON, as in [Full-text search](/sql/dql/fts); a bare
|
|
string such as `'puppy'` is refused.
|
|
|
|
<Note>
|
|
Whether SQL can express vector search
|
|
[depends on the deployment](/basics/sql#no-vector-sql). Where it cannot, run the
|
|
search in an SDK, then join over its output — or write the result to a table
|
|
first.
|
|
</Note>
|
|
|
|
## Making it fast {#performance}
|
|
|
|
Two things dominate join cost on a lakehouse table.
|
|
|
|
**Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets
|
|
the planner look rows up instead of scanning for them. Check with
|
|
[`EXPLAIN`](/sql/dql/explain): an indexed key shows a `ScalarIndexQuery`, an
|
|
unindexed one shows a `LanceScan` feeding the join.
|
|
|
|
**Do not carry wide columns through.** Select the columns you need before
|
|
joining, not after. Embeddings and blobs are the expensive part of a multimodal
|
|
table, and a join can materialize them many more times than the result has rows.
|
|
|
|
```sql
|
|
SELECT i.id, l.label
|
|
FROM (SELECT id, uploader_id FROM images) AS i
|
|
JOIN labels AS l ON l.image_id = i.id
|
|
```
|
|
|
|
## Reference {#reference}
|
|
|
|
The join types and their semantics are in [Joins](/sql/dql/join).
|