mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
fd177e0aea
The tab now runs write, read, curate, derive, train before it reaches the management side: Ingestion, Semantic Search, Data Curation, Feature Engineering, Training, then Management and Operations. Indexing joins Operations, where the index maintenance page already was, and gains it as Build & Repair. Ingestion gains a Create Table page. It was a section inside the overview, so the one thing a new reader needs first was the one thing the sidebar did not show. Data Curation is new: Aggregate, Sampling, Join, Windowing. The SQL reference answers what the syntax is; these answer why you would reach for it, framed as the questions curation actually asks -- what do I have, what can I afford, what else do I know, which one do I keep. Each links to the reference for the complete grammar rather than repeating it. They carry the things a reference page would not: that count(*) minus count(col) is your coverage, that sampling both sides of a join gives a percent rather than ten, that a window ordered by a value you recompute moves rows between train and test, and that a one percent sample tells you about the head and nothing about the tail.
125 lines
3.8 KiB
Plaintext
125 lines
3.8 KiB
Plaintext
---
|
|
title: "Ranking and deduplication"
|
|
sidebarTitle: "Windowing"
|
|
description: "Picking one row per thing, ranking within a group, and splitting a dataset."
|
|
icon: chart-simple
|
|
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> Window functions are SQL, and
|
|
[SQL needs a deployment](/basics/apis).
|
|
|
|
A window function computes across a set of related rows and returns a value
|
|
**per row** — nothing is collapsed. That is what makes it the tool for "which of
|
|
these do I keep", where an aggregate would throw away the rows you are choosing
|
|
between.
|
|
|
|
## One row per thing {#dedup}
|
|
|
|
The most common curation query there is. You have several rows per document —
|
|
revisions, re-uploads, repeated annotations — and want one.
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT *,
|
|
row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
|
|
FROM revisions
|
|
)
|
|
SELECT * FROM ranked WHERE rn = 1
|
|
```
|
|
|
|
`PARTITION BY` says what "per thing" means; `ORDER BY` says which one wins. The
|
|
outer query is not optional — a window function cannot appear in `WHERE`,
|
|
because windows are computed after filtering, so the predicate would have
|
|
nothing to read.
|
|
|
|
<Note>
|
|
Swap `row_number()` for `rank()` when ties should both survive. `row_number()`
|
|
always picks exactly one; `rank()` gives tied rows the same rank, so `rank = 1`
|
|
can return several.
|
|
</Note>
|
|
|
|
## A fixed number per class {#per-group}
|
|
|
|
Class imbalance found by [profiling](/curation/aggregate#balance) is fixed here:
|
|
rank within each class, then take the top N.
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT *, row_number() OVER (PARTITION BY label ORDER BY score DESC) AS rn
|
|
FROM images
|
|
)
|
|
SELECT * FROM ranked WHERE rn <= 1000
|
|
```
|
|
|
|
That gives at most 1,000 rows per label, taking the highest-scoring ones. Order
|
|
by something arbitrary but stable if you want an unbiased draw instead, or
|
|
combine with [sampling](/curation/sampling) to randomize within each class.
|
|
|
|
## Splitting a dataset {#splits}
|
|
|
|
`ntile` divides each partition into equal buckets, which is a train/test split
|
|
that stays balanced per class:
|
|
|
|
```sql
|
|
WITH split AS (
|
|
SELECT *, ntile(10) OVER (PARTITION BY label ORDER BY id) AS bucket
|
|
FROM images
|
|
)
|
|
SELECT * FROM split WHERE bucket <= 8 -- train
|
|
```
|
|
|
|
Buckets 9 and 10 are your validation and test sets. Because the partition is by
|
|
label, each split keeps the class proportions of the whole.
|
|
|
|
<Warning>
|
|
Order by something stable. Ordering by a value that changes — a score you
|
|
recompute, a timestamp that gets updated — means the split changes with it, and
|
|
a row can move from train to test between runs.
|
|
</Warning>
|
|
|
|
## Comparing a row to its neighbours {#offset}
|
|
|
|
`lag` and `lead` read the previous and next row in the window, which is how you
|
|
find gaps and changes without a self-join:
|
|
|
|
```sql
|
|
SELECT
|
|
id,
|
|
created_at,
|
|
created_at - lag(created_at) OVER (ORDER BY created_at) AS gap
|
|
FROM events
|
|
```
|
|
|
|
Useful for spotting ingestion gaps, duplicate timestamps, and runs of identical
|
|
values that suggest a broken upstream job.
|
|
|
|
## Running totals {#running}
|
|
|
|
Any [aggregate](/curation/aggregate) becomes a window function with `OVER`,
|
|
which is how cumulative coverage is expressed:
|
|
|
|
```sql
|
|
SELECT
|
|
label,
|
|
count(*) AS n,
|
|
sum(count(*)) OVER (ORDER BY count(*) DESC) AS cumulative
|
|
FROM images
|
|
GROUP BY label
|
|
```
|
|
|
|
Reading down the `cumulative` column tells you how many classes cover most of
|
|
your data — usually far fewer than you expect.
|
|
|
|
## Cost {#cost}
|
|
|
|
Window functions sort within each partition, so on a large table they are among
|
|
the more expensive things you can ask for. Filter first, partition on something
|
|
selective, and check the plan with
|
|
[Explain Analyze](/search/explain).
|
|
|
|
## Reference {#reference}
|
|
|
|
The full window syntax, framing, and function list are in
|
|
[Window functions](/sql/window).
|