lancedb/docs/web/curation/windowing.mdx

---
title: "Ranking and deduplication"
sidebarTitle: "Rank and Deduplicate"
description: "Choose records with explicit ranking rules and deterministic tie handling."
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
availability:
  embedded: unavailable
  enterprise: available
---

Window functions calculate values across related rows while keeping each row
in the result. Use them to rank documents, keep one record per item, or compare
a record with the one before it.

## Example data {#example-data}

Connect with an [Enterprise SQL client](/basics/sql). The examples use a small
document table and a separate label table. Expand the setup below to create them.

<Accordion title="Create the example tables">

`curation_documents` has six records describing four documents.
`curation_labels` has one label for each of three documents. `row_id` identifies
a record; `doc_id` identifies the document it describes.

Run each statement separately in a scratch database or namespace. Create these
tables once for the whole section, using fresh names if they already exist.

<CodeGroup>
```sql SQL icon="database"
CREATE TABLE curation_documents (
    row_id BIGINT, doc_id BIGINT, version BIGINT,
    source VARCHAR, caption VARCHAR, score DOUBLE
);
INSERT INTO curation_documents VALUES
    (1, 10, 1, 'web', NULL, 0.80),
    (2, 10, 2, 'web', 'A cat on a chair', 0.95),
    (3, 20, 1, 'books', 'A dog in a field', 0.90),
    (4, 30, 1, 'web', NULL, 0.70),
    (5, 30, 1, 'web', 'A cat by a window', 0.85),
    (6, 40, 1, 'books', 'A bird in a tree', 0.60);

CREATE TABLE curation_labels (doc_id BIGINT, label VARCHAR);
INSERT INTO curation_labels VALUES
    (10, 'cat'), (20, 'dog'), (30, 'cat');
```
</CodeGroup>

`score` is a sample quality score, not a search result score. Document 10 has
two versions. Document 30 has two records at the same version. Document 40
has no label. These cases let you check the effect of each operation.

</Accordion>

## Keep one record per document {#dedup}

`row_number()` numbers rows within each group. `PARTITION BY` defines the group,
and `ORDER BY` determines which record comes first. This query selects the
latest version of each document, using `row_id` to break ties:

<CodeGroup>
```sql SQL icon="database"
WITH ranked AS (
    SELECT *, row_number() OVER (
        PARTITION BY doc_id ORDER BY version DESC, row_id DESC
    ) AS rn
    FROM curation_documents
)
SELECT row_id, doc_id, version, caption
FROM ranked
WHERE rn = 1
ORDER BY doc_id;
```
</CodeGroup>

This returns row IDs **2, 3, 5 and 6**. For document 30, rows 4 and 5 have the
same version; the example deliberately chooses the larger `row_id`.
Without a complete ordering, the selected row among ties is not determined by
the query.

Use `rank()` instead of `row_number()` when tied records should all survive,
and order only by the fields that define a tie. Adding a unique tie-breaker
would make every rank distinct. The outer query filters the calculated rank
because window functions are evaluated after the inner query's `WHERE` clause.

## Select the highest-scoring records per class {#per-group}

Deduplicate first, then rank the labeled documents within each class:

<CodeGroup>
```sql SQL icon="database"
WITH latest AS (
    SELECT *, row_number() OVER (
        PARTITION BY doc_id ORDER BY version DESC, row_id DESC
    ) AS version_rank
    FROM curation_documents
), ranked AS (
    SELECT d.doc_id, l.label, d.score,
           row_number() OVER (
               PARTITION BY l.label ORDER BY d.score DESC, d.doc_id
           ) AS class_rank
    FROM latest AS d
    JOIN curation_labels AS l ON l.doc_id = d.doc_id
    WHERE d.version_rank = 1
)
SELECT doc_id, label, score
FROM ranked
WHERE class_rank <= 1
ORDER BY label;
```
</CodeGroup>

The result is **document 10 for cat and document 20 for dog**. Change the cap
from `1` to the maximum number you want per class. A class with fewer records
cannot supply the full cap.

This selects by score, not at random. For a random subset, use
[sampling](/curation/statistical-sampling) instead of ordering by score or ID.

## Training and evaluation splits {#splits}

Ranking can divide ordered records into groups, but ordering by ID does not
make those groups random or representative. `ntile()` produces approximately
equal-sized buckets within a partition; small classes may not populate every
bucket.

Choose the split policy for the task. Keep related records, such as revisions
of one document or frames from one video, in the same split to avoid leakage.
A time-based evaluation may instead require a cutoff date. Save the final
assignments or resulting datasets so later updates do not silently change the
evaluation population.

## Compare neighboring records {#offset}

`lag()` and `lead()` read neighboring rows in the specified order. For example,
inspect how scores change across a document's records:

<CodeGroup>
```sql SQL icon="database"
SELECT doc_id, row_id, score,
       lag(score) OVER (
           PARTITION BY doc_id ORDER BY version, row_id
       ) AS previous_score
FROM curation_documents
ORDER BY doc_id, version, row_id;
```
</CodeGroup>

The first record in each document has no previous score. In this dataset,
row 2 follows a score of 0.80 and row 5 follows a score of 0.70.

<span id="running"></span>
## Window query performance {#cost}

Window operations can require sorting. Filter out records you do not need
before ranking, and inspect the
[query plan](/basics/sql#query-plans) when cost matters.

For cumulative counts, sums and other running summaries, use an aggregate
with an explicit window order and frame. The
[window reference](/sql/dql/window#window) covers those expressions.

## Continue with the curated data {#reference}

[Profile the selected records](/curation/aggregate) and
[save the result](/curation#next) for downstream use.
See [Window functions](/sql/dql/window) for the full syntax and function list.