lancedb/docs/web/curation/chunking.mdx

---
title: "Chunk records into multiple rows"
sidebarTitle: "Chunking"
description: "Expand one source row into many derived rows -- documents into passages, videos into clips, images into tiles."
icon: diagram-subtask
keywords: ["chunking", "function", "udf", "lateral", "unnest", "1:N", "passages", "clips", "video", "frames"]
availability:
  embedded: unavailable
  enterprise: available
---

Every other curation task on this page keeps the row count the same or
smaller: profiling, sampling, joining, deduplicating. Chunking goes the other
way -- one source row becomes many derived rows, each carrying its parent's
columns along with it. Use it to split a document into passages for
embedding, a video into clips, or an image into tiles.

## Define a chunking Function {#function}

A Function that returns a `list` produces more than one value per input row.
A chunker is a Function whose output is a list -- of structs, of strings, of
anything -- registered like any other Function. See
[Create a Function](/curation/create-function) for CPU, GPU and remote-service
Functions, and scalar versus batch calling, if you have not registered one
before.

A function that splits text into fixed-size passages turns a `documents`
table into one row per passage. Registration details, environment and
dependency declarations, and how a version stays immutable once rows are
filled from it, are the same for a chunker as for any other Function --
register it from Python, or the same function from SQL with
[`CREATE FUNCTION`](/sql/ddl/function#create):

**Source: `documents`**

| doc_id | title | text |
| --- | --- | --- |
| 1 | "Intro to AI" | "Machine learning is a method of..." (long text) |

<CodeGroup>
```python Python icon="python"
import pyarrow as pa
from lancedb.functions import udf

@udf(
    output_schema=pa.list_(
        pa.struct([
            pa.field("chunk_index", pa.int64(), nullable=False),
            pa.field("chunk_text", pa.utf8(), nullable=False),
        ])
    ),
)
def split_into_passages(text: str) -> list[dict]:
    size = 200
    return [
        {"chunk_index": i, "chunk_text": text[i : i + size]}
        for i in range(0, len(text), size)
    ]

version = db.create_function_async(split_into_passages).wait()
```

```sql SQL icon="database"
CREATE FUNCTION split_into_passages(text utf8)
RETURNS list<struct<chunk_index: int64, chunk_text: utf8>>
LANGUAGE python
AS '
def split_into_passages(text):
    size = 200
    return [
        {"chunk_index": i, "chunk_text": text[i:i+size]}
        for i in range(0, len(text), size)
    ]
'
```
</CodeGroup>

## Explode it in a query {#explode}

Call the function in the `FROM` clause, after the source table, with an
alias: a Function there yields one row per element of the list it returns,
with every other column repeated. Reference the function's output fields
through the alias:

<CodeGroup>
```sql SQL icon="database"
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
```
</CodeGroup>

A document with five 200-character passages becomes five output rows, each
carrying `doc_id` and `title` from its parent. This is the fastest way to
check a chunker before committing to a materialized view: run it over a few
rows and inspect the output shape.

## Video into frames {#video}

Chunking a blob column is the same shape, with the chunker producing
timestamps rather than text spans, and a second Function pulling the actual
frame for each one. `videos` holds raw clips in a
[Blob v2](/management/table#managed-blobs) column, with each row's duration
already known:

**Source: `videos`**

| video_id | video | duration_seconds |
| --- | --- | --- |
| "clip-1" | (Blob v2 bytes) | 12.0 |

A chunker that samples one timestamp per second needs no blob access at all
-- it only looks at `duration_seconds`:

<CodeGroup>
```python Python icon="python"
@udf(
    output_schema=pa.list_(
        pa.struct([
            pa.field("frame_index", pa.int64(), nullable=False),
            pa.field("ts", pa.float64(), nullable=False),
        ])
    ),
)
def sample_frame_timestamps(duration_seconds: float) -> list[dict]:
    every = 1.0  # one frame per second
    count = int(duration_seconds // every)
    return [{"frame_index": i, "ts": i * every} for i in range(count)]

version = db.create_function_async(sample_frame_timestamps).wait()
```
</CodeGroup>

A second Function reads the blob -- incrementally, through a `BlobReader`,
rather than loading the whole clip -- and returns one JPEG per call:

<CodeGroup>
```python Python icon="python"
from datetime import timedelta
from lancedb import BlobReader

@udf(
    name="extract_frame",
    gpu=True,
    pip=["av==13.1.0", "pillow==11.0.0"],
    input_schema=pa.schema([
        lancedb.blob("video"),
        pa.field("ts", pa.float64(), nullable=False),
    ]),
    output_schema=lancedb.blob("jpeg", nullable=False),
)
def extract_frame(video: BlobReader, ts: float) -> bytes:
    ...  # decode the frame at ts, encode as JPEG, return its bytes

extract_version = db.create_function_async(extract_frame).wait(timeout=timedelta(minutes=15))
```
</CodeGroup>

Compose both in one query: explode `videos` into one row per sampled
timestamp, then call `extract_frame` on each row to produce its image:

<CodeGroup>
```sql SQL icon="database"
SELECT video_id, f.frame_index, f.ts, extract_frame(video, f.ts) AS jpeg
FROM videos, sample_frame_timestamps(duration_seconds) AS f
```
</CodeGroup>

A 12-second clip becomes 12 output rows, each with its own extracted JPEG. A
Function in `FROM` position yields one row per list element the same way
regardless of what the list holds; whether the second step produces a blob,
a plain value, or another struct does not change the explosion.

## Persist it as a materialized view {#persist}

Wrap either query in [`CREATE MATERIALIZED VIEW`](/sql/ddl/materialized-view)
to keep a real table that stays current as the source changes, instead of
recomputing the explosion on every query:

<CodeGroup>
```sql SQL icon="database"
CREATE MATERIALIZED VIEW chunks AS
SELECT doc_id, title, p.chunk_index, p.chunk_text
FROM documents, split_into_passages(text) AS p
```
</CodeGroup>

**Derived: `chunks`** (1:N expansion)

| doc_id | title | chunk_index | chunk_text |
| --- | --- | --- | --- |
| 1 | "Intro to AI" | 0 | "Machine learning is a method..." |
| 1 | "Intro to AI" | 1 | "...of teaching computers..." |

The view refreshes like any other materialized view: new source rows are
chunked and inserted, deleted source rows drop their children, and an
updated row's old children are replaced by freshly chunked ones on the next
[refresh](/feature-engineering/refresh-column). A single Function call or
`UNNEST(column)` is the only lateral source a materialized view can carry --
not both, and not combined with `GROUP BY` -- so keep aggregation or further
curation in a second step over the chunked table.

## Chunk, then curate {#next}

A freshly chunked table is a normal table: [dedupe](/curation/dedupe) it to
collapse near-identical chunks, [sample](/curation/statistical-sampling) it for a quick
look, or [profile it](/curation/aggregate) to check how many chunks each
source row produced. Continue with
[choosing a curation task](/curation#questions).