lancedb/docs/web/training/permutation.mdx

---
title: "Permutation"
sidebarTitle: "Permutation"
description: "The low-level, customizable row ordering that StreamingDataset is built on."
icon: shuffle
availability:
  embedded: available
  enterprise: available
---

Most model training frameworks feed data into a model by sequencing through successive batches, a process
often referred to as **data loading**. The simplest way to perform data loading is to read a LanceDB table in batches,
in the order its rows are stored, through an identity `Permutation`. A table itself is not iterable.

<CodeGroup>
```py Python icon=Python
from lancedb.permutation import Permutation

db = lancedb.connect("file://some/db/path")
table = db.open_table("some_table")

for batch in Permutation.identity(table).with_format("arrow").iter(batch_size=1024):
    print(batch.to_pydict())
```
</CodeGroup>

In specific use cases, you may need the flexibility to shuffle, split, and select data without using the full
[`StreamingDataset`](/training/data-loading) object. In these cases, use `Permutation`, the lower-level
functionality `StreamingDataset` is built on. A `Permutation` defines a customizable row ordering, and supports
map-style access through `__getitem__()` and batched access through `__getitems__()`.

## Version Pinning {#version-pinning}

A `Permutation` is pinned to the version of its base table at the time it was constructed, and every read (including future reads
from a `StreamingDataset` that wraps it, and every DataLoader worker after it starts) resolves against that pinned
version. This makes iteration deterministic across compactions and worker restarts: the permutation only addresses rows
that existed when it was built, and rows appended to the base table afterwards are not visible through the existing
permutation.

To include newly appended rows, build a new `Permutation` (and any `StreamingDataset` that depends on it)
against the updated table. Permutations written before this behavior shipped continue to
read as they did before.

## Output formats {#output-formats}

By default, a `Table` data loader emits Arrow data. `collate_fn` is PyTorch's batching hook: PyTorch calls it to
turn the fetched items into one batch. PyTorch's default collate function only knows how to combine tensors, NumPy
arrays, numbers, dicts, and lists, so it does not accept Arrow data directly. When using a `Table` directly, pass
LanceDB's `lancedb.util.tbl_to_tensor` helper as PyTorch's `collate_fn`; it converts numeric Arrow columns into a
column-major `torch.Tensor` with shape `(columns, rows)`.

`Permutation` works differently: its default output is a list of Python dicts, which PyTorch's default collate function
can batch into a dict of tensors. This is usually more convenient when you are getting started. However, there is a
significant performance penalty converting from Arrow, Lance's internal representation, to this default format. If you
want the default PyTorch dict-of-tensors behavior, use a `Permutation` as-is; if you want direct Arrow-to-tensor
conversion, either pass `lancedb.util.tbl_to_tensor` as `collate_fn` with a direct `Table` or configure a `Permutation`
with one of the transform formats described below.

To address this, the `Permutation` class provides a set of builtin transform functions that can be applied to map
the Arrow data in different ways. The `arrow` and `polars` formats will always avoid data copies. However, `numpy`,
`pandas`, and `torch_col` formats will also avoid data copies in most cases. The `python`, `python_col`, and
`torch` formats will all require at least one full copy of the data and are the slowest options.

### Using the torch column format with a torch data loader {#using-the-torch-column-format-with-a-torch-data-loader}

The `torch_col` format is the most efficient way to convert from Arrow to a `torch.Tensor`. It will convert the
entire Arrow batch to a _column-major_ `torch.Tensor`. In other words, given C columns and R rows, the resulting
Tensor will have shape `(C, R)`. However, this format generates an error if you are using a
`torch.utils.data.DataLoader` with the default collation function:

```text
TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor
```

This error occurs because the default collation function does not currently expect a single two-dimensional tensor.
It expects a list of tensors which it will then stack. This is what is output by the `torch` format but that format
requires a data copy. To avoid this error, and avoid data copies, you will need to provide a custom collation function
in addition to specifying the `torch_col` format.

<CodeGroup>
```python Python icon="python"
from lancedb.permutation import Permutation

permutation = Permutation.identity(table).with_format("torch_col")
dataloader = torch.utils.data.DataLoader(permutation, collate_fn=lambda x: x)
```
</CodeGroup>

This will now output a single two-dimensional tensor for each batch.

## Selecting columns {#selecting-columns}

By default, the `Table` class will return all columns in the table when used as input to PyTorch. If you only need
a subset of columns, you can significantly reduce your I/O requirements by selecting only the columns you need. The
`Permutation` class provides a `select_columns` method that provides this functionality.

<CodeGroup>
```python Python icon="python"
import pyarrow as pa
import torch
from lancedb.permutation import Permutation

db = lancedb.connect("memory://")
prompts = db.create_table(
    "prompts",
    pa.table(
        {
            "id": range(4),
            "prompt": ["north", "south", "east", "west"],
            "embedding": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]],
        }
    ),
)

permutation = Permutation.identity(prompts).select_columns(["id", "prompt"])
dataloader = torch.utils.data.DataLoader(
    permutation, batch_size=1024, shuffle=True
)

for batch in dataloader:
    print(sorted(batch))  # ['id', 'prompt']: the embedding is never read
```
</CodeGroup>

Each batch is a dict of the selected columns, the default `Permutation` format described above.