lancedb/docs/web/indexing/reindexing.mdx

---
title: "Building and repairing indexes"
sidebarTitle: "Build & Repair"
description: "Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing."
icon: "refresh"
keywords: ["reindexing", "re-index", "incremental indexing", "reindex"]
availability:
  oss: available
  enterprise: available
  summary: >-
    Indexes are kept up to date in the embedded library and in LanceDB Enterprise. In
    the embedded library you add new rows to them with `optimize()`; a
    deployment does it in the background, and `optimize()` is not available
    there.
---

import {
    PyReindexingIncremental as ReindexingIncremental,
    PyGpuIndexCuda as GpuIndexCuda,
    PyGpuIndexMps as GpuIndexMps,
} from '/snippets/indexing.mdx';

As you add new data to your LanceDB tables, your indexes may become outdated.
Reindexing is the process of updating the index to account for new data. This applies to either a full-text search (FTS) index or a vector index. Reindexing is an important operation to run periodically as your data grows, as it has performance implications.

As data is being added and a reindex operation is running, LanceDB will combine results from the existing index with exhaustive/flat search on the new data. This is done to ensure that you're still retrieving results over all your data, but it does come at a performance cost. The more data that you add without reindexing, the impact on latency (due to exhaustive search) can be noticeable.

Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports **incremental indexing**.

<Note>
**Enterprise job types:** an index build runs as `reindex_` followed by the
algorithm, such as `reindex_ivf_pq`, `reindex_btree` or `reindex_fts`, and so
does folding new rows into an existing index. A create-only build, one that must
not replace an existing index, runs as `create_` followed by the algorithm
instead; Python's `create_scalar_index` and `create_fts_index` ask for that by
default. The name says which algorithm is building, not a separate feature.
</Note>

## Incremental Reindexing {#incremental-reindexing}

In the embedded library, you trigger an incremental indexing operation on
updated data with the `optimize()` method on a table. A deployment does this
for you and does not support `optimize()`, as described below.

Table optimization performs three maintenance operations:

1. **Compaction**: merges small fragments into larger ones to improve read performance
2. **Pruning/Cleanup**: removes files from versions older than a retention window (7 days by default)
3. **Index update**: adds newly-ingested data to existing vector, scalar, and FTS indexes

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {ReindexingIncremental}
    </CodeBlock>
</CodeGroup>

<Badge color="red">Enterprise</Badge>

LanceDB Enterprise supports incremental reindexing through an automated background process. Once enough new rows have been added to a table, the system automatically adds them to its indexes. As the dataset grows, indexes are asynchronously updated in the background.

- While indexes are being rebuilt, queries use brute force methods on unindexed rows, which may temporarily increase latency. If you can omit unindexed rows, set `fast_search=True` to search only indexed data.
  This can return incomplete results while indexing catches up.
- Use `index_stats()` to view the number of unindexed rows. This will be zero when indexes are fully up-to-date. If you call `wait_for_index(...)`, it polls the same status and can time out while continuous writes keep adding unindexed rows.
- Rows below the deployment's threshold stay unindexed until more arrive, so the count can stay above zero without further writes. `FM` indexes are not updated in the background.

The benefit of using LanceDB Enterprise is that it automates the reindexing process
and operates continuously in the background, minimizing the impact on latency under high loads.
In OSS, you must manually manage the reindexing cadence based on your data growth and performance needs.

## Disk utilization {#disk-utilization}

Compaction by itself does not immediately free disk space, and can temporarily increase it because new
compacted files are written before old-version files are deleted. Disk space is reclaimed when old versions
are pruned during cleanup. Set retention only as low as your rollback and time-travel requirements allow.

If you need to reclaim space more aggressively in OSS, use a shorter retention window:
<CodeGroup>
    ```python Python icon=Python
    from datetime import timedelta

    table.optimize(cleanup_older_than=timedelta(days=1))
    ```
</CodeGroup>

## Building on GPUs {#gpu}

GPU acceleration can reduce the time needed to build vector indexes on large
datasets. Actual build time depends on the index configuration, vector
dimensions, storage throughput and available GPU resources.

<Info>
**Automatic GPU indexing**
<Badge color="red">Enterprise-only</Badge>

Automatic GPU Indexing is currently only available in [LanceDB Enterprise](/enterprise/).
Please [contact us](https://www.lancedb.com/contact) to enable this feature for your deployment.

The vector index is created when you call `create_index`. The backend uses GPU resources
to build `IVF_PQ` indexes, with the number of GPUs set by the deployment's configuration.

This process is also asynchronous by default, but you can use `wait_for_index` to convert it
into a synchronous process by waiting until the index is built.

</Info>

`wait_for_index(...)` waits for the index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. It can time out if the table is receiving continuous writes while the build is trying to catch up.

GPU acceleration changes how the vector index is built, not the lifecycle after the build. In the
embedded library, rows appended later still need `optimize()` before they are part of the index; a
deployment adds them in the background. `fast_search()` only searches indexed rows.

### Manual GPU indexing in LanceDB OSS {#gpu-manual-gpu-indexing-in-lancedb-oss}

You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
[PyTorch>2.0](https://pytorch.org/). Note that GPU-based indexing is currently only
supported by the synchronous SDK in LanceDB OSS.

Specify the values `cuda` or `mps` (on Apple Silicon) for the `accelerator` parameter
to enable GPU training on your device.

#### GPU indexing on Linux {#gpu-gpu-indexing-on-linux}

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {GpuIndexCuda}
    </CodeBlock>
</CodeGroup>

#### GPU indexing on macOS (Apple Silicon) {#gpu-gpu-indexing-on-macos-apple-silicon}

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {GpuIndexMps}
    </CodeBlock>
</CodeGroup>

### Performance considerations {#gpu-performance-considerations}

- GPU memory usage scales with `num_partitions` and vector dimensions
- Size GPU memory for the training batches and index configuration
- Monitor memory use while choosing training batch sizes
- Measure build time and recall before changing training settings

### Troubleshooting {#gpu-troubleshooting}

If you encounter the error `AssertionError: Torch not compiled with CUDA enabled`,
you need to [install PyTorch with CUDA support](https://pytorch.org/get-started/locally/).