mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-18 17:52:28 +00:00
Jack Ye
b21fb2a4dc
The pages describing this code lived in lancedb/docs, a repository with ten stars that neither contributing guide mentioned. A user who hit a wrong page while reading the source had to find a separate, near-invisible repository to fix it. These 102 pages now sit beside the code, and a documentation fix is a pull request in the repository the reader already had open. `docs/web` is a complete Mintlify site: `cd docs/web && mint dev` serves it with no other checkout. Enterprise pages stay private and are merged in when the published site is assembled, so links into them do not resolve here — every open-source page does. `docs/web-tests` holds the tests the code examples are extracted from. They ran against a published wheel before; here they can run against this repository's own SDK, so an example that stops compiling fails the pull request that broke it rather than being noticed later. Both directories sit beside `docs/src` and are invisible to it: mkdocs reads `docs_dir: src`, so the SDK reference builds and deploys exactly as before.
60 lines
3.5 KiB
Plaintext
60 lines
3.5 KiB
Plaintext
---
|
|
title: "Keeping Indexes Up-to-Date with Reindexing"
|
|
sidebarTitle: "Reindexing"
|
|
description: "Learn how to keep your indexes up-to-date in LanceDB using incremental indexing, including best practices for adding new records without full reindexing."
|
|
icon: "refresh"
|
|
keywords: ["reindexing", "re-index", "incremental indexing", "reindex"]
|
|
---
|
|
import { PyReindexingIncremental as ReindexingIncremental } from '/snippets/indexing.mdx';
|
|
|
|
As you add new data to your LanceDB tables, your indexes may become outdated.
|
|
Reindexing is the process of updating the index to account for new data -- this applies to either a full-text search (FTS) index or a vector index. Reindexing is an important operation to run periodically as your data grows, as it has performance implications.
|
|
|
|
As data is being added and a reindex operation is running, LanceDB will combine results from the existing index with exhaustive/flat search on the new data. This is done to ensure that you're still retrieving results over all your data, but it does come at a performance cost. The more data that you add without reindexing, the impact on latency (due to exhaustive search) can be noticeable.
|
|
|
|
Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports **incremental indexing**.
|
|
|
|
## Incremental Reindexing {#incremental-reindexing}
|
|
|
|
You can manually trigger an incremental indexing operation on updated data
|
|
using the `optimize()` method on a table.
|
|
|
|
Table optimization performs three maintenance operations:
|
|
|
|
1. **Compaction**: merges small fragments into larger ones to improve read performance
|
|
2. **Pruning/Cleanup**: removes files from versions older than a retention window (7 days by default)
|
|
3. **Index update**: adds newly-ingested data to existing vector, scalar, and FTS indexes
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{ReindexingIncremental}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
LanceDB Enterprise support incremental reindexing through an automated background process. When new data is added to a table, the system automatically triggers a new index build. As the dataset grows, indexes are asynchronously updated in the background.
|
|
|
|
- While indexes are being rebuilt, queries use brute force methods on unindexed rows, which may temporarily increase latency. To avoid this, set `fast_search=True` to search only indexed data.
|
|
- Use `index_stats()` to view the number of unindexed rows. This will be zero when indexes are fully up-to-date. If you call `wait_for_index(...)`, it polls the same status and can time out while continuous writes keep adding unindexed rows.
|
|
|
|
The benefit of using LanceDB Enterprise is that it automates the reindexing process
|
|
and operates continuously in the background, minimizing the impact on latency under high loads.
|
|
In OSS, you must manually manage the reindexing cadence based on your data growth and performance needs.
|
|
|
|
## Disk utilization {#disk-utilization}
|
|
|
|
Compaction by itself does not immediately free disk space, and can temporarily increase it because new
|
|
compacted files are written before old-version files are deleted. Disk space is reclaimed when old versions
|
|
are pruned during cleanup. Set retention only as low as your rollback and time-travel requirements allow.
|
|
|
|
If you need to reclaim space more aggressively in OSS, use a shorter retention window:
|
|
<CodeGroup>
|
|
```python Python icon=Python
|
|
from datetime import timedelta
|
|
|
|
table.optimize(cleanup_older_than=timedelta(days=1))
|
|
```
|
|
</CodeGroup>
|
|
|