Files
lancedb/docs/web/indexing/incremental-indexing.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

99 lines
5.0 KiB
Plaintext

---
title: "Incremental Indexing"
sidebarTitle: "Incremental Indexing"
description: "Keep an index up-to-date as new data arrives, without a full rebuild."
icon: "refresh"
keywords: ["reindexing", "re-index", "incremental indexing", "reindex", "index_remap", "spfresh_merge"]
availability:
embedded: available
enterprise: available
---
import {
PyReindexingIncremental as ReindexingIncremental,
} from '/snippets/indexing.mdx';
As you add new data to your LanceDB tables, your indexes may become outdated.
Reindexing is the process of updating the index to account for new data. This
applies to either a full-text search (FTS) index or a vector index.
Reindexing is an important operation to run periodically as your data grows,
as it has performance implications.
As data is being added and a reindex operation is running, LanceDB will
combine results from the existing index with exhaustive/flat search on the
new data. This is done to ensure that you're still retrieving results over
all your data, but it does come at a performance cost. The more data that you
add without reindexing, the impact on latency (due to exhaustive search) can
be noticeable.
Rather than dropping an existing index entirely and reindexing from scratch,
LanceDB supports **incremental indexing**.
<Note>
**Enterprise job types:** an index build runs as `reindex_` followed by the
algorithm, such as `reindex_ivf_pq`, `reindex_btree` or `reindex_fts`, and so
does folding new rows into an existing index. A create-only build, one that must
not replace an existing index, runs as `create_` followed by the algorithm
instead; Python's `create_scalar_index` and `create_fts_index` ask for that by
default. The name says which algorithm is building, not a separate feature.
</Note>
## Triggering an incremental update {#incremental-reindexing}
In the embedded library, you trigger an incremental indexing operation on
updated data with the `optimize()` method on a table. A deployment does this
for you and does not support `optimize()`, as described below.
Table optimization performs three maintenance operations:
1. **Compaction**: merges small fragments into larger ones to improve read performance
2. **Pruning/Cleanup**: removes files from versions older than a retention window (7 days by default) -- see [Cleanup](/maintenance/cleanup) for how retention works
3. **Index update**: adds newly-ingested data to existing vector, scalar, and FTS indexes
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{ReindexingIncremental}
</CodeBlock>
</CodeGroup>
<Badge color="red">Enterprise</Badge>
LanceDB Enterprise supports incremental reindexing through an automated background process. Once enough new rows have been added to a table, the system automatically adds them to its indexes. As the dataset grows, indexes are asynchronously updated in the background.
- While indexes are being rebuilt, queries use brute force methods on unindexed rows, which may temporarily increase latency. If you can omit unindexed rows, set `fast_search=True` to search only indexed data.
This can return incomplete results while indexing catches up.
- Use `index_stats()` to view the number of unindexed rows. This will be zero when indexes are fully up-to-date. If you call `wait_for_index(...)`, it polls the same status and can time out while continuous writes keep adding unindexed rows.
- Rows below the deployment's threshold stay unindexed until more arrive, so the count can stay above zero without further writes. `FM` indexes are not updated in the background.
The benefit of using LanceDB Enterprise is that it automates the reindexing process
and operates continuously in the background, minimizing the impact on latency under high loads.
In LanceDB Embedded, you must manually manage the reindexing cadence based on your data growth and performance needs.
Opening a table in LanceDB Embedded logs a warning at most once per table per
process when any index has at least 1,000,000 unindexed rows. Set
`LANCEDB_LOG=warn` to surface this log if your logger hides warnings by default.
## Index maintenance jobs {#index}
Two further Enterprise job types keep an index correct as the rows underneath
it move, rather than adding new rows to it:
| Job type | What it does |
| --- | --- |
| `index_remap` | Reassigns index entries after row ids shift underneath an index (the kind of remap a delta compaction can require). |
| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. |
See [Advanced Conflict Resolution](/enterprise/advanced-conflict-resolution)
for why `index_remap` exists as a separate, deferred job rather than
happening synchronously inside every compaction.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Distributed Indexing" icon="diagram-project" href="/enterprise/distributed-indexing">
How an index build itself scales across the jobs fleet.
</Card>
<Card title="GPU Indexing" icon="microchip" href="/indexing/gpu-indexing">
Building a vector index faster with GPU acceleration.
</Card>
</CardGroup>