docs: own every page path, and port the upstream changes

lancedb now holds a page at every path the site publishes, including the twelve Enterprise ones. Those are real open-source pages -- what the capability is, and what the embedded form does instead -- rather than blank placeholders, so the open-source site is complete on its own. sophon becomes an overlay whose page replaces the one at the same path, which means navigation lives in one place and the two repositories only have to agree on a path. That removes the whole class of problem where a page could only be reached from one of the two trees. The open-source site went from twenty-four dangling Enterprise links to one, and that one points at the dataset pages the build repository owns. Ports four changes that landed on the docs repository after this branch left it: #344 indexing and quantization rewritten #354 LeRobot native reader #361 training/index renamed to training/data-loading and rewritten #362 distance metric mismatch falls back to a brute-force scan None applied as patches, because these files were merged or split here -- quantization into the vector index page, multi-vector into vector search, performance into four pages. Each was ported by taking the upstream page and re-applying the restructure on top, which also picked up a correction that would otherwise have been lost: the stricter brute-force threshold for multi-vector columns is a Cloud limit, not an Enterprise one. The upstream LeRobot page linked to a Geneva page that no longer exists; it now points at the feature-engineering pages that replaced it.
2026-10-13 07:12:27 +00:00 · 2026-09-25 15:19:54 -07:00
parent 0913e2a436
commit 70c10e4aa4
34 changed files with 1292 additions and 1103 deletions
@@ -51,4 +51,4 @@ you ended up with.

 A curated result you want to keep is a
 [materialized view](/management/materialized-view) -- stored, refreshed, and
-readable by a [data loader](/training) like any other table.
+readable by a [data loader](/training/data-loading) like any other table.
@@ -90,7 +90,7 @@ dataloader = torch.utils.data.DataLoader(permutation)

 This is the right tool for epoch-level shuffling, where you want every row
 eventually and only the order varies. `TABLESAMPLE` is for when you want fewer
-rows. See [Training](/training).
+rows. See [Training](/training/data-loading).

 ## Reference {#reference}

@@ -134,6 +134,10 @@
    {
      "source": "/reranking/:slug*",
      "destination": "/reranking"
+    },
+    {
+      "source": "/training",
+      "destination": "/training/data-loading"
    }
  ],
  "navigation": {
@@ -209,7 +213,7 @@
            "group": "Training",
            "pages": [
              "training/why-lancedb",
-              "training/index",
+              "training/data-loading",
              "training/torch"
            ]
          },
@@ -222,6 +226,7 @@
              "management/table",
              "tables/branching",
              "management/materialized-view",
+              "management/view",
              "management/function"
            ]
          },
@@ -249,12 +254,37 @@
              }
            ]
          },
+          {
+            "group": "Security",
+            "pages": [
+              "enterprise/authentication",
+              "governance/authorization",
+              "enterprise/security"
+            ]
+          },
+          {
+            "group": "Deployment",
+            "pages": [
+              "enterprise/index",
+              "enterprise/architecture",
+              {
+                "group": "Cloud Providers",
+                "pages": [
+                  "enterprise/deployment/index",
+                  "enterprise/deployment/azure"
+                ]
+              },
+              "enterprise/benchmarks"
+            ]
+          },
          {
            "group": "SQL Reference",
            "pages": [
              "sql/index",
+              "search/sql/index",
              "sql/data-types",
              "sql/select",
+              "search/sql/fts-sql",
              "sql/subqueries",
              "sql/join",
              "sql/aggregation",
@@ -279,7 +309,8 @@
                "group": "FAQ",
                "pages": [
                  "faq/index",
-                  "faq/faq-oss"
+                  "faq/faq-oss",
+                  "faq/faq-enterprise"
                ]
              }
            ]
@@ -0,0 +1,20 @@
+---
+title: "Enterprise architecture"
+sidebarTitle: "Architecture"
+description: "The tiers a deployment runs, and how they scale independently."
+icon: cubes
+keywords: ["architecture", "control plane", "data plane", "indexer"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+A deployment separates compute from storage so each scales on its own: table
+data and indexes live in object storage, while query serving and background work
+run on machines that can be replaced without any of them owning the data.
+
+[Architecture](/architecture) covers this alongside the embedded form, which is
+the comparison most readers want first.
@@ -0,0 +1,26 @@
+---
+title: "Authentication"
+sidebarTitle: "Authentication"
+description: "Establishing who is calling: API keys and identity provider integration."
+icon: key
+keywords: ["authentication", "api key", "oidc", "identity"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+A deployment authenticates every request before
+[authorization](/governance/authorization) decides what it may do.
+
+```python
+db = lancedb.connect("db://my-database", api_key="...", region="us-east-1")
+```
+
+An API key belongs in the environment rather than in source. See
+[Database](/management/database#deployment) for the connection options.
+
+Embedded LanceDB has nothing to authenticate to -- your process opens the files
+directly, and the object store's own credentials are the only gate.
@@ -0,0 +1,20 @@
+---
+title: "Benchmarks"
+sidebarTitle: "Benchmarks"
+description: "Published performance results for LanceDB Enterprise."
+icon: chart-line
+keywords: ["benchmarks", "performance", "throughput", "latency"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+Benchmark results are specific to a deployment's shape -- cluster size, storage
+class, dataset -- so they are published with LanceDB Enterprise.
+
+For tuning your own workload rather than comparing published numbers, see
+[Query plans](/search/explain) and the performance notes on
+[Search](/search#performance) and [Insert](/ingestion/insert#performance).
@@ -0,0 +1,17 @@
+---
+title: "Azure deployment guide"
+sidebarTitle: "Azure"
+description: "Deploying LanceDB Enterprise on Azure."
+icon: cloud
+keywords: ["azure", "deployment", "aks"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+The Azure deployment procedure is published with LanceDB Enterprise. See
+[Deployment](/enterprise/deployment) for the general shape, and
+[contact us](mailto:contact@lancedb.com) to get started.
@@ -0,0 +1,20 @@
+---
+title: "Deployment guide"
+sidebarTitle: "Overview"
+description: "Standing up a LanceDB Enterprise deployment."
+icon: server
+keywords: ["deployment", "install", "cluster", "kubernetes"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+Deployment is specific to your cloud and account, so the procedure is published
+with LanceDB Enterprise. [Contact us](mailto:contact@lancedb.com) to get
+started.
+
+Embedded LanceDB has nothing to deploy -- it is a library your process links
+against. See [Architecture](/architecture#embedded).
@@ -0,0 +1,29 @@
+---
+title: "LanceDB Enterprise"
+sidebarTitle: "Overview"
+description: "What a deployment adds over embedded LanceDB."
+icon: building
+keywords: ["enterprise", "deployment", "cluster"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+LanceDB Enterprise runs the lakehouse as a service: a control plane for
+configuration, identity and policy; a data plane serving reads and writes; and
+indexers building indexes and compacting outside the request path.
+
+What it adds over the embedded form:
+
+| | |
+| --- | --- |
+| [SQL](/sql) | an Arrow Flight SQL endpoint |
+| [Functions](/management/function) | Python running on workers |
+| [Jobs](/basics/jobs) | server-side work with ids, status and events |
+| [Authorization](/governance/authorization) | per-object grants |
+| Scale | a cluster rather than your process |
+
+See [Architecture](/architecture) for the two shapes side by side.
@@ -0,0 +1,20 @@
+---
+title: "Compliance"
+sidebarTitle: "Compliance"
+description: "Certifications, data handling, and the controls a deployment provides."
+icon: certificate
+keywords: ["compliance", "security", "soc2", "encryption", "audit"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+Compliance posture -- certifications, data residency, encryption at rest and in
+transit, audit logging -- is a property of a deployment, so it is documented
+with LanceDB Enterprise.
+
+For embedded LanceDB, these are properties of the object store you point it at
+and the process you run it in; LanceDB adds no layer of its own.
@@ -0,0 +1,19 @@
+---
+title: "LanceDB Enterprise FAQ"
+sidebarTitle: "LanceDB Enterprise"
+description: "Common questions about running LanceDB Enterprise."
+icon: circle-question
+keywords: ["faq", "enterprise"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+Questions specific to running a deployment are answered with LanceDB
+Enterprise.
+
+For questions about the embedded form, see the
+[LanceDB OSS FAQ](/faq/faq-oss).
@@ -0,0 +1,26 @@
+---
+title: "Authorization"
+sidebarTitle: "Authorization"
+description: "Who may do what: privileges granted on catalog objects to principals, groups, roles, and API keys."
+icon: shield-halved
+keywords: ["authorization", "authz", "privileges", "rbac", "access control"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+[Authentication](/enterprise/authentication) establishes who is calling.
+Authorization decides what they may do.
+
+An access control entry grants **one privilege** on **one object** to **one
+subject**. Objects are addressed by kind and path -- `system`, `database:db`,
+`namespace:db/ns`, `table:db/ns/table` -- and the hierarchy is not decorative:
+reaching a table requires `USAGE` on the namespace containing it and on the
+database containing that.
+
+Embedded LanceDB has no such layer. Access is whatever the object store
+enforces on the bucket or directory, which is coarser and has no concept of a
+table.
@@ -61,7 +61,7 @@ feature engineering, search and retrieval, and efficient training data access.
  <Card title="Train and fine-tune models" icon="fire" href="/training/why-lancedb">
    Learn why LanceDB works well as the data layer for training workloads.
  </Card>
-  <Card title="Load data into PyTorch" icon="boxes-stacked" href="/training/">
+  <Card title="Load data into PyTorch" icon="boxes-stacked" href="/training/data-loading">
    Use LanceDB tables and permutations for projected, shuffled, random-access training reads.
  </Card>
  <Card title="Browse ready-to-use datasets" icon="database" href="/datasets">
@@ -1,169 +1,39 @@
 ---
 title: "Indexing Data"
 sidebarTitle: "Overview"
-description: "Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search."
+description: "Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more."
 icon: "list"
 ---

-An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads:
+An **index** is a data structure that facilitates efficient scans and lookups on an embedded dataset. 
+LanceDB provides a comprehensive suite of indexes to optimize performance across different use cases and data types:

- **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors
+- **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings)
 - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases
- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices)
+- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data

-<Tip title="Working with scalar indices">
-Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
-
- Vector search (prefilter or post-filter results using metadata)
- Full-text search (combining keyword matching with structured filters) 
- SQL scans (optimizing WHERE clauses on scalar columns)
- Key-value lookups (enabling rapid primary key-based retrievals)
-</Tip>
-
-<Note>
-In SQL, see [`CREATE INDEX` and `DROP INDEX`](/sql/ddl#indexes) -- one column, server-chosen name.
-</Note>
-
-## Supported Index Types {#supported-index-types}
-
-LanceDB provides a comprehensive suite of indexing strategies for different data types and use cases:
+## Supported Indexes

 | Index | Use Case | Description |
 | :--------- | :------- | :---------- |
-| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.<br />Distance metrics: `l2` `cosine` `dot` `hamming`<br />Quantizations: `None/Flat` `PQ` `SQ` `RQ`|
-| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.<br />Distance metrics: `l2` `cosine` `dot`<br />Quantizations: `None/Flat` `SQ` `PQ`|
+| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search. <br />**Quantizations**: `None/Flat` `PQ` `SQ` `RQ`|
+| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs in each partition. Provides improved search quality over pure IVF while maintaining scalability.<br />**Quantizations**: `None/Flat` `SQ` `PQ`|
 | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. |
 | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
 | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
 | `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
 | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |

-<Note>
-TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
-</Note>
-
-<Info>
-**Operational checks**
-
-For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.
-
-By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to
-`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar
-columns, not list columns; use `LabelList` for list containment filters.
-</Info>
-
-### Quantization Types {#quantization-types}
-
-Vector indexes can use different quantization methods to compress vectors and improve search performance:
+## Quantization
+LanceDB also supports several [quantization](/indexing/vector-index) methods, used by vector indexes to compress vectors and reduce storage requirements:

 | Quantization | Use Case | Description |
 | :----------- | :------- | :---------- |
 | `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. |
-| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
+| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |

-## Understanding the IVF-PQ Index {#understanding-the-ivf-pq-index}
-
-An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-nearest neighbors (kNN)** problem. 
-It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below.
-
-LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy.
-
-## IVF-PQ {#ivf-pq}
-
-LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. 
-The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall.
-
-### Product Quantization {#product-quantization}
-
-Quantization is a compression technique used to reduce the dimensionality of an embedding to speed up search.
-
-Product quantization (PQ) works by dividing a large, high-dimensional vector of size into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector.
-
-![](/static/assets/images/indexing/ivfpq_pq_desc.png)
-
-It's important to remember that quantization is a *lossy process*, i.e., the reconstructed vector is not identical to the original vector. This results in a trade-off between the size of the index and the accuracy of the search results.
-
-As an example, consider starting with 128-dimensional vector consisting of 32-bit floats. Quantizing it to an 8-bit integer vector with 4 dimensions as in the image above, we can significantly reduce memory requirements.
-
-<Note title="Effect of quantization">
-Original: `128 × 32 = 4096` bits
-Quantized: `4 × 8 = 32` bits
-
-Quantization results in a **128x** reduction in memory requirements for each vector in the index, which is substantial.
-</Note>
-
-### Inverted File Index (IVF) Implementation {#inverted-file-index-ivf-implementation}
-
-While PQ helps with reducing the size of the index, IVF primarily addresses search performance. The primary purpose of an inverted file index is to facilitate rapid and effective nearest neighbor search by narrowing down the search space.
-
-In IVF, the PQ vector space is divided into *Voronoi cells*, which are essentially partitions that consist of all the points in the space that are within a threshold distance of the given region's seed point. These seed points are initialized by running K-means over the stored vectors. The centroids of K-means turn into the seed points which then each define a region. These regions are then are used to create an inverted index that correlates each centroid with a list of vectors in the space, allowing a search to be restricted to just a subset of vectors in the index.
-
-![](/static/assets/images/indexing/ivfpq_ivf_desc.webp)
-
-During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query.
-![](/static/assets/images/indexing/ivfpq_query_vector.webp)
-
-## HNSW Index Implementation {#hnsw-index-implementation}
-
-Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly.
-
-### Types of ANN Search Algorithms {#types-of-ann-search-algorithms}
-
-Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. 
-For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly.
-
-There are three main types of ANN search algorithms:
-
-* **Tree-based search algorithms**: Use a tree structure to organize and store data points.
-* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice.
-* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. 
-
-HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a k-nearest neighbor (or k-approximate nearest neighbor) graph, which we outline below.  
-HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list.
-
-### Understanding k-Nearest Neighbor Graphs {#understanding-k-nearest-neighbor-graphs}
-
-The k-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple:
-
-* Each vector in the dataset is given an associated vertex.
-* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex.
-* For some applications (including nearest-neighbor search), the incoming edges are also added.
-
-Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors:
-
-* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node).
-* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex.
-* Repeat until a local optimum is found.
-
-The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors.
-
-Computing a k-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the k-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster.  
-In fact, another data structure is not needed: This can be done "incrementally".
-That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex.
-
-One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index.
-
-### Hierarchical Navigable Small Worlds (HNSW) {#hierarchical-navigable-small-worlds-hnsw}
-
-HNSW builds on k-ANN in two main ways:
-
-* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant.
-* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list.
-
-This recursive structure can be thought of as separating into layers:
-
-* At the bottom-most layer, a k-ANN graph on the whole dataset is present.
-* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present.
-* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer.
-
-Then the greedy search routine operates as follows:
-
-* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
-* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
-* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
-
 ## Performance {#performance}

 #### Vector indexes {#vector-indexes}
@@ -177,7 +47,7 @@ If vector search latency climbs with table size (i.e., queries that ran in milli
 | `IVF_HNSW_SQ` | Best recall/latency for unfiltered search; higher latency variance under selective filters. |
 | `IVF_FLAT` | Required for binary vectors with `hamming`. |

-The distance metric is fixed once the index is built. Pick the distance metric based on how the embedding model was trained: `cosine` (unnormalized), `dot` (already-normalized, best performance), `l2` (general-purpose, default), `hamming` (binary). For parameter tuning, see [Vector Indexing](/indexing/vector-index).
+The distance metric is fixed once the index is built. Pick the distance metric based on how the embedding model was trained: `cosine` (unnormalized), `dot` (already-normalized, best performance), `l2` (general-purpose, default), `hamming` (binary). If a query explicitly requests a different metric, the index cannot be used and the query falls back to a brute-force scan; LanceDB Enterprise may reject that scan on a large table. For parameter tuning, see [Vector Indexing](/indexing/vector-index).

 #### Scalar indexes {#scalar-indexes}

@@ -4,6 +4,7 @@ sidebarTitle: "Scalar Index"
 description: "Learn how to use scalar indexes in LanceDB for efficient metadata filtering and query optimization."
 icon: "tree"
 ---
+
 import {
    PyScalarIndexBuild as ScalarIndexBuild,
    PyScalarIndexWait as ScalarIndexWait,
@@ -27,7 +28,16 @@ LanceDB supports four types of scalar indexes:
 - `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries.
 - `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.

-## Choosing the Right Index Type {#choosing-the-right-index-type}
+<Tip title="Working with scalar indices">
+Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
+
+- Vector search (prefilter results using metadata)
+- Full-text search (combining keyword matching with structured filters) 
+- SQL scans (optimizing WHERE clauses on scalar columns)
+- Key-value lookups (enabling rapid primary key-based retrievals)
+</Tip>
+
+## Choosing the Right Index Type

 | Data Type                                                       | Filter                                    | Index Type   |
 |:----------------------------------------------------------------|:------------------------------------------|:-------------|
@@ -36,9 +46,9 @@ LanceDB supports four types of scalar indexes:
 | List of low cardinality of numbers or strings                   | `array_has_any`, `array_has_all`          | `LABEL_LIST` |
 | String or binary (`Utf8`, `LargeUtf8`, `Binary`, `LargeBinary`) | `contains(col, 'needle')`                 | `FM`         |

-## Scalar Index Operations {#scalar-index-operations}
+## Scalar Index Operations

-### 1\. Build the Index {#1-build-the-index}
+### 1. Build the Index

 You can create multiple scalar indexes within a table. By default, the index will be `BTREE`, but you can always configure another type like `BITMAP`

@@ -52,7 +62,7 @@ You can create multiple scalar indexes within a table. By default, the index wil
 If you are using LanceDB Enterprise, the `create_scalar_index` API returns immediately, but the building of the scalar index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_scalar_index()` or call `wait_for_index()` on the table.
 </Note>

-### 2\. Check Index Status {#2-check-index-status}
+### 2. Check Index Status

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -62,7 +72,7 @@ If you are using LanceDB Enterprise, the `create_scalar_index` API returns immed

 `wait_for_index(...)` waits until the named scalar indexes exist and `index_stats(...)` reports `num_unindexed_rows == 0`. If a table is receiving steady writes, that fully indexed state may not stabilize before the timeout.

-### 3\. Update the Index {#3-update-the-index}
+### 3. Update the Index

 Updating the table data (adding, deleting, or modifying records) requires that you also update the scalar index. This can be done by calling `optimize`, which will trigger an update to the existing scalar index.

@@ -76,7 +86,7 @@ Updating the table data (adding, deleting, or modifying records) requires that y
 New data added after creating the scalar index will still appear in search results if optimize is not used, but with increased latency due to a flat search on the unindexed portion. LanceDB Enterprise automates the optimize process, minimizing the impact on search speed.
 </Note>

-### 4\. Run Indexed Searches {#4-run-indexed-searches}
+### 4. Run Indexed Searches

 The following scan will be faster if the column `book_id` has a scalar index:

@@ -94,7 +104,7 @@ Scalar indexes can also speed up scans containing a vector search or full text s
    </CodeBlock>
 </CodeGroup>

-## Indexing nested fields {#indexing-nested-fields}
+## Indexing nested fields

 Scalar indexes can target a scalar field inside a struct by passing its full dotted path. The path is preserved end to end: it's the value you pass to `create_scalar_index`, it's what `list_indices()` reports under `columns`, and it's the column reference you use in filter predicates.

@@ -110,7 +120,7 @@ table.search().where("metadata.user_id = 42").limit(1).to_list()
 Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to FTS and vector indexes.
 </Note>

-## FM-Index for substring search {#fm-index-for-substring-search}
+## FM-Index for substring search

 The `FM` index is a scalar index built over string or binary columns that
 accelerates substring lookups expressed as `contains(col, 'needle')`. Unlike the
@@ -129,7 +139,7 @@ Use the FM-Index when:
 Pick `FTS` instead when you need word-level relevance ranking, phrase queries,
 or language-aware tokenization.

-### Create an FM-Index {#create-an-fm-index}
+### Create an FM-Index

 Build an FM-Index with the async `create_index` API by passing the `Fm` config in
 Python or `Index.fm()` in TypeScript. In Rust, use `Index::Fm(FmIndexBuilder::default())`.
@@ -166,7 +176,7 @@ table.search().where("contains(text, 'needle')").limit(10).to_pandas()

 `list_indices()` reports the index type as `"Fm"`.

-## Index UUID Columns {#index-uuid-columns}
+## Index UUID Columns

 LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`), enabling efficient lookups and filtering on UUID-based primary keys.

@@ -177,7 +187,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
 - TypeScript SDK version `0.19.0` or later
 </Note>

-### 1\. Define UUID Type {#1-define-uuid-type}
+### 1. Define UUID Type

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -185,7 +195,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
    </CodeBlock>
 </CodeGroup>

-### 2\. Generate UUID Data {#2-generate-uuid-data}
+### 2. Generate UUID Data

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -193,7 +203,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
    </CodeBlock>
 </CodeGroup>

-### 3\. Create Table with UUID Column {#3-create-table-with-uuid-column}
+### 3. Create Table with UUID Column

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -201,7 +211,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
    </CodeBlock>
 </CodeGroup>

-### 4\. Create and Wait for the Index {#4-create-and-wait-for-the-index}
+### 4. Create and Wait for the Index

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -209,7 +219,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
    </CodeBlock>
 </CodeGroup>

-### 5\. Perform Operations with the UUID Index {#5-perform-operations-with-the-uuid-index}
+### 5. Perform Operations with the UUID Index

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -217,7 +227,7 @@ LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`
    </CodeBlock>
 </CodeGroup>

-## Index nested fields {#index-nested-fields}
+## Index nested fields

 You can build a scalar index on a field inside a struct column by passing the
 canonical dot-separated path to `create_index`. This is useful when filters
@@ -1,7 +1,7 @@
 ---
 title: "Vector Indexes"
 sidebarTitle: "Vector Index"
-description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes."
+description: "Build and manage LanceDB vector indexes."
 icon: "arrow-up-right-dots"
 ---

@@ -23,216 +23,156 @@ import {
    PyVectorIndexDistanceRange as VectorIndexDistanceRange,
    PyVectorIndexBypassRecall as VectorIndexBypassRecall,
    PyVectorIndexCustomName as VectorIndexCustomName,
+    PyQuantizationCustomParams as QuantizationCustomParams,
 } from '/snippets/indexing.mdx';

-You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two kinds of vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**.
+Vector indexes are robust tools in facilitating fast searches across large numeric datasets. 
+LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with several techniques that provide benefits across a variety of use cases.

+## Choosing the Right Index
+
+LanceDB vector indexes are combined with several [quantization](/indexing/vector-index) techniques to admit efficient storage.
+The following table lists provided quantized vector indexes and their common use cases. You can specify index type manually in
+Lance with `index_type`.
+
+
+| If your priority is... | Use this index | Why | Approx. compression ratio | Python config class |
+| :--- | :--- | :--- | :--- | :--- |
+| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | `IvfRq` |
+| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` |
+| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` |
+| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` |
+
+<Warning>
+If your vector search frequently includes metadata filters (`where(...)`), use `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance.
+</Warning>
+
+## Understanding Vector Indexes
+
+LanceDB offers two vector indexes, which can be created on any numeric Lance dataset: 
+**Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**.
+
+### IVF
+
+The **Inverted File Index (IVF)** accelerates ANN searches by drastically reducing the search space. The index consists of
+a small set of *centroids* corresponding to an approximate solution to the 
+[$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. 
+Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, 
+called its *partition*. 
+<Frame caption="Example partition of a geometric space, with colored marks denoting centroids.">
+  <img src="/static/assets/images/indexing/ivfpq_ivf_desc.webp" alt="IVF vector-space partitioning" />
+</Frame>
+
+At query time, we can compare the queried vector to the smaller set of *centroids* (as opposed to the entire dataset)
+for a closest match, then run a brute-force comparison against its resulting partition. This technique quickly prunes a large search space,
+giving an approximate ANN result.
+
+However, observe that a queried vector may lie near the boundary of $2$ or more partitions; thus, the true nearest neighbors are scattered across several different partitions.
+ To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of partitions searched. 
+A high `nprobes` parameter will yield more accurate results at slightly higher runtime.
+
+<Frame caption="Edge case for single-partition brute force.">
+  <img src="/static/assets/images/indexing/ivfpq_query_vector.webp" alt="IVF vector-space partitioning" />
+</Frame>
+
+### HNSW
+**Hierarchical Navigable Small World (HNSW)** constructs a layered graph hierarchy on the vector set, with edges representing distances. 
+We can visualize an HNSW index as a vertical stack of graphs, with the top layer having very few edges and
+each other layer having a multiplicative factor more edges than the layer above it.
+
+High layers of a HNSW hierarchy represent sparse, higher-distance networks, and lower layers represent finer, lower-distance networks. 
+To query a vector $q$, the index proceeds iteratively through layers, first finding $q$'s nearest neighbor in the graph, then proceeding recursively
+through the induced subhierarchy until the lowest layer is reached.
+
+<Frame caption="HNSW hierarchy: sparse network at the top layer, fine network at the bottom.">
+  <img src="/static/assets/images/indexing/hnsw_layered_graph.png" alt="HNSW layered graph hierarchy" />
+</Frame>
+
+To visualize this process, imagine that you must drive your car from San Francisco to a specific house in Boston. Initially, you must first drive thousands of miles
+on the interstate freeway I-90 E. Eventually, you merge onto the Massachusetts Turnpike, the center
+of the greater Boston highway system. From there, you use a series of increasingly smaller, narrower roads within the city (Charles River Bridge, St. Paul St,
+Thatcher St) before finally reaching the house. 
+
+A key observation is that you must initially travel far distances through long-distance road networks 
+(the interstate freeway system), before proceeding to finer and finer road networks (greater Boston highway system, central Brookline neighborhood connectors) before finally reaching
+your destination. This iterative series of road networks mimics the layered graph traversals performed by a HNSW query.
 <Info>
 **IVF + HNSW**

-In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions.
+In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a substructure which 
+further indexes the selected vectors inside each IVF partition. 
+This combines the scalability of IVF with the high recall of HNSW.
+LanceDB supports IVF-HNSW-based quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW_SQ`. 
 </Info>

-### Manual Indexing {#manual-indexing}

-If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process.
+## Using Vector Indexes
+Learn how to configure, build, and search LanceDB vector indexes, including build and search time parameters, asynchronous objects, and several examples.

-### Automatic Indexing {#automatic-indexing}
+### Configuration

- <Badge color="red">Enterprise-only</Badge> 
-Vector indexing is managed **automatically** in LanceDB Enterprise. As soon as data is updated, the system updates the index and optimizates it. *This is done asynchronously as a background process*.
-
-When you create a table in LanceDB Enterprise, LanceDB automatically:
-
- Infers the vector columns from the schema
- Create an optimized `IVF_PQ` index without manual configuration
- Automatically configure indexing parameters
-
-The default distance is `l2` (Euclidean).
-
-<Note>
-You can call `create_index()` with different parameters to create a new index -- this replaces any existing index.
-Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter.
-</Note>
-
-Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving.
-
-Rows appended after an index build remain outside that index until optimization refreshes it. Normal
-search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
-fallback and searches only indexed rows.
-
-## Choose the Right Index {#choose-the-right-index}
-
-Use this table as a quick starting point for choosing the right index type and quantization method for your use case:
-
-| If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors |
-| :--- | :--- | :--- | :--- |
-| Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead |
-| Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size |
-| Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size |
-| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) |
-
-<Warning>
-If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance.
-</Warning>
-
-Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration.
-For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`.
-
-### Index Tuning {#index-tuning}
-
-Start with these values, then tune for your workload:
-
- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`)
-  - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer)
-  - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger.
-  - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing.
- `IVF_RQ`
-  - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets.
- `IVF_PQ`
-  - `num_partitions`: start at `num_rows // 4096` (rounded to an integer).
-  - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes.
-  - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance.
-
-## Example: Construct an IVF Index {#example-construct-an-ivf-index}
-
-In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. 
-
-Make sure you have enough data in your table (at least a few thousand rows) for effective index training.
-
-### Index Configuration {#index-configuration}
-
-Sometimes you need to configure the index beyond default parameters:
-
- Index Types:
-    - `IVF_HNSW_FLAT`: highest recall, with no vector quantization
-    - `IVF_HNSW_SQ`: best recall/latency trade-off
-    - `IVF_RQ`: best compression for large, high-dimensional datasets
-    - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance
- `metrics`: default is `l2`, other available are `cosine` or `dot`
-    - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar)
- `num_partitions`: use index-specific starting points from the section above:
-    - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576`
-    - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096`
- `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition
-  count from a target number of rows per partition. If you set both `num_partitions` and
-  `target_partition_size`, `num_partitions` takes precedence.
- `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search.
-
-Let's take a look at a sample request for an IVF index:
-
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexConfigureIvf}
-    </CodeBlock>
-</CodeGroup>
-
-### 1\. Setup {#1-setup}
-
-Connect to LanceDB and open the table you want to index.
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexSetup}
-    </CodeBlock>
-</CodeGroup>
-
-### 2\. Construct an IVF Index {#2-construct-an-ivf-index}
-
-Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target.
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBuildIvf}
-    </CodeBlock>
-</CodeGroup>
-
-#### Indexing nested vector fields {#indexing-nested-vector-fields}
-
-If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`:
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexNestedField}
-    </CodeBlock>
-</CodeGroup>
-
-<Note>
-Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes.
-</Note>
-
-### Async API and Config Objects {#async-api-and-config-objects}
-
-With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings:
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexAsyncConfig}
-    </CodeBlock>
-</CodeGroup>
-
-Use these Python config classes for the index types shown on this page:
-
-| Index type | Python config class |
-| :--- | :--- |
-| `IVF_FLAT` | `IvfFlat` |
-| `IVF_PQ` | `IvfPq` |
-| `IVF_RQ` | `IvfRq` |
-| `IVF_SQ` | `IvfSq` |
-| `IVF_HNSW_FLAT` | `IvfHnswFlat` |
-| `IVF_HNSW_PQ` | `IvfHnswPq` |
-| `IVF_HNSW_SQ` | `IvfHnswSq` |
-
-### 3\. Query the IVF Index {#3-query-the-ivf-index}
-
-Search using a random 1,536-dimensional embedding.
-
-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexQueryIvf}
-    </CodeBlock>
-</CodeGroup>
-
-#### Search Configuration {#search-configuration}
-
-Core knobs available on a vector search call:
+#### Build-time Parameters

 | Parameter | Description |
 | :--- | :--- |
-| `limit` | Number of results to return (`k`). |
-| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. |
-| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. |
-| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. |
-| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. |
-| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. |
+| `distance_type` | Default is `l2`, others available are `cosine` and `dot`. 
+| `num_partitions` | The number of IVF partitions constructed (corresponds to the $k$ in $k$-means clustering). This takes precedence over `target_partition_size` if both are set. |
+| `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. `num_partitions` takes precedence over `target_partition_size` if both are set. If neither is set the number of partitions is automatically calculated based on the number of rows and the index type. |
+| `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. |
+| `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. |
+| `sample_rate` | Number of k-means training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. |
+
+#### Search-time Parameters
+
+| Parameter | Description |
+| :--- | :--- |
+| `limit` | Number of results to return (the `k` in `k-ANN`). |
+| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB defaults this to 20 which is a good default for many cases.  At very high scales you will likely need to experiment to find your preferred recall / latency balance. |
+| `minimum_nprobes` | Minimum number of partitions scanned. |
+| `maximum_nprobes` | Maximum number of partitions scanned. Only scans more than `minimum_nprobes` if an initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. |
+| `ef` | HNSW search-time exploration factor. Start around `1.5 * k` and increase up to `10 * k` for higher recall. |
+| `refine_factor` | Reads and reranks additional candidates in memory to recover recall lost to quantization. |
+| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. |
+| `bypass_vector_index()` | Ignore the ANN index entirely and perform an exact (flat) scan. Can be used to measure ANN `recall@k`, or to query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). |

 <Note>
-**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count.
+The `distance_type` can be specified at search time in addition to build time. If
+you do this, set it to the same value used to build the vector index. If the search
+metric does not match the index metric, LanceDB cannot use the index and falls
+back to a costly flat search. On LanceDB Cloud, the brute-force KNN safety limit
+can reject that search on a large table instead of executing it. LanceDB
+Enterprise doesn't have the limit.
+
+When the search metric is omitted, current SDKs use the index metric if an index
+is available. Some older SDK versions sent `l2` when it was omitted, so explicitly
+set the matching metric or upgrade the SDK when querying a non-`l2` index with an
+older client.
 </Note>

+**Recommended `nprobes` behavior by index type:**
+
+| Index type | Guidance |
+| :--- | :--- |
+| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the default `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. |
+| `IVF_RQ`, `IVF_PQ` | Keep the default `nprobes` and raise when recall is insufficient.  If using a selective filter you may need to increase `maximum_nprobes` if you do not received enough results. |
+
+
+<Note>
+**Filtered queries and adaptive `nprobes`.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` 
+partitions and uses a wider scan if too few rows are found. 
+Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count.
+</Note>
+
+Here is an example of a vector search exercising several of the above parameters.
+
 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {VectorIndexNprobes}
    </CodeBlock>
 </CodeGroup>

-Recommended `nprobes` behavior by index type:

-| Index type | Guidance |
-| :--- | :--- |
-| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. |
-| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. |
-| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. |
-
-#### Advanced Search Controls {#advanced-search-controls}
-
-These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints.
-
-| Method | Description |
-| :--- | :--- |
-| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. |
-| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). |
+LanceDB also supports advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints.

 **Thresholding with `distance_range`:**

@@ -242,9 +182,9 @@ These controls are useful for thresholded retrieval, recall measurement, and wor
    </CodeBlock>
 </CodeGroup>

-**Measuring recall with `bypass_vector_index`:**
+**Using `bypass_vector_index`:**

-Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload.
+Use `bypass_vector_index` to compute an exact **kNN** result. Note that exact queries may be prohibitively slow on production scales.

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -252,105 +192,102 @@ Compare ANN results against a flat-scan ground truth to compute recall@k. This i
    </CodeBlock>
 </CodeGroup>

-<Warning>
-Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries.
-</Warning>
-
 <Note title="Multivector distance constraint">
-Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/vector-search) for the full rules.
+Multivector indexing currently requires `distance_type="cosine"`. Use `bypass_vector_index()` for non-`cosine` queries on a multivector column. See [Multivector Search](/search/vector-search) for the full rules.
 </Note>

-## Example: Construct an HNSW Index {#example-construct-an-hnsw-index}
+#### Async API and Config Objects

-### Index Configuration {#index-configuration-2}
-
-There are four key parameters to set when constructing an HNSW index:
-
- `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization.
- `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`.
- `m`: The number of neighbors to select for each vector in the HNSW graph.
- `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph.
-
-### 1\. Construct an HNSW Index {#1-construct-an-hnsw-index}
-
-The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`.
+Create vector indexes asynchronously with `await table.create_index("vector", config=...)`. The `config` object 
+admits the same build-time parameters described above — pass an instance of a Python config class from the table in [Choosing the Right Index](#choosing-the-right-index):

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBuildHnsw}
+    {VectorIndexAsyncConfig}
    </CodeBlock>
 </CodeGroup>

-### 2\. Query the HNSW Index {#2-query-the-hnsw-index}
+### IVF Indexes
+
+This example creates and queries an `IVF_PQ` index for a table of vectors with respect to `cosine` similarity. 
+Specify `vector_column_name` if you have multiple vector columns or non-default names. 

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexQueryHnsw}
+    {VectorIndexBuildIvf + VectorIndexQueryIvf}
    </CodeBlock>
 </CodeGroup>

-## Example: Construct a Binary Vector Index {#example-construct-a-binary-vector-index}
+<Note>
+TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
+</Note>

-Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.
-
-### Index Configuration {#index-configuration-3}
-
- Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. 
- Index Type: `IVF_FLAT` is used for indexing binary vectors
- `metric`: the `hamming` distance is used for similarity search
- The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16.
-
-<Warning>
-**`IVF_FLAT` + `hamming` is the only supported path for binary vectors.**
-
- `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns.
- Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`.
-</Warning>
-
-### 1\. Create Table and Schema {#1-create-table-and-schema}
+For a vector field nested inside a struct, pass its full dotted path as `vector_column_name` (e.g. `image.embedding`) — the same path is used at query time and is what `list_indices()` reports under `columns`. See [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full path syntax.

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBinarySchema}
+    {VectorIndexNestedField}
    </CodeBlock>
 </CodeGroup>

-### 2\. Generate and Add Data {#2-generate-and-add-data}
+### IVF-HNSW Indexes
+
+Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes:
+
+| Parameter | Description |
+| :--- | :--- |
+| `m` | The number of neighbors to select for each vector in the HNSW graph. |
+| `ef_construction` | The number of candidates to evaluate during the construction of the HNSW graph. Start at `150`; increase for better recall, decrease for faster indexing. |
+
+Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above.
+The snippet below builds and queries an `IVF_HNSW_SQ` index. 

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBinaryAddData}
+    {VectorIndexBuildHnsw + VectorIndexQueryHnsw}
    </CodeBlock>
 </CodeGroup>

-### 3\. Construct the Binary Index {#3-construct-the-binary-index}
+### Managing Vector Indexes

-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBinaryBuildIndex}
-    </CodeBlock>
-</CodeGroup>
+ <Badge color="red">Enterprise-only</Badge> 
+In LanceDB Enterprise, vector indexes are managed **automatically**. The system asynchronously updates and optimizes indexes as a background process:
+- Automatically manages indexing parameters and storage
+- Infers vector columns from the schema

-### 4\. Vector Search {#4-vector-search}
+ <Badge color="gray">Open-Source</Badge> 
+ LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. 
+ See the above sections for guidance on manually tuning index parameters as data changes.

-<CodeGroup>
-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexBinarySearch}
-    </CodeBlock>
-</CodeGroup>
+<Note>
+`create_index()` returns immediately, but the vector index builds asynchronously. 
+To wait until all data is indexed, specify the `wait_timeout` parameter, or call `wait_for_index(...)` afterward — 
+it waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`.
+</Note>

-## Check Index Status {#check-index-status}
+<Note>
+Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal
+search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
+fallback and searches only indexed rows.
+</Note>

-Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**.
+<Info>
+**Operational checks**
+
+After appends or other writes, use `optimize()` to fold new rows into existing indexes.
+</Info>
+#### Check Index Status
+
+Vector index creation runs in the background and may take some time to complete. 
+While it is ongoing, you can check its status through the API or the **LanceDB Enterprise UI**.

 In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes.

-Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built.
-To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table.
+To check status programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built.

 Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`):

-| Field | What it tells you |
+| Parameter | Description |
 | :--- | :--- |
 | `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table |
 | `size_bytes` | Total size of the index files on disk |
@@ -359,9 +296,6 @@ Each entry returned by `list_indices()` also carries detailed per-index metadata
 | `index_uuid`, `type_url` | Internal identifiers for the index segment |
 | `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) |

-<Note>
-These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them.
-</Note>

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -369,9 +303,14 @@ These fields are populated for local and embedded tables. On LanceDB Enterprise
    </CodeBlock>
 </CodeGroup>

-## Custom Index Names {#custom-index-names}

-The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name.
+<Note>
+These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them.
+</Note>
+#### Custom Index Names
+
+The `{column}_idx` suffix is the default naming convetion. 
+Pass `name=...` to `create_index()` to override it. Once set, the custom name will be reflected in `list_indices()`, `index_stats(name)`, and `wait_for_index([name])`.

 <CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
@@ -379,74 +318,78 @@ The `{column}_idx` suffix is a default convention, not the only supported naming
    </CodeBlock>
 </CodeGroup>

-Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
+## Quantization {#quantization}

-Use quantization when:
+**Quantization** is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here;
+discussion of LanceDB vector indexes and quantized vector indexes can be found [here](/indexing/vector-index).

- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
- Index build time and query latency matter
+### Quantization Techniques
+LanceDB provides $3$ distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ).
+Recall that all quantizations perform **lossy** compression, in that they irreversibly lose some degree of precision in order to 
+compactly store an index.

-LanceDB currently exposes multiple quantized vector index types, including:
- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
+#### Product Quantization (PQ)
+To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has
+ $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on.

-Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
+Now, for each $i$, let $S_i$ represent the set containing chunk $i$ of each vector (entries $(i-1)d$ to $di - 1$).
+For each $S_i$ independently, a small set of **centroids** is computed corresponding to an approximate solution to the $k$-means clustering problem.

-Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
+For each original vector in the dataset, every chunk is associated to its nearest centroid; thus, the vector itself is associated with the concatenation of $m$ centroids.
+We then gather all our centroids (from all chunks) into a single lookup table, and for each vector, we store the concatenation of IDs of its corresponding centroids. 
+At query time, we need only to compare each chunk of the queried vector with our set of centroids.

-## RaBitQ quantization {#rabitq-quantization}
+<Frame caption="">
+  <img src="/static/assets/images/indexing/ivfpq_pq_desc.png" alt="IVF vector-space partitioning" />
+</Frame>

-RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
+In the above example, the original vector is split into chunks (subvectors), each of which is associated to a centroid. The stored quantization code for this vector 
+is the concatenation $2 || 1 || 4 || 3$. The original vector required $128$ dimensions $\times 32$-bit floats $ =4096$ bits total,
+which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of quantized storage.

-### How RaBitQ works {#how-rabitq-works}
+#### RaBitQ Quantization (RQ)

- Embeddings are grouped around centroids (as in other IVF indexes).
- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
- The sign pattern of that vector is stored as bits (1 bit per dimension).
- Two small corrective factors are stored:
-  1. The distance from the original vector to its centroid
-  2. The dot product between the normalized vector and its quantized version
+RaBitQ is an advanced quantization technique that outperforms PQ in several ways.
+It needs no codebook to train, estimates distances very quickly at query-time, 
+and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!** 
+In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few thousand bits, while maintaining good recall.

-Compared to `IVF_PQ`, RaBitQ:
- Avoids training expensive PQ codebooks
- Builds indexes faster and handles updates more easily
- Maintains or improves recall at high dimensionality under the same storage budget
+The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by 
+applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube, 
+where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497)

-For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
+##### Using RaBitQ
+Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`. 
+Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`.

-### Using RaBitQ {#using-rabitq}
-
-You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`. 
-
-<Note title="Dimension requirement">
-When using `IVF_RQ`, vector dimensions must be divisible by `8`.
-</Note>
-
-`num_bits` controls how many bits per dimension are used:
-
-1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
-It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
+`num_bits` determines how many bits are used to quantize each dimension.
+`1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute.

 <Warning title="Reading multi-bit indexes across versions">
-Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
+RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout, and cannot be read by some older LanceDB versions. 
 </Warning>

-## API Reference {#api-reference}
+See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation.

-The full list of parameters to the algorithm are listed below.
+#### Scalar Quantization (SQ)
+Scalar quantization quantizes each entry of a vector independently, by simply replacing it with the closest of a pre-defined set of values. 
+In practice, it often uses $8$ bits per dimension of a vector, and supports very fast encoding and decoding.

- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"  
-  The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
- `num_partitions`: Optional[int], defaults to None  
-  Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
- `num_bits`: int, defaults to 1  
-  Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
- `max_iterations`: int, defaults to 50  
-  Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
- `sample_rate`: int, defaults to 256  
-  Number of samples per partition during training. Higher values may improve accuracy but increase training time.
- `target_partition_size`: Optional[int], defaults to None  
-  Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
+For example, suppose we know all vector entires across our dataset lie in the range $[-128 \times 10^5, 127 \times 10^5]$. 
+In this case, we could quantize a given value $v$ as an $8$-bit representation of the integer $j$, where $j \times 10^5$ is the closest value to $v$
+among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$.
+
+### Quantization API Reference
+
+
+| Parameter | Description |
+| :--- | :--- |
+| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (default `1`, RaBitQ) — not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. |
+
+`max_iterations` and `sample_rate` also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they're documented as general [Build-time Parameters](/indexing/vector-index#build-time-parameters) instead. All three are passed as keyword arguments to `create_index`, alongside `index_type`:
+
+<CodeGroup>
+    <CodeBlock filename="Python" language="Python" icon="python">
+    {QuantizationCustomParams}
+    </CodeBlock>
+</CodeGroup>
@@ -1,94 +1,159 @@
 ---
 title: "LeRobotDataset"
 sidebarTitle: "LeRobotDataset"
-description: "Use Lance-backed LeRobotDataset loaders and LanceDB to inspect, filter, and train on robotics datasets from the Hugging Face Hub."
+description: "Train LeRobot policies straight from object storage, and curate, search and visualize robotics datasets as Lance tables."
+keywords: ["lerobot", "robotics", "droid", "training", "dataloader"]
 ---

 import {
  PyFrameworksLerobotFilterFrames,
-  PyFrameworksLerobotLancedbImageDataset,
-  PyFrameworksLerobotLancedbVideoDataset,
+  PyFrameworksLerobotLoad,
  PyFrameworksLerobotOpenLanceTables,
+  PyFrameworksLerobotTrainReady,
 } from '/snippets/integrations.mdx';

-[LeRobot](https://huggingface.co/docs/lerobot/index) is Hugging Face's open-source robotics stack for collecting data, training policies, running simulations, and sharing robotics datasets and models on the Hub.
+[LeRobot](https://huggingface.co/docs/lerobot/index) is Hugging Face's open-source robotics stack. Since [lerobot#4363](https://github.com/huggingface/lerobot/pull/4363) it reads Lance datasets natively: same `LeRobotDataset` class, same items, same training pipeline. The dataset's `"storage_format": "lance"` picks the reader. No subclasses, no wrapper library.

-[LeRobotDataset v3.0](https://huggingface.co/docs/lerobot/lerobot-dataset-v3) standardizes robot learning data across sensorimotor time series, actions, multi-camera video, and task metadata. Its v3 layout stores high-frequency tabular signals in Parquet, visual streams as MP4 shards, and metadata that reconstructs episode-level views from larger files.
+What you get:

-Lance pairs well with LeRobot when you need high-performance random access, lazy multimodal blob reads, and a single table interface for curation, search, and training data preparation. The `lerobot-lancedb` package ships Lance-backed `LeRobotDataset` subclasses, and LanceDB can open Lance-formatted LeRobot datasets on the Hub directly through `hf://` URIs.
+- **Train from object storage while keeping your GPUs fed.** Lance reads random frames straight from S3, GCS, the HF Hub or HF Storage Buckets, fetching only the bytes each batch needs, so every batch is a global shuffle: any frame, from anywhere in the dataset. In the table below (samples/s, 8 workers, batch 64), Lance and upstream local both draw every batch from the whole dataset. Upstream streaming does a reservoir shuffle, reading the dataset in order and drawing each batch from a pool of recently seen frames, so those two columns are not the same job:

-## Install {#install}
+| dataset | frames | cameras | Lance, S3 (global shuffle) | upstream, local (global shuffle) | same job | upstream, streaming (reservoir shuffle) | vs streaming |
+|---|---|---|---|---|---|---|---|
+| DROID | 27,630,375 | 3 | 2,258.5 | 346.7 | 6.5x | 898.6 | 1.7x |
+| berkeley_rpt | 392,578 | 1 | 1,740.6 | 1,209.9 | 1.4x | 274.5 | 6.3x |
+| toto | 325,699 | 1 | 1,097.0 | 936.3 | 1.2x | 121.3 | 9.0x |
+| roboturk | 187,507 | 1 | 1,276.8 | 1,082.3 | 1.2x | 128.2 | 10.0x |
+| aloha_mobile_cabinet | 127,500 | 3 | 815.7 | 294.2 | 2.8x | 77.7 | 10.5x |
+| koch_pick_place | 37,972 | 2 | 854.7 | 274.9 | 3.1x | 58.1 | 14.7x |
+
+The gap grows with how much video each sample carries: about 1.2x with one camera, 2.8x with three, 6.5x on DROID.
+
+- **Faster iteration, end to end.** 10,000 steps of SmolVLA on DROID, 8xH100, lerobot's default `num_workers=4`, same seed:
+
+| | Lance, S3, nothing downloaded | upstream, local NVMe after a 384 GB download |
+|---|---|---|
+| wall clock | **1 h 27 m** | 2 h 00 m |
+| steady rate | 495 samples/s | 361 samples/s |
+| of each step spent waiting on data | 1.7% | 37.4% |
+| final loss | 0.2380 | 0.2380 |
+
+Same model, same loss, 33 minutes sooner, and no 384 GB download first. The difference is time the GPUs spend waiting on data, so it grows with the run: at 100k steps it is over five hours, and every re-run and sweep point pays it again.
+
+- **The training table is also the index.** Parquet has no secondary indexes, so finding frames instead of scanning them usually means another system per question: a vector database for embeddings, a search service for instructions, a feature store for scores. With Lance those are columns and indexes on the same table the DataLoader reads. One query spans all of them, at a version the trainer can reopen.
+
+## Install

 ```bash
-pip install lancedb lance lerobot-lancedb
+pip install "lerobot[lancedb]"   # the reader, inside lerobot itself
+pip install lerobot-lancedb      # the converter
 ```

-## Use Lance-backed LeRobotDataset loaders {#use-lance-backed-lerobotdataset-loaders}
+## Convert a dataset

-`LeRobotLanceDataset` is useful when your Lance-backed dataset stores decoded image observations. It's a drop-in replacement for `LeRobotDataset`, so existing policy training code keeps working with the usual PyTorch dataset and dataloader patterns.
+Any LeRobot v3.0 dataset, from the Hub or a local directory:
+
+```bash
+lerobot-lance-convert --repo-id lerobot/pusht --out ./pusht-lance
+```
+
+**Migration costs about as much as a file copy.** Videos are stored as blobs without re-encoding, so items are bit-identical to the source: 392,578 frames convert in 10 seconds, DROID's 27.6M in 34 minutes. Nothing about your recording or training code changes. The output keeps `meta/` as is and adds three Lance tables (`frames`, `videos`, `meta`). `lerobot-lance-doctor` audits the result and often finds problems in the source: DROID carries 44% orphaned rows, and three of eight public datasets we converted have frame-count defects that only random access exposes.
+
+Host the result anywhere: an HF dataset repo (`hf upload`, tag it `v3.0`), an HF Storage Bucket (`hf buckets sync`), or any object store (`aws s3 sync`).
+
+## Load and train

 <CodeBlock filename="Python" language="Python" icon="python">
-  {PyFrameworksLerobotLancedbImageDataset}
+  {PyFrameworksLerobotLoad}
 </CodeBlock>

-For datasets that store camera observations as MP4 video segments, use `LeRobotLanceVideoDataset` instead.
+An explicit `root` works for any object store: `LeRobotDataset("lerobot/pusht", root="s3://my-bucket/pusht-lance")`. Remote roots download only `meta/`. Data is fetched per batch. Temporal windows, `DataLoader` and the rest of the training stack work unchanged:

 <CodeBlock filename="Python" language="Python" icon="python">
-  {PyFrameworksLerobotLancedbVideoDataset}
+  {PyFrameworksLerobotTrainReady}
 </CodeBlock>

-<Info>
-Use the image loader for Lance-backed repos that store image frames. Use the video loader for MP4-backed LeRobot datasets such as `lance-format/lerobot-pusht-lance`.
-</Info>
+Or from the CLI. Held-out evaluation on a remote dataset needs random access, so it only works on the Lance path:

-## Open LeRobot Lance tables with LanceDB {#open-lerobot-lance-tables-with-lancedb}
+```bash
+lerobot-train \
+  --dataset.repo_id=my-org/my-bucket/pusht-lance \
+  --dataset.repo_type=bucket \
+  --dataset.eval_split=0.02 \
+  --policy.type=act
+```

-Lance-formatted LeRobot datasets published by `lance-format` expose each `.lance` file under `data/` as a LanceDB table. The PushT dataset, for example, has `frames`, `episodes`, and `videos` tables.
+## Visualize TB-scale datasets in Foxglove
+
+LeRobot's dataset viewer can serve an episode to [Foxglove](https://foxglove.dev) over its WebSocket protocol ([lerobot#4542](https://github.com/huggingface/lerobot/pull/4542)), and with a Lance dataset it works wherever the data lives:
+
+```bash
+lerobot-dataset-viz --repo-id lance-format/pusht-lance --episode-index 3 --display-mode foxglove
+```
+
+Open the Foxglove app, connect to `ws://localhost:8765`, and scrub. Cameras, joint state and actions show up as topics you can lay out however you like:
+
+![Foxglove showing episode 7 of DROID served from S3: three camera panels and the joint state, scrubbed to 19.4 s](/static/assets/images/integrations/lerobot-foxglove-droid.png)
+
+Frames are fetched as you scrub, so looking at one episode of the 373 GB DROID from S3 moves a few MB, not the dataset. No MCAP export, no download, and nothing new to install on the dataset side: this is the same reader the trainer uses.
+
+## Query, curate, enrich
+
+The frames table is a LanceDB table like any other:

 <CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotOpenLanceTables}
 </CodeBlock>

-Opening the tables directly is handy for inspecting schemas, counting rows, sampling metadata, or building curation workflows before any data reaches the training loop.
-
-## Filter a frame window {#filter-a-frame-window}
-
-Most robotics workflows want a deterministic slice by `episode_index`, `frame_index`, or task metadata long before training begins. LanceDB filters those rows without touching the video blobs.
+Filter with SQL over any feature, using the indexes built at conversion:

 <CodeBlock filename="Python" language="Python" icon="python">
  {PyFrameworksLerobotFilterFrames}
 </CodeBlock>

-With the filtered set in hand, you can materialize a smaller local LanceDB database, add derived columns, attach embeddings, or build vector and scalar indexes for faster repeated access.
+From here it is standard LanceDB. The examples below are from our DROID walkthrough, all on the table the trainer reads.

-## Example Lance-formatted LeRobot datasets {#example-lance-formatted-lerobot-datasets}
+**Feature engineering costs nothing until you use it.** A new column is a [declaration](/feature-engineering/add-computed-column): `add_columns` registers the expression or [function](/management/function) and computes nothing, and a [refresh](/feature-engineering/refresh-column) materializes it in parallel when you want it. No copy of the dataset, and the videos are never touched:

-<CardGroup cols={2}>
-  <Card title="LeRobot PushT" href="/datasets/lerobot-pusht">
-    A Lance-formatted version of `lerobot/pusht` with frame, episode, and video tables.
-  </Card>
-  <Card title="LeRobot X-VLA Soft-Fold" href="/datasets/lerobot-xvla-soft-fold">
-    A multi-camera robotics dataset packaged as Lance tables for frame-level and episode-level access.
-  </Card>
-</CardGroup>
+```python
+@udf(data_type=pa.float32(), input_columns=["action_joint_velocity"])
+def jerk_score(v):                      # per-frame motion roughness
+    ...

-## More resources {#more-resources}
+tbl.add_columns({"jerk_score": jerk_score})   # nothing computed yet
+tbl.backfill("jerk_score", concurrency=8)     # 610,403 rows in 16.7 s
+```

-<CardGroup cols={2}>
-  <Card title="LeRobotDataset v3.0" href="https://huggingface.co/docs/lerobot/lerobot-dataset-v3">
-    Hugging Face's guide to the v3 dataset layout, streaming, transforms, and migration.
-  </Card>
-  <Card title="lerobot-lancedb" href="https://lancedb.github.io/lerobot-lancedb/">
-    API documentation for the Lance-backed LeRobotDataset implementations.
-  </Card>
-</CardGroup>
+Embeddings work the same way, reading frames straight from the video blobs in the same table:

-## When to use each interface {#when-to-use-each-interface}
+```python
+@udf(data_type=pa.list_(pa.float32(), 768), num_gpus=1,
+     input_columns=["episode_index", "frame_index"])
+class EmbedFromBlob:                    # SigLIP2 loads once per worker
+    def __call__(self, episode_index, frame_index):
+        blob = self.videos.take_blobs("video_bytes", indices=[vrow])[0]
+        frame = VideoDecoder(blob).get_frames_at([pos]).data[0]
+        ...
+```

-| Interface | Best for |
-|:---|:---|
-| `LeRobotDataset` | Standard LeRobot training loops and policy code |
-| `LeRobotLanceDataset` | Drop-in training on Lance-backed image datasets |
-| `LeRobotLanceVideoDataset` | Drop-in training on Lance-backed video datasets |
-| LanceDB | Interactive inspection, filtering, curation, search, indexing, and materializing subsets |
-| `lance.dataset(...)` | Lower-level schema, fragment, index, and blob access |
+**Search** is an index on those columns: a vector index over the embeddings built in 29.7 s across 27.6M rows, a full-text index over the instructions in 7.0 s, queries in 10 to 54 ms.
+
+**Curation** is one query across all of it. Semantic search with a predicate on the derived score, no join, no second system:
+
+```python
+tbl.search(vec("a gripper closing on an object"), vector_column_name="emb_siglip2") \
+   .where("jerk_score > 1.2875")        # the roughest 1% of frames
+   .limit(4)
+```
+
+**Mining** is the same query pointed at the rare cases. `tbl.search().where("jerk_score > 1.2875 AND success = false")` pulls the rough, failed frames out of 27.6M rows in milliseconds, ready to become a fine-tuning set or a labeling queue.
+
+**EDA** is a scan of the tabular columns: a pass over all 27.6M rows reads 2.52 GB in 190 s and decodes no video.
+
+Then drop what you found from training with `episodes=[...]` instead of rewriting files. The reader ignores columns it does not know, and the table is versioned, so curation and training share one copy of the data at a version you can reopen.
+
+<Info>
+Lance datasets are read-only in LeRobot: record and edit in the default format, then convert.
+</Info>
+
+Public examples to try everything above: [`lance-format/pusht-lance`](https://huggingface.co/datasets/lance-format/pusht-lance) and friends under the [`lance-format`](https://huggingface.co/lance-format) org.
@@ -0,0 +1,32 @@
+---
+title: "Views"
+sidebarTitle: "View"
+description: "A saved query, evaluated when read. The non-materialized counterpart to a materialized view."
+icon: eye
+keywords: ["view", "create view", "sql", "catalog"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+A view stores a **query**, not rows. Reading it runs the query against current
+data, so a view is never stale -- and never faster than the query it wraps.
+
+It is the counterpart to a
+[materialized view](/management/materialized-view), which stores rows and is
+brought up to date by a refresh. Both are [catalog objects](/basics/catalog)
+defined by a query; the difference is entirely when that query runs.
+
+| | View | Materialized view |
+| --- | --- | --- |
+| Stores | the query | the rows |
+| Freshness | always current | as of the last refresh |
+| Read cost | the full query, every read | a table scan |
+| Write cost | none | a refresh |
+
+Views are created in SQL, over the [Flight SQL endpoint](/search/sql) -- there
+is no SDK method, because the thing a view stores *is* a SQL statement. See
+[DDL](/sql/ddl#views).
@@ -491,7 +491,7 @@ permutation = Permutation.identity(characters)
 dataloader = torch.utils.data.DataLoader(permutation)
 ```

-See [Training](/training) for shuffling, multi-worker loading, and reading from
+See [Training](/training/data-loading) for shuffling, multi-worker loading, and reading from
 a remote table.

 ## Code {#code}
@@ -535,7 +535,7 @@ full-text and hybrid search, reranking, and server-side feature engineering.
 <Card
  title="Data loading and shuffles"
  icon="boxes-stacked"
-  href="/training/"
+  href="/training/data-loading"
 >
  Use LanceDB for projected, shuffled, random-access reads in training workflows.
 </Card>
@@ -0,0 +1,32 @@
+---
+title: "Full-text search with SQL"
+sidebarTitle: "Full-Text Search"
+description: "Running full-text search from SQL, with fuzzy, phrase, boolean and boost queries."
+icon: magnifying-glass
+keywords: ["fts", "sql", "full text search", "fuzzy", "phrase"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+Full-text search is available in SQL as the [`fts()`](/sql/functions#fts) table
+function, which returns matching rows plus a `_score` column:
+
+```sql
+SELECT id, text, _score
+FROM fts('my_docs', 'puppy')
+ORDER BY _score DESC
+LIMIT 5
+```
+
+The second argument is a bare query string, or a JSON query object for the
+fuzzy, phrase, boolean, boost and multi-match forms.
+
+For a keyword test that needs no score,
+[`contains_tokens`](/sql/scalar-functions#contains-tokens) is a predicate usable
+directly in `WHERE`, and is accelerated by the full-text index.
+
+The query-builder equivalent is [Full-Text Search](/search/full-text-search).
@@ -0,0 +1,29 @@
+---
+title: "Connecting a SQL client"
+sidebarTitle: "Client Connection"
+description: "Reaching the Arrow Flight SQL endpoint from Python or TypeScript."
+icon: plug
+keywords: ["flight sql", "client", "connect", "arrow"]
+---
+
+<Badge color="red">Enterprise</Badge>
+
+This page describes a LanceDB Enterprise capability. The embedded form
+of LanceDB has no server to provide it -- see
+[Architecture](/architecture) for what differs between the two.
+
+SQL is served over an
+[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
+endpoint, so any Flight SQL client reaches it and results arrive as Arrow
+record batches.
+
+From Python, the connection runs SQL directly without a separate client:
+
+```python
+reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
+table = reader.read_all()
+```
+
+The TypeScript SDK has no SQL method; use a Flight SQL client. See
+[SDKs and SQL](/basics/apis) for the full comparison, and the
+[SQL reference](/sql) for the dialect.
@@ -53,7 +53,7 @@ Raw data (e.g. text, images, audio, etc.) is converted into embeddings via an em
 Vector search has **no SQL form**. Run it here and use [SQL](/sql) over the result.
 </Note>

-## Supported distance metrics {#supported-distance-metrics}
+## Supported distance metrics

 Distance metrics determine how LanceDB compares vectors to find similar matches. Euclidean or `l2` is the default, and used for general-purpose similarity, `cosine` for unnormalized embeddings, `dot` for normalized embeddings (best performance), or `hamming` for binary vectors. 

@@ -65,10 +65,10 @@ The right metric improves both search accuracy and query performance. Currently,

 | Distance metric | Mathematical form | Notes |
 |---|---|---|
-| `l2` | $\|x-y\|_2=\sqrt{\sum_i (x_i-y_i)^2}$ | Measures the straight-line distance between two points in vector space. Calculated as the square root of the sum of squared differences between corresponding vector components. |
-| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Computed as 1 minus cosine similarity (the dot product normalized by both vector magnitudes), so vector length does not affect the score. Use for unnormalized vectors. |
-| `dot` | $x\cdot y=\sum_i x_i y_i$ | Calculates the sum of products of corresponding vector components. Provides raw similarity scores without normalization, sensitive to vector magnitudes. Use for normalized vectors for best performance. |
-| `hamming` | $\sum_i \mathbf{1}[x_i\neq y_i]$ | Counts the number of positions where corresponding bits differ between binary vectors. Only applicable to binary vectors stored as packed uint8 arrays. |
+| `l2` | $$\|x-y\|_2= \sqrt{\sum_i (x_i-y_i)^2}$$ | The $\ell_2$ norm. Measures Euclidean distance between two points in geometric space. |
+| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Insensitive to nonzero scaling of vectors. |
+| `dot` | $x\cdot y= \underset{i}{\sum} x_i y_i$ | The standard dot product of real vectors. Sensitive to nonzero scaling of vectors; use on normalized vectors. |
+| `hamming` | $\underset{i}{\sum} \,\: \mathbf{1}_{\, [x_i\neq y_i]}$ | Counts the number of bitwise-differing bits between binary vectors. |

 For indexed search, supported distance metrics vary by index type:

@@ -82,12 +82,20 @@ For indexed search, supported distance metrics vary by index type:
 | `IVF_HNSW_PQ` | `["l2", "cosine", "dot"]` |
 | `IVF_HNSW_SQ` | `["l2", "cosine", "dot"]` |

-### Configure Distance Metric {#configure-distance-metric}
+### Configure Distance Metric

-By default, `l2` will be used as metric type. You can specify the metric type as
-`cosine` or `dot` if required (`hamming` is supported for `IVF_FLAT` index only).
+Without a vector index, searches use `l2` by default. You can specify `cosine` or
+`dot` instead (`hamming` is supported for `IVF_FLAT` indexes only).

-**Note:** You can configure the distance metric during search only if there's no vector index. If a vector index exists, the distance metric will always be the one you specified when creating the index.
+When a vector index exists, the search metric must match the metric used to build
+the index. If you explicitly request a different metric, LanceDB cannot use that
+index and falls back to an exhaustive (brute-force) search. On LanceDB Cloud, the
+brute-force KNN safety limit can reject that search on a large table instead of
+executing it. LanceDB Enterprise doesn't have the limit.
+
+If you omit the search metric, current SDKs use the index metric when an index is
+available. Some older SDK versions sent `l2` when the metric was omitted; with a
+non-`l2` index, explicitly set the matching metric or upgrade the SDK.

 <CodeGroup>
 <CodeBlock filename="Python" language="python" icon="python">
@@ -103,11 +111,14 @@ By default, `l2` will be used as metric type. You can specify the metric type as
 </CodeBlock>
 </CodeGroup>

-Here you can see the same search but using `cosine` similarity instead of `l2` distance. The result focuses on vector direction rather than absolute distance, which works better for normalized embeddings.
+Here you can see the same search but using `cosine` similarity instead of `l2`
+distance. If the table has a vector index, make sure it was also built with
+`cosine`. The result focuses on vector direction rather than absolute distance,
+which works better for unnormalized embeddings.

 Set `.limit(...)` on vector searches you run in applications. You can page through results with `.offset(...)` and include LanceDB's internal row id with `.with_row_id()` / `.withRowId()` when you need a stable handle for follow-up operations.

-## Selecting the vector column {#selecting-the-vector-column}
+## Selecting the vector column

 If your table has exactly one vector column, you can omit the column name and LanceDB will pick it for you. This works for both top-level columns (such as `vector`) and vector fields nested inside a struct (such as `image.embedding`).

@@ -142,7 +153,7 @@ The same field-path syntax works when creating an index on a nested vector colum

 When several columns share a name across structs (for example, `image.embedding` and `text.embedding`), LanceDB still picks the one whose dimension matches your query vector. If two candidates have the same dimension, you must pass the column name explicitly.

-## Vector Search With ANN Index {#vector-search-with-ann-index}
+## Vector Search With ANN Index

 Instead of performing an exhaustive search on the entire database for each and every query, approximate nearest neighbour (ANN) algorithms use an index to narrow down the search space, which significantly reduces query latency. 

@@ -154,7 +165,7 @@ Use ANN search for large-scale applications where speed matters more than perfec
 When a vector index is used, `_distance` is not always the true distance between full vectors. On quantized ANN indexes, LanceDB may compute `_distance` from the compressed representation for speed. Use `refine_factor` when you want reranking on full vectors.
 </Warning>

-### Exact vs Approximate Distances {#exact-vs-approximate-distances}
+### Exact vs Approximate Distances

 When doing vector search, the meaning of "distance" depends on whether you are using an index and whether `refine_factor` is specified as part of your query.
 `nprobes` controls how many partitions are searched to find candidates, `approx_mode` controls the query-time speed/recall trade-off for RQ-quantized indexes, and `refine_factor` controls how many candidates are rescored on full vectors for better distance fidelity and reranking quality.
@@ -185,7 +196,7 @@ The table below summarizes the behavior of `_distance` in search results based o
 For deeper tuning guidance on indexing and performance estimation, see the [vector indexes](/indexing/vector-index/#search-configuration) page,
 For tuning `nprobes`, see below.

-### Tuning approximate mode {#tuning-approximate-mode}
+### Tuning `approx_mode`

 Use `approx_mode` when you want to adjust the speed/recall trade-off for approximate vector search at query time. This setting currently applies only to RQ-quantized indexes, such as `IVF_RQ`; other index types ignore it.

@@ -199,7 +210,7 @@ The supported values are:

 You can change `approx_mode` per query without rebuilding the index. For RQ indexes built with `num_bits=1`, `normal` uses the same one-bit scoring path as `fast`. If you also set `refine_factor`, LanceDB first uses `approx_mode` while finding candidates, then reranks the selected candidates on full vectors.

-### Tuning `nprobes` {#tuning-nprobes}
+### Tuning `nprobes`

 - `nprobes` controls how many partitions are searched at query time.
 - `nprobes` improves candidate recall, but does not by itself make `_distance` exact.
@@ -213,7 +224,7 @@ For filtered ANN searches, you can also set `minimum_nprobes` and `maximum_nprob
 with the minimum and can scan more partitions up to the maximum if the filter leaves too few
 candidates. Calling `nprobes(n)` fixes both values to `n`, which disables that adaptive behavior.

-### Vector Search with Prefiltering {#vector-search-with-prefiltering}
+### Vector Search with Prefiltering

 This is the default vector search setting. You can use prefiltering to boost query performance by reducing the search space before vector calculations begin. The system first applies your filter criteria to the dataset, then conducts vector search operations only on the remaining relevant subset.

@@ -237,7 +248,7 @@ The `.where("label > 2")` applies a filter before vector search, `.select(["text

 As a result, you'll see a result with just the data you want from the most similar vectors.

-### Vector Search with Postfiltering {#vector-search-with-postfiltering}
+### Vector Search with Postfiltering

 Use postfiltering to prioritize vector similarity by searching the full dataset first, then applying metadata filters to the top results. This approach ensures you get the most similar vectors before filtering, which can be crucial when similarity is more important than metadata constraints.

@@ -266,7 +277,7 @@ In the end, you receive a query result with the best matches that also meet your
 the filter condition after obtaining the nearest neighbors based on vector similarity.
 </Card>

-## Multivector Search {#multivector-search}
+## Multivector Search

 Use multivector search when your documents contain multiple embeddings and you need sophisticated matching between query and document vector pairs. The late interaction approach finds the most relevant combinations across all available embeddings and provides nuanced similarity scoring.

@@ -283,9 +294,9 @@ Here you can see how to take 2 query vectors and find the best matching pairs be

 **Read more:** [Multivector search](/search/vector-search)

-## Advanced Search Scenarios {#advanced-search-scenarios}
+## Advanced Search Scenarios

-### Search With Distance Range {#search-with-distance-range}
+### Search With Distance Range

 Use `distance_range` search when you need vectors within particular similarity bounds rather than just the closest neighbors. The system filters results to only include vectors that fall within your specified distance thresholds from the query.

@@ -309,7 +320,7 @@ The `distance_range()` method filters results by similarity thresholds - the fir

  Each approach returns Arrow tables with vectors that fall within your specified distance thresholds.

-### Search With Binary Vectors {#search-with-binary-vectors}
+### Search With Binary Vectors

 Use binary vector search for scenarios involving binary embeddings, such as those produced by hashing algorithms. The system stores these efficiently as packed uint8 arrays and uses Hamming distance calculations to determine vector similarity.

@@ -337,9 +348,9 @@ The schema defines a 32-byte vector field (256 bits ÷ 8), `np.random.randint(0,

 The search produces an Arrow table with binary vectors ranked by how many bits differ from the query.

-## Scaling Vector Search {#scaling-vector-search}
+## Scaling Vector Search

-### Batch Search {#batch-search}
+### Batch Search

 Use batch search to handle multiple query vectors simultaneously. This gives you significant efficiency gains over individual queries. LanceDB processes all vectors in parallel and organizes results with a `query_index` field that maps each result set back to its originating query.

@@ -369,7 +380,7 @@ to explicitly associate each result set with its corresponding query in
 the input batch. 
 </Tip>

-### Search With Asynchronous Indexing {#search-with-asynchronous-indexing}
+### Search With Asynchronous Indexing

 To optimize for speed over completeness, enable the `fast_search` flag in your query to skip searching unindexed data.

@@ -398,9 +409,9 @@ The `fast_search=True` parameter tells LanceDB to only search indexed vectors, s

 You'll obtain a query result with the top `5` matches from indexed vectors, but might miss data that was just added.

-## Brute Force Search {#brute-force-search}
+## Brute Force Search

-### Search With No Index {#search-with-no-index}
+### Search With No Index

 The simplest way to perform vector search is to perform a brute force search, without an index, where the distance between the query vector and all the vectors in the database are computed, with the top-k closest vectors returned. 

@@ -428,7 +439,7 @@ This carries out a brute force search through every vector in the table to find

 As you can imagine, the brute force approach is not scalable for datasets larger than a few hundred thousand vectors, as the latency of the search grows linearly with the size of the dataset. This is where approximate nearest neighbour (ANN) algorithms come in.

-### Bypass the Vector Index {#bypass-the-vector-index}
+### Bypass the Vector Index

 Use `bypass_vector_index` to get exact, ground-truth results by performing exhaustive searches across all vectors. Instead of relying on approximate methods, the system directly compares your query against every vector in the table, ensuring 100% recall at the cost of increased query time.

@@ -457,13 +468,15 @@ This approach is particularly useful when:
 - Calculating recall metrics to tune index parameters
 - Ensuring exact results for critical applications

+## Multi-vector search {#multivector}
+
 LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item.

 This capability is particularly valuable when working with late-interaction models like ColBERT and ColPaLi, which generate multiple embeddings per document.

 In this tutorial, you'll create a table with multiple vector embeddings per document and learn how to perform multivector search. For more end-to-end examples, see the [VectorDB recipes repository](https://github.com/lancedb/vectordb-recipes/tree/main/examples).

-## Multivector Support {#multivector-support}
+### Multivector Support

 Each item in your dataset can have a column containing multiple vectors, which LanceDB can efficiently index and search. When performing a search, you can query with either a single vector embedding or multiple vector embeddings.

@@ -473,7 +486,7 @@ Currently, only the `cosine` metric is supported for multivector search. The vec

 Each query vector must match the inner vector dimension in the multivector column. This applies to both single-vector queries and multi-vector query matrices.

-## Computing Similarity {#computing-similarity}
+### Computing Similarity

 MaxSim (Maximum Similarity) is a key concept in late-interaction models that:

@@ -495,9 +508,9 @@ $$

 $Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ represents the document embeddings.

-## Using Multivector Search {#using-multivector-search}
+### Using Multivector Search

-### 1\. Setup {#1-setup}
+#### 1. Setup

 Connect to LanceDB and import the required libraries.

@@ -515,7 +528,7 @@ db = lancedb.connect(
 ```
 </CodeGroup>

-### 2\. Define Schema {#2-define-schema}
+#### 2. Define Schema

 Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with:

@@ -538,7 +551,7 @@ schema = pa.schema(
 ```
 </CodeGroup>

-### 3\. Generate Multivectors {#3-generate-multivectors}
+#### 3. Generate Multivectors

 Generate sample data where each document contains multiple vector embeddings, which can represent different aspects or views of the same document.

@@ -556,7 +569,7 @@ data = [
 ```
 </CodeGroup>

-### 4\. Create a Table {#4-create-a-table}
+#### 4. Create a Table

 Create a table with the defined schema and sample data, which will store multiple vectors per document for similarity search.

@@ -566,7 +579,7 @@ tbl = db.create_table("multivector_example", data=data, schema=schema)
 ```
 </CodeGroup>

-### 5\. Build an Index {#5-build-an-index}
+#### 5. Build an Index

 Only cosine similarity is supported for multivector search operations.
 For faster search, build the standard `IVF_PQ` index over your vectors:
@@ -583,10 +596,10 @@ tbl.create_index(metric="cosine", vector_column_name="vector")
 A brute-force scan over a multivector column has to compare every query vector to every document vector in every row, so the cost grows with both the row count and the number of vectors per row. 
 In LanceDB OSS, the query will just run, so a large unindexed multivector table can stall a process for a long time before returning results.

-On LanceDB Enterprise, the brute-force KNN safety check applies a stricter row threshold to multivector columns — roughly 10× lower than for single-vector columns. So an unindexed multivector table will start being rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
+On LanceDB Cloud, the brute-force KNN safety check applies a stricter row threshold to multivector columns — roughly 10× lower than for single-vector columns. So an unindexed multivector table can be rejected with a "vector search would use brute-force KNN" error well before a comparable single-vector table would. LanceDB Enterprise doesn't have the limit. Build the index before you start hitting it from production traffic, even if the dataset is small enough that you'd skip indexing for a single-vector workload.
 </Info>

-### 6\. Query a Single Vector {#6-query-a-single-vector}
+#### 6. Query a Single Vector

 When searching with a single query vector, it will be compared against all vectors in each document, and the similarity scores will be aggregated to find the most relevant documents.

@@ -597,7 +610,7 @@ results_single = tbl.search(query).limit(5).to_pandas()
 ```
 </CodeGroup>

-### 7\. Query Multiple Vectors {#7-query-multiple-vectors}
+#### 7. Query Multiple Vectors

 With multiple query vectors, LanceDB calculates similarity using late interaction, a late-interaction technique that computes relevance by finding the best-matching pairs between query and document vectors. This approach provides more nuanced matching while maintaining fast retrieval speeds.

@@ -611,7 +624,7 @@ results_multi = tbl.search(query_multi).limit(5).to_pandas()

 Visit the [Hugging Face embedding integration](/integrations/embedding/huggingface/) page for information on embedding models.

-## Simple Example: ColBERT Embeddings {#simple-example-colbert-embeddings}
+### Simple Example: ColBERT Embeddings

 [ColBERT](https://arxiv.org/abs/2004.12832) is the most well-known late-interaction retrieval model that
 represents each document and query as multiple token embeddings and scores matches by taking the best
@@ -630,14 +643,14 @@ import pyarrow as pa
 import lancedb
 from pylate import models

-# 1) Load a late-interaction model via PyLate
-# PyLate docs show ColBERT() + encode(..., is_query=...) :contentReference[oaicite:2]{index=2}
+## 1) Load a late-interaction model via PyLate
+## PyLate docs show ColBERT() + encode(..., is_query=...) :contentReference[oaicite:2]{index=2}
 model = models.ColBERT(model_name_or_path="lightonai/GTE-ModernColBERT-v1")

-# You can discover dim from one embedding (avoid guessing)
+## You can discover dim from one embedding (avoid guessing)
 dim = model.encode(["hello"], is_query=True)[0].shape[1]

-# 2) Create a LanceDB table with a multivector column
+## 2) Create a LanceDB table with a multivector column
 db = lancedb.connect("./pylate_lancedb")
 schema = pa.schema([
    pa.field("doc_id", pa.string()),
@@ -652,7 +665,7 @@ docs = [
    {"doc_id": "3", "text": "This is a noodle bar in Osaka, Japan."},
 ]

-# 3) Encode documents with PyLate (token vectors per doc)
+## 3) Encode documents with PyLate (token vectors per doc)
 doc_texts = [d["text"] for d in docs]
 doc_embs = model.encode(doc_texts, is_query=False)  # list/array of (T, dim) per doc :contentReference[oaicite:4]{index=4}

@@ -663,10 +676,10 @@ for d, emb in zip(docs, doc_embs):

 tbl = db.create_table("docs", data=rows, schema=schema, mode="overwrite")

-# 4) Build an index + query using a query matrix.
-# Multivector brute-force scales with rows × vectors-per-row, so build the index
-# at much smaller dataset sizes than you would for single-vector search — and
-# always before exposing the table to remote traffic.
+## 4) Build an index + query using a query matrix.
+## Multivector brute-force scales with rows × vectors-per-row, so build the index
+## at much smaller dataset sizes than you would for single-vector search — and
+## always before exposing the table to remote traffic.
 tbl.create_index(vector_column_name="mv", metric="cosine")

 query = "Tell me about ramen in Japan"
@@ -680,7 +693,7 @@ print(out[["doc_id", "text"]])
 Late-interaction model implementations evolve rapidly, so it's a good idea to check the latest popular models
 when trying multivector search.

-## Advanced Example: XTR Embeddings {#advanced-example-xtr-embeddings}
+### Advanced Example: XTR Embeddings

 [ConteXtualized Token Retriever (XTR)](https://arxiv.org/abs/2304.01982) is a late-interaction retrieval model that represents text as token-level vectors instead of a single embedding.
 This lets search score token-to-token matches (MaxSim), which can improve fine-grained relevance.
@@ -12,6 +12,8 @@ export const PyGpuIndexCuda = "table.create_index(\n    num_partitions=256,\n

 export const PyGpuIndexMps = "table.create_index(\n    num_partitions=256,\n    num_sub_vectors=96,\n    accelerator=\"mps\",\n)\n";

+export const PyQuantizationCustomParams = "table.create_index(\n    index_type=\"IVF_RQ\",\n    num_bits=2,\n    max_iterations=100,\n    sample_rate=512,\n)\n";
+
 export const PyReindexingIncremental = "table = db.open_table(\"reindexing_incremental\")\ntable.add([{\"vector\": [3.1, 4.1], \"text\": \"Frodo was a happy puppy\"}])\ntable.optimize()\n";

 export const PyScalarIndexBuild = "tbl = db.open_table(\"scalar_index_build\")\ntbl.create_scalar_index(\"book_id\")\ntbl.create_scalar_index(\"publisher\", index_type=\"BITMAP\")\n";
@@ -38,24 +40,14 @@ export const PyScalarIndexWait = "index_name = \"label_idx\"\ntable.wait_for_ind

 export const PyVectorIndexAsyncConfig = "import lancedb\nimport numpy as np\nfrom lancedb.index import IvfPq\n\nasync def main():\n    data = [\n        {\"id\": i, \"vector\": np.random.random(8).astype(np.float32).tolist()}\n        for i in range(512)\n    ]\n\n    db = await lancedb.connect_async(\"ex_lancedb\")\n    table = await db.create_table(\n        \"vector_index_async\", data=data, mode=\"overwrite\"\n    )\n\n    await table.create_index(\n        \"vector\",\n        config=IvfPq(\n            distance_type=\"cosine\",\n            num_partitions=16,\n            num_sub_vectors=4,\n        ),\n    )\n    return await table.list_indices()\n";

-export const PyVectorIndexBinaryAddData = "table.add(data)\n";
-
-export const PyVectorIndexBinaryBuildIndex = "table.create_index(\n    metric=\"hamming\",\n    vector_column_name=\"vector\",\n    index_type=\"IVF_FLAT\",\n)\n";
-
-export const PyVectorIndexBinarySchema = "table = tmp_db.create_table(table_name, schema=schema, mode=\"overwrite\")\n";
-
-export const PyVectorIndexBinarySearch = "query = np.random.randint(0, 2, size=ndim)\nquery = np.packbits(query)\ndf = table.search(query).metric(\"hamming\").limit(10).to_pandas()\ndf.vector = df.vector.apply(np.unpackbits)\n";
-
 export const PyVectorIndexBuildHnsw = "table.create_index(index_type=\"IVF_HNSW_SQ\")\n";

-export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n    metric=\"cosine\",\n    vector_column_name=\"keywords_embeddings\",\n)\n";
+export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n    metric=\"cosine\",\n    vector_column_name=\"keywords_embeddings\",\n    index_type=\"IVF_PQ\",\n)\n";

 export const PyVectorIndexBypassRecall = "query = np.random.random(128)\nk = 10\n\n# Ground truth: flat (exhaustive) scan, ignoring the ANN index.\ntruth = set(table.search(query).bypass_vector_index().limit(k).to_pandas()[\"id\"])\n\n# ANN results with the current nprobes setting.\nann = set(table.search(query).nprobes(20).limit(k).to_pandas()[\"id\"])\n\nrecall_at_k = len(truth & ann) / k\n";

 export const PyVectorIndexCheckStatus = "index_name = \"keywords_embeddings_idx\"\ntable.wait_for_index([index_name])\nprint(table.index_stats(index_name))\n";

-export const PyVectorIndexConfigureIvf = "table.create_index(metric=\"l2\", num_partitions=16, num_sub_vectors=4)\n";
-
 export const PyVectorIndexCustomName = "# Override the default `{column}_idx` convention by passing `name=...`.\ntable.create_index(\n    metric=\"cosine\",\n    vector_column_name=\"keywords_embeddings\",\n    name=\"my_custom_index\",\n)\ntable.wait_for_index([\"my_custom_index\"])\nprint(table.index_stats(\"my_custom_index\"))\n";

 export const PyVectorIndexDistanceRange = "# Only return results whose distance falls within [0.0, 0.5).\n# Useful for near-duplicate detection or thresholded similarity search.\n(\n    table.search(np.random.random(128))\n    .distance_range(lower_bound=0.0, upper_bound=0.5)\n    .limit(10)\n    .to_pandas()\n)\n";
@@ -68,5 +60,3 @@ export const PyVectorIndexQueryHnsw = "tbl = table\ntbl.search(np.random.random(

 export const PyVectorIndexQueryIvf = "tbl = table\ntbl.search(np.random.random((1536))).limit(2).nprobes(20).refine_factor(\n    10\n).to_pandas()\n";

-export const PyVectorIndexSetup = "table_name = \"vector-index-tbl\"\ntable = db.open_table(table_name)\n";
-
@@ -74,11 +74,11 @@ export const PyFrameworksLangchainVectorStoreConfig = "db_url = \"db://lang_test

 export const PyFrameworksLerobotFilterFrames = "frame_rows = (\n    frames.search()\n    .where(\"episode_index = 0 AND frame_index < 10\", prefilter=True)\n    .select([\"episode_index\", \"frame_index\", \"timestamp\", \"action\"])\n    .limit(10)\n    .to_list()\n)\n\nfor row in frame_rows:\n    print(row[\"episode_index\"], row[\"frame_index\"], row[\"timestamp\"])\n";

-export const PyFrameworksLerobotLancedbImageDataset = "from lerobot_lancedb import LeRobotLanceDataset\n\ndataset = LeRobotLanceDataset(\n    repo_id=\"your-org/your-lerobot-lance-images\",\n    delta_timestamps={\n        \"observation.images.front\": [-0.2, -0.1, 0.0],\n    },\n    return_uint8=True,\n)\n\nsample = dataset[0]\nprint(sample[\"observation.state\"].shape)\nprint(sample[\"action\"].shape)\n";
+export const PyFrameworksLerobotLoad = "from lerobot.datasets import LeRobotDataset\n\n# Hub dataset repo: storage_format in meta/info.json routes to the Lance reader\ndataset = LeRobotDataset(\"lance-format/pusht-lance\")\n\n# HF Storage Bucket, read in place, nothing downloaded\ndataset = LeRobotDataset(\"lance-format/lerobot-tests/pusht-lance\", repo_type=\"bucket\")\n\nsample = dataset[100]\nprint(sample[\"observation.state\"].shape)\nprint(sample[\"observation.image\"].shape)\n";

-export const PyFrameworksLerobotLancedbVideoDataset = "from lerobot_lancedb import LeRobotLanceVideoDataset\n\nvideo_dataset = LeRobotLanceVideoDataset(\n    repo_id=\"lance-format/lerobot-pusht-lance\",\n    delta_timestamps={\n        \"observation.images.image\": [-0.2, -0.1, 0.0],\n    },\n    return_uint8=True,\n)\n\nvideo_sample = video_dataset[0]\nprint(video_sample[\"observation.images.image\"].shape)\n";
+export const PyFrameworksLerobotOpenLanceTables = "import lancedb\n\ndb = lancedb.connect(\"hf://datasets/lance-format/pusht-lance\")\nframes = db.open_table(\"frames\")\nvideos = db.open_table(\"videos\")\n\nprint(frames.count_rows(), videos.count_rows())\nprint(frames.schema)\n";

-export const PyFrameworksLerobotOpenLanceTables = "import lancedb\n\ndb = lancedb.connect(\"hf://datasets/lance-format/lerobot-pusht-lance/data\")\nframes = db.open_table(\"frames\")\nepisodes = db.open_table(\"episodes\")\nvideos = db.open_table(\"videos\")\n\nprint(len(frames), len(episodes), len(videos))\nprint(frames.schema)\n";
+export const PyFrameworksLerobotTrainReady = "import torch\n\ndataset = LeRobotDataset(\n    \"lance-format/pusht-lance\",\n    delta_timestamps={\"observation.image\": [-0.2, -0.1, 0.0]},\n    return_uint8=True,\n)\nloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)\nbatch = next(iter(loader))\nprint(batch[\"observation.image\"].shape)  # [32, 3, C, H, W]\n";

 export const PyFrameworksLlamaindexAddReranker = "from lancedb.rerankers import ColbertReranker\n\nreranker = ColbertReranker()\nvector_store._add_reranker(reranker)\n";

@@ -97,5 +97,5 @@ would draw from are not all in the fragments yet.
 ## Sampling outside SQL {#sdk}

 For a training loop rather than a query, the SDKs read a table in a shuffled
-order without going through SQL at all -- see [Training](/training), where a
+order without going through SQL at all -- see [Training](/training/data-loading), where a
 permutation drives the data loader.
@@ -3,6 +3,20 @@
  --banner-right: #e55a2b;
 }

+/* Mintlify's KaTeX stylesheet currently serves the AMS font from a broken URL. */
+@font-face {
+  font-family: "LanceDB KaTeX AMS";
+  src: url("https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/fonts/KaTeX_AMS-Regular.woff2") format("woff2");
+  font-display: swap;
+  font-style: normal;
+  font-weight: 400;
+}
+
+.katex .mathbb,
+.katex .textbb {
+  font-family: "LanceDB KaTeX AMS", "KaTeX_AMS", serif !important;
+}
+
 /* Mintlify banner gradient */
 #banner,
 :where(.banner, [data-banner], [class*="Banner_banner"]) {
@@ -0,0 +1,475 @@
+---
+title: "Loading Data for Model Training"
+sidebarTitle: "Data Loading & Shuffles"
+description: "Stream, shuffle, transform, and resume model training data with LanceDB."
+icon: boxes-stacked
+---
+
+LanceDB provides an excellent data backend for training machine learning models. For basic use cases, a `Table` can be fed directly into a data loader. For a more complete solution, a `StreamingDataset` adds prefetching, elastic determinism, resumability, multithreaded transformations, and more.
+
+## Basic Data Loading
+
+Most model training frameworks feed data into a model by sequencing through successive batches, a process 
+often referred to as **data loading**. The simplest way to perform data loading is to iterate through a LanceDB table in
+a loop.
+
+<CodeGroup>
+```py Python icon=Python 
+import lancedb
+
+db = lancedb.connect("file://some/db/path")
+table = db.open_table("some_table")
+
+for batch in table:
+    print(batch.to_pydict())
+```
+</CodeGroup>
+
+## Advanced Data Loading
+Advanced data loading scenarios can be handled by the `StreamingDataset`. This object allows you to apply preprocessing before training, pipelined reads, and more. Use this when the training data does not fit in memory, or when you need deterministic global batches across cluster sizes.
+
+<CodeGroup>
+```py Python icon=Python
+import lancedb
+import torch
+from lancedb.streaming import StreamingDataset
+
+db = lancedb.connect("file://some/db/path")
+table = db.open_table("some_table")
+
+dataset = StreamingDataset(table, shuffle_seed=42)
+dataloader = torch.utils.data.DataLoader(
+    dataset,
+    batch_size=128,
+    num_workers=0,
+)
+
+for batch in dataloader:
+    train_step(batch)
+```
+</CodeGroup>
+
+The `StreamingDataset` wraps a LanceDB `Table` and yields plain Python dictionaries by default. Whether iterating directly or through a PyTorch `DataLoader`, only one iterator can be simultaneously active on a `StreamingDataset` instance. 
+
+<Note>
+`StreamingDataset` is built on the `permutation` API and works with both LanceDB OSS and LanceDB Enterprise. The underlying table data can live on local disk or object storage.
+</Note>
+
+To understand the wide range of scenarios handled by `StreamingDataset`, consider a model trained with stochastic gradient descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs
+across multiple servers. After each batch is processed, the GPUs exchange weights and the next batch is loaded. We
+fix the following terminology:
+
+- **World size** - The number of GPUs being loaded simultaneously. For example, if we have $2$ servers with $4$
+  GPUs each, the world size is $8$.
+- **Rank** - The identifier of an individual GPU. It is an integer in the range `[0, world_size)`.
+  Each rank is allocated a disjoint portion of the source data.
+- **Batch size** - The number of rows processed by one GPU in a single training step. For example, $128$.
+- **Global batch size** - The total number of rows processed across all GPUs in a training step (sometimes called a "minibatch" in machine learning literature). For
+  example, if we have $8$ GPUs with batch size $128$, the global batch size is $1024$.
+
+
+Other concepts, such as read batch size and `num_workers`, are introduced in the relevant sections below.
+
+### Filtering
+
+The `StreamingDataset` object allows you to train models on only the subset of rows that satisfy a user-defined filter, specified using the `filter` parameter. This can be useful for removing large rows, badly-sourced data, and a wide range of other applications. 
+You can also use the `columns` parameter to load only a subset of columns.
+
+<CodeGroup>
+```py Python icon=Python
+ds = StreamingDataset(
+    table,
+    shuffle_seed=42,
+    columns=["image", "label"],       # skip all other columns
+    filter="category = 'train'",      # only training rows
+)
+```
+</CodeGroup>
+
+### Shuffling Rows
+
+In model training, model quality can be hurt by unintentionally learning patterns in the ordering of the source data. Shuffling the dataset before training alleviates this problem. However, fetching rows with scattered sparse reads across object storage can be dramatically slower than fetching contiguous batches. `StreamingDataset` offers two different implementations to read data in shuffled order, toggled by the presence of the `block_size` parameter. 
+
+<Note>
+`StreamingDataset` shuffle seeds combine `shuffle_seed` with `epoch`; each epoch produces a different permutation while runs
+with the same inputs remain reproducible. Set `shuffle_seed = None` to generate a random seed upon pipeline initialization.
+</Note>
+
+#### One-Phase Shuffle
+
+Setting `block_size = None` corresponds to the naive $1$-phase shuffled read: the rows of the source dataset are shuffled according to a pseudorandom permutation, then fed in batches to the accelerator. This gives the best possible randomness, but can exhibit low throughput. 
+
+
+<CodeGroup> 
+```py Python icon=Python 
+# 1-phase shuffled read: rows shuffled individually.
+for epoch in range(num_epochs): 
+    ds = StreamingDataset( 
+        table, 
+        shuffle_seed=42, 
+        epoch=epoch, # changes the permutation each epoch 
+    ) 
+    for sample in ds: 
+        train_step(sample)
+
+#Evaluation: deterministic sequential order, no shuffle.
+eval_ds = StreamingDataset(table, shuffle=False)
+for sample in eval_ds:
+    eval_step(sample)
+```
+
+</CodeGroup>
+
+
+#### Two-Phase Shuffle
+
+Setting `block_size` to an integer corresponds to selecting $2$-phase shuffled read. In this implementation, the source dataset is first partitioned into fixed-size `blocks`, which are then assigned randomly to splits. Within each split, rows are then further shuffled, with each row moving at most `max_shuffle_distance` distance away from its original position. Reading blocks allows storage I/O to constitute mostly large, contiguous reads rather than single row fetches. This yields much higher throughput while slightly compromising randomness.
+ 
+<CodeGroup>
+```py Python icon=Python
+# 2-phase shuffled read: rows are shuffled block-by-block instead of
+# individually, trading a bit of randomness for much cheaper I/O.
+for epoch in range(num_epochs):
+    ds = StreamingDataset(
+        table,
+        shuffle_seed=42,
+        epoch=epoch,               
+        block_size=1024,           # enables 2-phase
+        window_blocks=4,           # blocks resident per split at once
+        max_shuffle_distance=256,  # how far a row may move from its
+                                    # original position
+    )
+    for sample in ds:
+        train_step(sample)
+
+# Evaluation: deterministic sequential order, no shuffle.
+eval_ds = StreamingDataset(table, shuffle=False, block_size=1024)
+for sample in eval_ds:
+    eval_step(sample)
+```
+</CodeGroup>
+
+### Parallelism & Elasticity
+
+When loading data, `StreamingDataset` partitions the the source dataset among disjoint parallel threads called **splits**. Each rank is allocated a contiguous group of splits, and each DataLoader worker associated to a rank gets a contiguous subgroup of its rank's splits.
+
+If using `shuffle = false`, data is allocated to splits naively in ordered contiguous blocks of equal size. If `shuffle = true`, 
+in one-phase shuffling, the rows of the source dataset are distributed randomly among splits such that each split receives the same number of rows. In two-phase shuffling, the dataset is first partitioned into blocks, which are then distributed randomly among splits such that each split receives the same number of blocks.
+
+The number of splits defaults to `world_size` if `num_splits` is omitted during initialization. This is not recommended, as a later run using a different world size will then construct a different layout and structure of global batches.
+
+To get **elastic determinism**, choose a fixed `num_splits` that is compatible with every topology you plan to use.
+For example, a run with `num_splits=8` and $4$ GPUs assigns $2$ splits to each rank. The ranks pull from their splits in
+round-robin order, producing the same global batches as a run with $8$ GPUs and the same `num_splits`, `shuffle_seed`,
+and `epoch`.
+
+The following constraints apply:
+
+- `num_splits` must be divisible by `world_size`.
+- With DataLoader worker processes, `num_splits` must be divisible by `world_size * num_workers`.
+- For the same samples to form each global training step across topologies, `global_batch_size` must be a multiple of
+  `num_splits`.
+- The filtered row count must be at least `num_splits`. If it is not evenly divisible by `num_splits`, up to
+  `num_splits - 1` surplus rows are dropped so that every split has the same length.
+
+For example, to switch between $8$ and $6$ GPUs, use a common multiple such as $24$ for `num_splits`. Highly composite values can support many layouts; for example, `num_splits = 48` is compatible with $10$ distinct possible GPU counts. Remember to include `num_workers` when checking divisibility.
+
+<CodeGroup>
+```py Python icon=Python
+import torch.distributed as dist
+
+dist.init_process_group("nccl")
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+
+# num_splits=48 is divisible by many integers,
+# so this dataset works unchanged as you add or remove GPUs.
+ds = StreamingDataset(
+    table,
+    num_splits=48,
+    shuffle_seed=42,
+    epoch=current_epoch,
+    rank=rank,
+    world_size=world_size,
+)
+
+for sample in ds:
+    train_step(sample)
+```
+</CodeGroup>
+
+### Fetching
+
+Traditional datasets have often been designed to yield a single sample at a time upon iteration. However, in modern use cases,  accessing data on object storage one row at a time introduces too much per-call overhead. Thus, `StreamingDataset` fetches data in read batches, whose size is controlled by the `read_batch_size` parameter (defaults = `64`).
+
+In addition to batching requests, the prefetching mechanism reads ahead in the background. While one batch is being processed, `StreamingDataset` reads several subsequent batches. The `prefetch_batches` parameter specifies the number of batches simultaneously kept in flight per split (default = `4`). A larger value provides more buffering against jittery workloads, but also increases memory use and I/O concurrency.
+
+<CodeGroup>
+```py Python icon=Python
+ds = StreamingDataset(
+    table,
+    shuffle_seed=42,
+    read_batch_size=256,   # rows fetched per LanceDB call per split
+    prefetch_batches=8,    # batches to keep in flight per split
+)
+```
+</CodeGroup>
+
+### Transforming
+
+Many model training workloads require performing a transformation to the data after it is loaded from object storage and before it is passed to the GPU. For example, we may need to decode images, tokenize text, or normalize vectors. A transformation function can be provided using the `transform` parameter. 
+
+A transformation function must receive a PyArrow `RecordBatch` and return an iterable. For example, the default transform function returns a list of Python dictionaries. Furthermore, the input expected by PyTorch must match the collation function used in the PyTorch data loader. PyTorch's default collation function accepts a Python dictionary and several other types, and is thus compatible with the default `StreamingDataset` transform function. If your custom transform function returns something different (for example, Arrow record batches), you may need to provide a compatible PyTorch collation function.
+
+
+<CodeGroup>
+```py Python icon=Python
+import pyarrow as pa
+
+def normalize(batch: pa.RecordBatch) -> list[dict]:
+    # This pure-Python loop holds the GIL and is shown for illustration only.
+    # In practice, prefer a library like torchvision or numpy that releases the
+    # GIL so the ThreadPoolExecutor can run transforms in parallel.
+    rows = batch.to_pylist()
+    for row in rows:
+        row["image"] = [v / 255.0 for v in row["image"]]
+    return rows
+
+ds = StreamingDataset(table, shuffle_seed=42, transform=normalize)
+```
+</CodeGroup>
+
+`StreamingDataset` manages expensive transforms through a
+`ThreadPoolExecutor` whose worker count equals the number of available CPUs. Furthermore, transformations are applied to batches (instead of individual samples) to amortize per-batch overhead. 
+
+##### DataLoader Workers
+
+The thread-based transformation model that `StreamingDataset` uses by default is only effective when the transform
+function releases the GIL. This is true for many Python scientific libraries, including NumPy, PyArrow, and
+TorchVision, but not for pure-Python transforms. PyTorch can launch multiple DataLoader worker processes per rank,
+and `StreamingDataset` uses `get_worker_info()` to give each worker a non-overlapping group of splits.
+
+Multiprocessing adds pickling and transfer overhead and increases memory use. Start with `num_workers=0`, which keeps
+loading in the rank's main process, and add workers only after confirming an unavoidable GIL bottleneck. If you use
+workers, `num_splits` must be divisible by `world_size * num_workers`. Use the `forkserver` or `spawn` multiprocessing
+context because LanceDB uses internal threads.
+
+```py Python icon=Python
+dataloader = torch.utils.data.DataLoader(
+    dataset,
+    batch_size=batch_size,
+    num_workers=2,
+    multiprocessing_context="forkserver",
+    persistent_workers=True,
+)
+```
+
+By default, `StreamingDataset` serializes enough table state to reopen the table in each worker. For custom connection
+setup, pass a picklable `connection_factory` callable that accepts a table name and returns an open table. This avoids
+serializing connection credentials into worker state.
+
+### Observability
+
+Optimizing data loader performance can be tricky, as runtime bottlenecks are often difficult to locate across various stages of the I/O and CPU pipelines. To help distinguish bottlenecks, `StreamingDataset` exposes several pipeline counters: `raw_queue_depth` shows the number of loaded rows waiting to be transformed,
+and `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and
+`consumed_rows` show how far the current iterator has progressed.
+
+If the `prefetch_queue_depth` is consistently zero but the `raw_queue_depth` is not, then you likely have a CPU
+transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation.
+This can often be done by batching the compute work. If both `prefetch_queue_depth` and `raw_queue_depth` are
+consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped
+shuffling could help.
+
+<CodeGroup>
+```py Python icon=Python
+import threading, time
+
+ds = StreamingDataset(table, shuffle_seed=42)
+
+def log_pipeline_health():
+    while True:
+        print(
+            f"unscanned={ds.unscanned_rows} "
+            f"raw={ds.raw_queue_depth} "
+            f"cooked={ds.prefetch_queue_depth} "
+            f"consumed={ds.consumed_rows}"
+        )
+        time.sleep(1.0)
+
+monitor = threading.Thread(target=log_pipeline_health, daemon=True)
+monitor.start()
+
+for sample in ds:
+    train_step(sample)
+
+print(
+    f"bytes loaded: {ds.bytes_loaded} "
+    f"fetch time: {ds.fetch_time:.2f}s "
+    f"transform time: {ds.transform_time:.2f}s"
+)
+```
+</CodeGroup>
+
+`bytes_loaded` measures raw Arrow buffer bytes before transformation. `bytes_loaded`, `fetch_time`, and `transform_time`
+are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times
+can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters
+reflect the latest iterator snapshot.
+
+### Checkpointing 
+`StreamingDataset` provides checkpointing in case your model training fails partway through a run. Use `StreamingDataset.state_dict()` to capture the current number of samples consumed from each split in a Python dictionary, and `load_state_dict()` to restore the last saved position. 
+
+A stored `StreamingDataset` state reflects only round-robin cycles that have been completed across all splits; samples completed partway through a cycle at the time of saving can be replayed. To ensure efficient saving, make sure `global_batch_size` is a multiple of `num_splits`, as in the example below. 
+
+<CodeGroup>
+```py Python icon=Python
+import torch
+
+global_batch_size = 48  # one sample per split in each global step
+batch_size = global_batch_size // world_size
+
+dataset = StreamingDataset(
+    table,
+    num_splits=48,
+    shuffle_seed=42,
+    epoch=current_epoch,
+    rank=rank,
+    world_size=world_size,
+)
+dataloader = torch.utils.data.DataLoader(
+    dataset,
+    batch_size=batch_size,
+    num_workers=0,
+)
+
+for step, batch in enumerate(dataloader):
+    train_step(batch)
+
+    if (step + 1) % checkpoint_interval == 0:
+        torch.save(
+            {"model": model.state_dict(), "dataset": dataset.state_dict()},
+            f"checkpoint_{step + 1}.pt",
+        )
+
+# --- resuming after a crash ---
+checkpoint = torch.load("checkpoint_100.pt")
+model.load_state_dict(checkpoint["model"])
+
+dataset_state = checkpoint["dataset"]
+dataset = StreamingDataset(
+    table,
+    num_splits=dataset_state["num_splits"],
+    shuffle_seed=dataset_state["shuffle_seed"],
+    epoch=dataset_state["epoch"],
+    rank=rank,
+    world_size=world_size,  # may differ from the run that saved the checkpoint
+)
+dataset.load_state_dict(dataset_state)
+dataloader = torch.utils.data.DataLoader(
+    dataset,
+    batch_size=global_batch_size // world_size,
+    num_workers=0,
+)
+
+for batch in dataloader:
+    train_step(batch)
+```
+</CodeGroup>
+
+`load_state_dict()` rejects a checkpoint whose `num_splits` or `shuffle_seed` differs from the new dataset. For an
+exact resume, also use the same table snapshot, `epoch`, `shuffle`, `filter`, and other data-selection settings. The
+`world_size` and number of DataLoader workers may change as long as the split and batch-size divisibility constraints
+still hold.
+
+##### Checkpointing with Multiple DataLoader Workers
+
+Python's built-in `torch.utils.data.DataLoader` produces a safe checkpoint when `num_workers = 0`. With `num_workers > 0`, calling `state_dict()` often raises `RuntimeError`. In this case, use `StreamingDataLoader` for multi-worker prefetch and exact resumability. It carries a state snapshot
+alongside every internal batch and commits it to the parent `StreamingDataset` only when that batch is returned to the
+trainer. The trainer receives the same collated batch it would receive from a standard `DataLoader`.
+
+<CodeGroup>
+```py Python icon=Python
+from lancedb.streaming import StreamingDataset, StreamingDataLoader
+
+dataset = StreamingDataset(
+    table,
+    num_splits=48,
+    shuffle_seed=42,
+    epoch=current_epoch,
+    rank=rank,
+    world_size=world_size,
+)
+dataloader = StreamingDataLoader(
+    dataset,
+    batch_size=batch_size,
+    num_workers=4,
+)
+
+for step, batch in enumerate(dataloader):
+    train_step(batch)
+
+    if (step + 1) % checkpoint_interval == 0:
+        torch.save(
+            {"model": model.state_dict(), "dataset": dataset.state_dict()},
+            f"checkpoint_{step + 1}.pt",
+        )
+```
+</CodeGroup>
+
+`StreamingDataLoader` accepts the same arguments as `torch.utils.data.DataLoader`, with a few restrictions:
+
+- `dataset` must be a `StreamingDataset` (subclasses that override `__iter__` are not supported).
+- `in_order = True` is required so that consumer-committed checkpoints stay deterministic.
+- `persistent_workers = True` is not supported, because prefetched worker copies cannot be restored from parent-committed state.
+- `drop_last = True` is not supported, because incomplete tails discarded by worker replicas cannot be checkpointed topology-independently.
+
+With more than one worker, call `state_dict()` at a complete logical step boundary where every split assigned to the
+rank has the same consumed-sample count. Calling it mid-step raises `RuntimeError` asking you to consume more batches
+first.
+
+##### Resuming Across Different Topologies
+
+When training across ranks, each rank owns its own subset of splits and only its own splits have exact progress. To
+resume on a different `world_size`, collect the `state_dict()` from every rank of the previous run and merge them with
+`StreamingDataset.merge_state_dicts` before calling `load_state_dict()` on the new run. The merge is topology-agnostic:
+pass the full list of per-rank states in, and hand the identical merged dict to every rank of the resumed job,
+regardless of whether the topology grew, shrank, or stayed the same.
+
+<CodeGroup>
+```py Python icon=Python
+# On each rank of the previous run:
+state = dataset.state_dict()
+# ...gather `state` from every rank into a single list `states` (for example
+# via torch.distributed all-gather or by saving one file per rank).
+
+merged = StreamingDataset.merge_state_dicts(states)
+
+# On each rank of the resumed run (world_size may differ):
+dataset = StreamingDataset(
+    table,
+    num_splits=merged["num_splits"],
+    shuffle_seed=merged["shuffle_seed"],
+    epoch=merged["epoch"],
+    rank=rank,
+    world_size=new_world_size,
+)
+dataset.load_state_dict(merged)
+```
+</CodeGroup>
+
+## Permutations
+
+In specific use cases, you may need the flexibility to shuffle, split, and select data without using the full `StreamingDataset` object. In these cases, use `Permutation`, the lower-level functionality on which `StreamingDataset`
+is built. A `Permutation` defines a customizable row ordering, and supports map-style access through `__getitem__()`
+and batched access through `__getitems__()`.
+
+##### Version Pinning
+
+A `Permutation` is pinned to the version of its base table at the time it was constructed, and every read (including future reads
+from a `StreamingDataset` that wraps it, and every DataLoader worker after a `fork`) resolves against that pinned
+version. This makes iteration deterministic across compactions and worker forks: the permutation only addresses rows
+that existed when it was built, and rows appended to the base table afterwards are not visible through the existing
+permutation.
+
+To include newly appended rows, build a new `Permutation` (and any `StreamingDataset` that depends on it)
+against the updated table. Permutations written before this behavior shipped continue to
+read as they did before.
@@ -1,512 +0,0 @@
---
-title: "Loading Data for Model Training"
-sidebarTitle: "Data Loading & Shuffles"
-description: "Stream, shuffle, transform, and resume model training data from LanceDB."
-icon: boxes-stacked
---
-
-LanceDB makes an excellent data backend for training machine learning models. A `Table` can be used directly as input
-to a data loader, but this is typically limited. A `Permutation` gives you control over which rows are accessed and in
-what order. For a more complete solution, LanceDB also provides a streaming data loader through `StreamingDataset`.
-This PyTorch `IterableDataset` adapts the lower-level `Permutation` API and adds prefetching, elastic determinism,
-resumability, and multithreaded transformations.
-
-## Basic data loading {#basic-data-loading}
-
-Most model training frameworks iterate through data in batches and feed this data into the model. This process is
-often referred to as **data loading**. The simplest way to load data into a model is to iterate a LanceDB table in
-a loop and feed the data into the model.
-
-<CodeGroup>
-```py Python icon=Python 
-import lancedb
-
-db = lancedb.connect("file://some/db/path")
-table = db.open_table("some_table")
-
-for batch in table:
-    print(batch.to_pydict())
-```
-</CodeGroup>
-
-In practice, this is too simplistic for effective training. We may not want to load all the data, or we may want
-to load the data in a different order, or we may need to apply some sort of processing to the data before training.
-To achieve this, we can use the `StreamingDataset`.
-
-<CodeGroup>
-```py Python icon=Python
-import lancedb
-import torch
-from lancedb.streaming import StreamingDataset
-
-db = lancedb.connect("file://some/db/path")
-table = db.open_table("some_table")
-
-dataset = StreamingDataset(table, shuffle_seed=42)
-dataloader = torch.utils.data.DataLoader(
-    dataset,
-    batch_size=128,
-    num_workers=0,
-)
-
-for batch in dataloader:
-    train_step(batch)
-```
-</CodeGroup>
-
-`StreamingDataset` yields plain Python dictionaries by default. PyTorch's default collation function combines those
-samples into batches. You can also iterate the dataset directly when you do not need collation.
-
-<Note>
-`StreamingDataset` is built on the permutation API and works with both local LanceDB tables when using OSS, and
-remote tables accessed through LanceDB Enterprise. The underlying table data can live on local disk or object storage.
-</Note>
-
-Use the streaming data loader when the training data does not fit in memory, when you need deterministic global batches
-across cluster sizes, or when you want filtering and prefetching to happen before PyTorch requests individual samples.
-Use `Permutation` directly when you need map-style random access instead. Only one iterator can be active on a
-`StreamingDataset` instance at a time; create a separate instance for each concurrent consumer.
-
-## Advanced data loading {#advanced-data-loading}
-
-The `StreamingDataset` wraps a LanceDB `Table` and, by default, adds prefetching and conversion from Arrow to Python.
-It can also handle more advanced scenarios. To explain these, consider a model trained with stochastic gradient
-descent (SGD) and distributed data parallelism (DDP). In this example, we need to load batches onto multiple GPUs
-across multiple servers. After each batch is processed, the GPUs exchange weights and the next batch is loaded. We
-will use the following PyTorch terms:
-
- **World size** - The number of GPUs that we are loading. For example, if we have 2 servers and each server has 4
-  GPUs, the world size is 8.
- **Rank** - The identifier of the process loading data for a GPU. It is an integer in the range `[0, world_size)`.
-  Each rank gets its own portion of the data.
- **Global batch size** - The number of rows processed across all GPUs in each step of the SGD algorithm. For
-  example, if we have 8 GPUs and a global batch size of 1024, we load 128 rows onto each GPU for each step.
- **Batch size** - The number of rows processed by one GPU in each step. In the preceding example, the batch size
-  is 128.
-
-Other concepts, such as read batch size and `num_workers`, are introduced in the relevant sections below.
-
-### Prefetching {#prefetching}
-
-PyTorch datasets were originally built around in-memory structures like a Pandas DataFrame. When they are iterated,
-they yield a single sample at a time. This makes sense for a simple in-memory structure, but accessing data on object
-storage one row at a time introduces too much per-call overhead. To avoid this, `StreamingDataset` fetches and
-transforms data in batches. The `read_batch_size` parameter controls how many rows are fetched from each split per
-storage request and defaults to `64`.
-
-In addition to batching requests, the prefetching mechanism reads ahead in the background. While one batch is being
-transformed and processed by the GPU, `StreamingDataset` reads subsequent batches. The `prefetch_batches` parameter
-controls how many batches are kept in flight per split and defaults to `4`. A larger value can provide more buffering
-against jittery workloads, but it also increases memory use and I/O concurrency.
-
-<CodeGroup>
-```py Python icon=Python
-ds = StreamingDataset(
-    table,
-    shuffle_seed=42,
-    read_batch_size=256,   # rows fetched per LanceDB call per split
-    prefetch_batches=8,    # batches to keep in flight per split
-)
-```
-</CodeGroup>
-
-### Transformation {#transformation}
-
-Many model training workloads require a transformation step between loading the data and training the model. For
-example, we may need to decode images, tokenize text, or normalize data. A transformation function can be provided
-using the `transform` parameter. Transformations can be expensive, so `StreamingDataset` applies them with a
-`ThreadPoolExecutor` whose worker count equals the number of available CPUs.
-
-Transformations are applied to batches, not individual samples, to amortize per-batch overhead. A transformation
-function receives a PyArrow `RecordBatch` and must return an iterable with exactly one output sample for every input
-row. The sample format should match what your data loader expects. For example, PyTorch's default collation function
-accepts several sample types, with a Python dictionary being one of the most common. When no transform is provided,
-the default transform converts the Arrow record batch into Python dictionaries without further processing.
-
-<CodeGroup>
-```py Python icon=Python
-import pyarrow as pa
-
-def normalize(batch: pa.RecordBatch) -> list[dict]:
-    # This pure-Python loop holds the GIL and is shown for illustration only.
-    # In practice, prefer a library like torchvision or numpy that releases the
-    # GIL so the ThreadPoolExecutor can run transforms in parallel.
-    rows = batch.to_pylist()
-    for row in rows:
-        row["image"] = [v / 255.0 for v in row["image"]]
-    return rows
-
-ds = StreamingDataset(table, shuffle_seed=42, transform=normalize)
-```
-</CodeGroup>
-
-#### DataLoader workers {#dataloader-workers}
-
-The thread-based transformation model that `StreamingDataset` uses by default is only effective when the transform
-function releases the GIL. This is true for many Python scientific libraries, including NumPy, PyArrow, and
-TorchVision, but not for pure-Python transforms. PyTorch can launch multiple DataLoader worker processes per rank,
-and `StreamingDataset` uses `get_worker_info()` to give each worker a non-overlapping group of splits.
-
-Multiprocessing adds pickling and transfer overhead and increases memory use. Start with `num_workers=0`, which keeps
-loading in the rank's main process, and add workers only after confirming an unavoidable GIL bottleneck. If you use
-workers, `num_splits` must be divisible by `world_size * num_workers`. Use the `forkserver` or `spawn` multiprocessing
-context because LanceDB uses internal threads.
-
-```py Python icon=Python
-dataloader = torch.utils.data.DataLoader(
-    dataset,
-    batch_size=batch_size,
-    num_workers=2,
-    multiprocessing_context="forkserver",
-    persistent_workers=True,
-)
-```
-
-By default, `StreamingDataset` serializes enough table state to reopen the table in each worker. For custom connection
-setup, pass a picklable `connection_factory` callable that accepts a table name and returns an open table. This avoids
-serializing connection credentials into worker state.
-
-### Observability and performance {#observability-and-performance}
-
-Optimizing data loader performance is tricky because it can be difficult to locate the bottleneck. What is often
-blamed on I/O can be a CPU bottleneck in the transform stage, or vice versa. To help distinguish them,
-`StreamingDataset` exposes pipeline counters. `raw_queue_depth` is the number of loaded rows waiting to be transformed,
-while `prefetch_queue_depth` is the number of transformed rows ready to be consumed. `unscanned_rows` and
-`consumed_rows` show how far the current iterator has progressed.
-
-If the `prefetch_queue_depth` is consistently zero but the `raw_queue_depth` is not then you have a CPU
-transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation.
-This can often be done by batching the compute work. If both `prefetch_queue_depth` and `raw_queue_depth` are
-consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped
-shuffling could help.
-
-<CodeGroup>
-```py Python icon=Python
-import threading, time
-
-ds = StreamingDataset(table, shuffle_seed=42)
-
-def log_pipeline_health():
-    while True:
-        print(
-            f"unscanned={ds.unscanned_rows} "
-            f"raw={ds.raw_queue_depth} "
-            f"cooked={ds.prefetch_queue_depth} "
-            f"consumed={ds.consumed_rows}"
-        )
-        time.sleep(1.0)
-
-monitor = threading.Thread(target=log_pipeline_health, daemon=True)
-monitor.start()
-
-for sample in ds:
-    train_step(sample)
-
-print(
-    f"bytes loaded: {ds.bytes_loaded} "
-    f"fetch time: {ds.fetch_time:.2f}s "
-    f"transform time: {ds.transform_time:.2f}s"
-)
-```
-</CodeGroup>
-
-`bytes_loaded` measures raw Arrow buffer bytes before transformation. `bytes_loaded`, `fetch_time`, and `transform_time`
-are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times
-can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters
-reflect the latest iterator snapshot.
-
-### Filtering data {#filtering-data}
-
-By default, the streaming data loader includes all rows and columns. LanceDB is a columnar database that also supports
-efficient random access. Reducing the number of columns you load has a direct impact on I/O performance. Reducing the
-number of rows can also help, especially with a selective filter, large values, local data, or the LanceDB Enterprise
-cache.
-
-Use the `columns` parameter to specify which columns to load and the `filter` parameter to specify which rows to load.
-The filter is evaluated once when the permutation is constructed, before the rows are divided into splits. Filtered-out
-rows are not loaded from storage during iteration, and split sizes reflect the filtered row count.
-
-<CodeGroup>
-```py Python icon=Python
-ds = StreamingDataset(
-    table,
-    shuffle_seed=42,
-    columns=["image", "label"],       # skip all other columns
-    filter="category = 'train'",      # only training rows
-)
-```
-</CodeGroup>
-
-### Shuffling rows {#shuffling-rows}
-
-By default, `StreamingDataset` sets `shuffle=True` and randomly assigns rows to splits. This helps prevent the model
-from learning artifacts from storage order. Set `shuffle=False` to divide rows into splits sequentially, which is
-useful for deterministic evaluation and debugging.
-
-The effective shuffle seed combines `shuffle_seed` with `epoch`, so each epoch has a different ordering while runs
-with the same inputs remain reproducible. Keep `epoch=0` if you want the same ordering across iterations. The default
-`shuffle_seed` is `0`; set it to `None` to generate a random seed when the dataset is constructed.
-
-<CodeGroup>
-```py Python icon=Python
-# Training loop: each epoch gets a different shuffled ordering
-for epoch in range(num_epochs):
-    ds = StreamingDataset(
-        table,
-        shuffle_seed=42,
-        epoch=epoch,       # changes the permutation each epoch
-    )
-    for sample in ds:
-        train_step(sample)
-
-# Evaluation: deterministic sequential order, no shuffle
-eval_ds = StreamingDataset(table, shuffle=False)
-for sample in eval_ds:
-    eval_step(sample)
-```
-</CodeGroup>
-
-<Note>
-Shuffling can have significant impacts on I/O performance, especially if you are loading data from cloud storage.
-In many cases the GPU pipeline is slow enough that this penalty will not be noticeable.  However,
-you can use the `shuffle_clump_size` parameter to shuffle the data in clumps (small contiguous batches that get
-shuffled together).  This will give some penalty to the randomness of the shuffle, but will significantly improve
-I/O performance.
-</Note>
-
-<CodeGroup>
-```py Python icon=Python
-# Clumped shuffle: groups of 16 contiguous rows are shuffled together,
-# preserving read locality while still randomising the global ordering.
-ds = StreamingDataset(
-    table,
-    shuffle_seed=42,
-    shuffle_clump_size=16,
-)
-```
-</CodeGroup>
-
-### Data splits and elasticity {#data-splits-and-elasticity}
-
-`StreamingDataset` partitions the permutation into a fixed number of equal-sized groups called splits. Each rank gets
-a contiguous group of splits, and each DataLoader worker gets a contiguous subgroup of its rank's splits. Samples are
-then yielded by round-robining over the assigned splits.
-
-If `num_splits` is omitted, it defaults to `world_size`. This works when `num_workers=0`, but it ties the split layout
-to the current number of ranks. A run resumed with a different world size would then construct a different layout and
-would not see the same global batches.
-
-To get **elastic determinism**, choose a fixed `num_splits` that is compatible with every topology you plan to use.
-For example, a run with `num_splits=8` and 4 GPUs assigns 2 splits to each rank. The ranks pull from their splits in
-round-robin order, producing the same global batches as a run with 8 GPUs and the same `num_splits`, `shuffle_seed`,
-and `epoch`.
-
-The following constraints apply:
-
- `num_splits` must be divisible by `world_size`.
- With DataLoader worker processes, `num_splits` must be divisible by `world_size * num_workers`.
- For the same samples to form each global training step across topologies, `global_batch_size` must be a multiple of
-  `num_splits`.
- The filtered row count must be at least `num_splits`. If it is not evenly divisible by `num_splits`, up to
-  `num_splits - 1` surplus rows are dropped so that every split has the same length.
-
-For example, to switch between 8 and 6 GPUs, use a `num_splits` value divisible by both, such as 24. Highly composite
-values can support several layouts: 48 supports 1, 2, 3, 4, 6, 8, 12, 16, 24, and 48 GPUs, while 60 supports 1, 2, 3,
-4, 5, 6, 10, 12, 15, 20, 30, and 60 GPUs. Remember to include `num_workers` when checking divisibility.
-
-<CodeGroup>
-```py Python icon=Python
-import torch.distributed as dist
-
-dist.init_process_group("nccl")
-rank = dist.get_rank()
-world_size = dist.get_world_size()
-
-# num_splits=48 is divisible by 1, 2, 3, 4, 6, 8, 12, 16, 24, 48
-# so this dataset works unchanged as you scale up or down GPUs.
-ds = StreamingDataset(
-    table,
-    num_splits=48,
-    shuffle_seed=42,
-    epoch=current_epoch,
-    rank=rank,
-    world_size=world_size,
-)
-
-for sample in ds:
-    train_step(sample)
-```
-</CodeGroup>
-
-### Checkpointing and resumability {#checkpointing-and-resumability}
-
-Model training is expensive, and failures can occur partway through a run. A model checkpoint is not enough for an
-exact resume: the streaming data loader must also continue from the same position. `StreamingDataset.state_dict()`
-captures the number of samples consumed from every split in a plain Python dictionary, and `load_state_dict()` restores
-that position.
-
-Save this state at a global-step boundary where every split has contributed equally. The easiest way to guarantee this
-is to make `global_batch_size` a multiple of `num_splits`, as in the example below. If you checkpoint partway through a
-round-robin cycle, the state reflects the preceding complete cycle and those partial-cycle samples can be replayed.
-
-<CodeGroup>
-```py Python icon=Python
-import torch
-
-global_batch_size = 48  # one sample per split in each global step
-batch_size = global_batch_size // world_size
-
-dataset = StreamingDataset(
-    table,
-    num_splits=48,
-    shuffle_seed=42,
-    epoch=current_epoch,
-    rank=rank,
-    world_size=world_size,
-)
-dataloader = torch.utils.data.DataLoader(
-    dataset,
-    batch_size=batch_size,
-    num_workers=0,
-)
-
-for step, batch in enumerate(dataloader):
-    train_step(batch)
-
-    if (step + 1) % checkpoint_interval == 0:
-        torch.save(
-            {"model": model.state_dict(), "dataset": dataset.state_dict()},
-            f"checkpoint_{step + 1}.pt",
-        )
-
-# --- resuming after a crash ---
-checkpoint = torch.load("checkpoint_100.pt")
-model.load_state_dict(checkpoint["model"])
-
-dataset_state = checkpoint["dataset"]
-dataset = StreamingDataset(
-    table,
-    num_splits=dataset_state["num_splits"],
-    shuffle_seed=dataset_state["shuffle_seed"],
-    epoch=dataset_state["epoch"],
-    rank=rank,
-    world_size=world_size,  # may differ from the run that saved the checkpoint
-)
-dataset.load_state_dict(dataset_state)
-dataloader = torch.utils.data.DataLoader(
-    dataset,
-    batch_size=global_batch_size // world_size,
-    num_workers=0,
-)
-
-for batch in dataloader:
-    train_step(batch)
-```
-</CodeGroup>
-
-`load_state_dict()` rejects a checkpoint whose `num_splits` or `shuffle_seed` differs from the new dataset. For an
-exact resume, also use the same table snapshot, `epoch`, `shuffle`, `filter`, and other data-selection settings. The
-`world_size` and number of DataLoader workers may change as long as the split and batch-size divisibility constraints
-still hold.
-
-#### Checkpointing with multiple DataLoader workers {#checkpointing-with-multiple-dataloader-workers}
-
-The plain `torch.utils.data.DataLoader` only produces a safe checkpoint when `num_workers=0`. With `num_workers > 0`,
-PyTorch runs `StreamingDataset.__iter__` in separate worker processes and prefetches batches ahead of the trainer, so
-the counters on the parent dataset do not reflect what the trainer has actually consumed. Calling `state_dict()` on the
-parent in that setup raises `RuntimeError` and points you at `StreamingDataLoader`.
-
-Use `StreamingDataLoader` when you want multi-worker prefetch and exact resumability. It carries a state snapshot
-alongside every internal batch and commits it to the parent `StreamingDataset` only when that batch is returned to the
-trainer. The trainer receives the same collated batch it would receive from a standard `DataLoader`.
-
-<CodeGroup>
-```py Python icon=Python
-from lancedb.streaming import StreamingDataset, StreamingDataLoader
-
-dataset = StreamingDataset(
-    table,
-    num_splits=48,
-    shuffle_seed=42,
-    epoch=current_epoch,
-    rank=rank,
-    world_size=world_size,
-)
-dataloader = StreamingDataLoader(
-    dataset,
-    batch_size=batch_size,
-    num_workers=4,
-)
-
-for step, batch in enumerate(dataloader):
-    train_step(batch)
-
-    if (step + 1) % checkpoint_interval == 0:
-        torch.save(
-            {"model": model.state_dict(), "dataset": dataset.state_dict()},
-            f"checkpoint_{step + 1}.pt",
-        )
-```
-</CodeGroup>
-
-`StreamingDataLoader` accepts the same arguments as `torch.utils.data.DataLoader` with a few restrictions:
-
- `dataset` must be a `StreamingDataset` (subclasses that override `__iter__` are not supported).
- `in_order=True` is required so that consumer-committed checkpoints stay deterministic.
- `persistent_workers=True` is not supported, because prefetched worker copies cannot be restored from parent-committed state.
- `drop_last=True` is not supported, because incomplete tails discarded by worker replicas cannot be checkpointed topology-independently.
-
-With more than one worker, call `state_dict()` at a complete logical step boundary where every split assigned to the
-rank has the same consumed-sample count. Calling it mid-step raises `RuntimeError` asking you to consume more batches
-first.
-
-#### Resuming across different topologies {#resuming-across-different-topologies}
-
-When training across ranks, each rank owns its own subset of splits and only its own splits have exact progress. To
-resume on a different `world_size`, collect the `state_dict()` from every rank of the previous run and merge them with
-`StreamingDataset.merge_state_dicts` before calling `load_state_dict()` on the new run. The merge is topology-agnostic:
-pass the full list of per-rank states in, and hand the identical merged dict to every rank of the resumed job,
-regardless of whether the topology grew, shrank, or stayed the same.
-
-<CodeGroup>
-```py Python icon=Python
-# On each rank of the previous run:
-state = dataset.state_dict()
-# ...gather `state` from every rank into a single list `states` (for example
-# via torch.distributed all-gather or by saving one file per rank).
-
-merged = StreamingDataset.merge_state_dicts(states)
-
-# On each rank of the resumed run (world_size may differ):
-dataset = StreamingDataset(
-    table,
-    num_splits=merged["num_splits"],
-    shuffle_seed=merged["shuffle_seed"],
-    epoch=merged["epoch"],
-    rank=rank,
-    world_size=new_world_size,
-)
-dataset.load_state_dict(merged)
-```
-</CodeGroup>
-
-## Permutations {#permutations}
-
-In more complicated scenarios, you may want the flexibility to shuffle, split, and select data without using the full
-iterable streaming data loader. In these cases, use `Permutation`, the lower-level class on which `StreamingDataset`
-is built. A `Permutation` defines a custom ordering of the data and supports map-style access through `__getitem__()`
-and batched access through `__getitems__()`.
-
-### Base table version pinning {#base-table-version-pinning}
-
-A `Permutation` is pinned to the version of the base table at the time it was built, and every read (including reads
-from a `StreamingDataset` that wraps it, and every DataLoader worker after a `fork`) resolves against that pinned
-version. This makes iteration deterministic across compactions and worker forks: the permutation only addresses rows
-that existed when it was built, and rows appended to the base table afterwards are not visible through the existing
-permutation.
-
-<Note>
-To include newly appended rows, build a new `Permutation` (and rebuild any `StreamingDataset` that depends on it)
-against the updated table. Permutations written before this behavior shipped carry no pinned version and continue to
-read as they did before.
-</Note>
@@ -7,7 +7,7 @@ icon: car

 This example walks through fine-tuning an autonomous vehicle (AV) perception model on targeted failure-mode slices of [BDD100K](https://www.bdd100k.com/) — riders, nighttime pedestrians, and distant pedestrians — using LanceDB as a single multimodal table from raw JPEG bytes through to the PyTorch training loop.

-The full pipeline lives in the [lancedb/training](https://github.com/lancedb/training/tree/main/object-detection) repository. This page focuses on the parts most relevant to training: defining curated splits as materialized views, loading them through the [`Permutation`](/training/) API, and pinning checkpoints to an exact data version.
+The full pipeline lives in the [lancedb/training](https://github.com/lancedb/training/tree/main/object-detection) repository. This page focuses on the parts most relevant to training: defining curated splits as materialized views, loading them through the [`Permutation`](/training/data-loading) API, and pinning checkpoints to an exact data version.

 ## What you get {#what-you-get}

@@ -155,7 +155,7 @@ with gconn.local_ray_context():

 ## 4\. PyTorch DataLoader via the Permutation API {#4-pytorch-dataloader-via-the-permutation-api}

-The training script doesn't know about the filter — it opens a view by name and reads through the [`Permutation`](/training/) API. Each DataLoader worker reopens its own connection lazily, reads Arrow batches directly from Lance (zero-copy, no intermediate file format), and the collate function decodes the whole batch in one pass. `Permutation` provides random-access indexing over the table, so shuffling is a cheap pointer rewrite rather than a full-dataset shuffle on disk.
+The training script doesn't know about the filter — it opens a view by name and reads through the [`Permutation`](/training/data-loading) API. Each DataLoader worker reopens its own connection lazily, reads Arrow batches directly from Lance (zero-copy, no intermediate file format), and the collate function decodes the whole batch in one pass. `Permutation` provides random-access indexing over the table, so shuffling is a cheap pointer rewrite rather than a full-dataset shuffle on disk.

 ```py Python icon=Python
 import lancedb
@@ -32,7 +32,7 @@ for batch in dataloader:
 ```

 Although the `Table` class in LanceDB implements the `torch.utils.data.Dataset` interface, you may find that using
-a table [Permutation](/training/) is more flexible.
+a table [Permutation](/training/data-loading) is more flexible.

 ```py Python icon=Python 
 from lancedb.permutation import Permutation
@@ -103,7 +103,7 @@ access patterns in one format instead of scattering them across task-specific st
 ## Next steps {#next-steps}

 <CardGroup cols={2}>
-  <Card title="Data loading and shuffles" icon="boxes-stacked" href="/training/">
+  <Card title="Data loading and shuffles" icon="boxes-stacked" href="/training/data-loading">
    Learn how to use LanceDB permutations to select rows, project columns, split datasets, and shuffle training reads.
  </Card>
  <Card title="PyTorch integration" icon="fire" href="/training/torch">
@@ -39,4 +39,4 @@ are runnable in Colab; the rest are written up here.
 | [Reranking](/reranking) | Reordering results to improve the top-k. |
 | [Feature engineering](/feature-engineering) | Deriving columns and tables from data already stored. |
 | [Materialized views](/management/materialized-view) | A table defined by a query, refreshed incrementally. |
-| [Training](/training) | Reading a table into a data loader. |
+| [Training](/training/data-loading) | Reading a table into a data loader. |