Bump version: 0.17.1-beta.7 → 0.17.1

Bump version: 0.17.1-beta.6 → 0.17.1-beta.7
feat: support IVF_FLAT, binary vectors and hamming distance (#1955 )
2025-12-23 05:19:58 +00:00 · 2024-12-24 18:37:28 +00:00 · 2024-12-24 18:37:27 +00:00 · 2024-12-24 10:36:20 -08:00 · 2024-12-24 10:33:47 -08:00 · 2024-12-24 10:33:26 -08:00
47 changed files with 648 additions and 119 deletions
--- a/.bumpversion.toml
+++ b/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.14.1-beta.4"
+current_version = "0.14.1-beta.6"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/.github/workflows/make-release-commit.yml
+++ b/.github/workflows/make-release-commit.yml
@@ -97,3 +97,7 @@ jobs:
        if: ${{ !inputs.dry_run && inputs.other }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
+      - uses: ./.github/workflows/update_package_lock_nodejs
+        if: ${{ !inputs.dry_run && inputs.other }}
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
--- a/.github/workflows/npm-publish.yml
+++ b/.github/workflows/npm-publish.yml
@@ -571,7 +571,7 @@ jobs:
        uses: actions/checkout@v4
        with:
          ref: main
-          persist-credentials: false
+          token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
          fetch-depth: 0
          lfs: true
      - uses: ./.github/workflows/update_package_lock
@@ -589,7 +589,7 @@ jobs:
        uses: actions/checkout@v4
        with:
          ref: main
-          persist-credentials: false
+          token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
          fetch-depth: 0
          lfs: true
      - uses: ./.github/workflows/update_package_lock_nodejs
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -238,3 +238,41 @@ jobs:
          $env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
          cargo build --target aarch64-pc-windows-msvc
          cargo test --target aarch64-pc-windows-msvc
+
+  msrv:
+    # Check the minimum supported Rust version
+    name: MSRV Check - Rust v${{ matrix.msrv }}
+    runs-on: ubuntu-24.04
+    strategy:
+      matrix:
+        msrv: ["1.78.0"] # This should match up with rust-version in Cargo.toml
+    env:
+      # Need up-to-date compilers for kernels
+      CC: clang-18
+      CXX: clang++-18
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: true
+      - name: Install dependencies
+        run: |
+          sudo apt update
+          sudo apt install -y protobuf-compiler libssl-dev
+      - name: Install ${{ matrix.msrv }}
+        uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: ${{ matrix.msrv }}
+      - name: Downgrade  dependencies
+        # These packages have newer requirements for MSRV
+        run: |
+          cargo update -p aws-sdk-bedrockruntime --precise 1.64.0
+          cargo update -p aws-sdk-dynamodb --precise 1.55.0
+          cargo update -p aws-config --precise 1.5.10
+          cargo update -p aws-sdk-kms --precise 1.51.0
+          cargo update -p aws-sdk-s3 --precise 1.65.0
+          cargo update -p aws-sdk-sso --precise 1.50.0
+          cargo update -p aws-sdk-ssooidc --precise 1.51.0
+          cargo update -p aws-sdk-sts --precise 1.51.0
+          cargo update -p home --precise 0.5.9
+      - name: cargo +${{ matrix.msrv }} check
+        run: cargo check --workspace --tests --benches --all-features
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -18,19 +18,19 @@ repository = "https://github.com/lancedb/lancedb"
 description = "Serverless, low-latency vector database for AI applications"
 keywords = ["lancedb", "lance", "database", "vector", "search"]
 categories = ["database-implementations"]
-rust-version = "1.80.0"                                                     # TODO: lower this once we upgrade Lance again.
+rust-version = "1.78.0"

 [workspace.dependencies]
 lance = { "version" = "=0.21.0", "features" = [
    "dynamodb",
-], git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-io = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-index = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-linalg = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-table = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-testing = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-datafusion = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
-lance-encoding = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.4" }
+], git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-io = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-index = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-linalg = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-table = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-testing = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-datafusion = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
+lance-encoding = { version = "=0.21.0", git = "https://github.com/lancedb/lance.git", tag = "v0.21.0-beta.5" }
 # Note that this one does not include pyarrow
 arrow = { version = "53.2", optional = false }
 arrow-array = "53.2"
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -62,6 +62,7 @@ plugins:
            # for cross references
            - https://arrow.apache.org/docs/objects.inv
            - https://pandas.pydata.org/docs/objects.inv
+            - https://lancedb.github.io/lance/objects.inv
  - mkdocs-jupyter
  - render_swagger:
      allow_arbitrary_locations: true
--- a/docs/src/python/python.md
+++ b/docs/src/python/python.md
@@ -129,8 +129,12 @@ lists the indices that LanceDb supports.

 ::: lancedb.index.LabelList

+::: lancedb.index.FTS
+
 ::: lancedb.index.IvfPq

+::: lancedb.index.IvfFlat
+
 ## Querying (Asynchronous)

 Queries allow you to return data from your database. Basic queries can be
--- a/docs/src/python/saas-python.md
+++ b/docs/src/python/saas-python.md
@@ -17,4 +17,8 @@ pip install lancedb
 ## Table

 ::: lancedb.remote.table.RemoteTable
-
+    options:
+        filters:
+            - "!cleanup_old_versions"
+            - "!compact_files"
+            - "!optimize"
--- a/docs/src/search.md
+++ b/docs/src/search.md
@@ -14,10 +14,14 @@ Distance metrics are a measure of the similarity between a pair of vectors.
 Currently, LanceDB supports the following metrics:

 | Metric    | Description                                                                 |
-| -------- | --------------------------------------------------------------------------- |
+| --------- | --------------------------------------------------------------------------- |
 | `l2`      | [Euclidean / L2 distance](https://en.wikipedia.org/wiki/Euclidean_distance) |
 | `cosine`  | [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity)        |
 | `dot`     | [Dot Production](https://en.wikipedia.org/wiki/Dot_product)                 |
+| `hamming` | [Hamming Distance](https://en.wikipedia.org/wiki/Hamming_distance)          |
+
+!!! note
+    The `hamming` metric is only available for binary vectors.

 ## Exhaustive search (kNN)

@@ -107,6 +111,31 @@ an ANN search means that using an index often involves a trade-off between recal
 See the [IVF_PQ index](./concepts/index_ivfpq.md) for a deeper description of how `IVF_PQ`
 indexes work in LanceDB.

+## Binary vector
+
+LanceDB supports binary vectors as a data type, and has the ability to search binary vectors with hamming distance. The binary vectors are stored as uint8 arrays (every 8 bits are stored as a byte):
+
+!!! note
+    The dim of the binary vector must be a multiple of 8. A vector of dim 128 will be stored as a uint8 array of size 16.
+
+=== "Python"
+
+    === "sync API"
+
+        ```python
+        --8<-- "python/python/tests/docs/test_binary_vector.py:imports"
+
+        --8<-- "python/python/tests/docs/test_binary_vector.py:sync_binary_vector"
+        ```
+
+    === "async API"
+
+        ```python
+        --8<-- "python/python/tests/docs/test_binary_vector.py:imports"
+
+        --8<-- "python/python/tests/docs/test_binary_vector.py:async_binary_vector"
+        ```
+
 ## Output search results

 LanceDB returns vector search results via different formats commonly used in python.
--- a/docs/test/md_testing.py
+++ b/docs/test/md_testing.py
@@ -16,6 +16,7 @@ excluded_globs = [
    "../src/concepts/*.md",
    "../src/ann_indexes.md",
    "../src/basic.md",
+    "../src/search.md",
    "../src/hybrid_search/hybrid_search.md",
    "../src/reranking/*.md",
    "../src/guides/tuning_retrievers/*.md",
--- a/java/core/pom.xml
+++ b/java/core/pom.xml
@@ -8,7 +8,7 @@
    <parent>
        <groupId>com.lancedb</groupId>
        <artifactId>lancedb-parent</artifactId>
-        <version>0.14.1-beta.4</version>
+        <version>0.14.1-beta.6</version>
        <relativePath>../pom.xml</relativePath>
    </parent>

--- a/java/pom.xml
+++ b/java/pom.xml
@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.14.1-beta.4</version>
+    <version>0.14.1-beta.6</version>
    <packaging>pom</packaging>

    <name>LanceDB Parent</name>
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.14.1-beta.4",
+  "version": "0.14.1-beta.6",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.14.1-beta.4",
+      "version": "0.14.1-beta.6",
      "cpu": [
        "x64",
        "arm64"
@@ -52,14 +52,14 @@
        "uuid": "^9.0.0"
      },
      "optionalDependencies": {
-        "@lancedb/vectordb-darwin-arm64": "0.14.1-beta.4",
-        "@lancedb/vectordb-darwin-x64": "0.14.1-beta.4",
-        "@lancedb/vectordb-linux-arm64-gnu": "0.14.1-beta.4",
-        "@lancedb/vectordb-linux-arm64-musl": "0.14.1-beta.4",
-        "@lancedb/vectordb-linux-x64-gnu": "0.14.1-beta.4",
-        "@lancedb/vectordb-linux-x64-musl": "0.14.1-beta.4",
-        "@lancedb/vectordb-win32-arm64-msvc": "0.14.1-beta.4",
-        "@lancedb/vectordb-win32-x64-msvc": "0.14.1-beta.4"
+        "@lancedb/vectordb-darwin-arm64": "0.14.1-beta.6",
+        "@lancedb/vectordb-darwin-x64": "0.14.1-beta.6",
+        "@lancedb/vectordb-linux-arm64-gnu": "0.14.1-beta.6",
+        "@lancedb/vectordb-linux-arm64-musl": "0.14.1-beta.6",
+        "@lancedb/vectordb-linux-x64-gnu": "0.14.1-beta.6",
+        "@lancedb/vectordb-linux-x64-musl": "0.14.1-beta.6",
+        "@lancedb/vectordb-win32-arm64-msvc": "0.14.1-beta.6",
+        "@lancedb/vectordb-win32-x64-msvc": "0.14.1-beta.6"
      },
      "peerDependencies": {
        "@apache-arrow/ts": "^14.0.2",
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.14.1-beta.4",
+  "version": "0.14.1-beta.6",
  "description": " Serverless, low-latency vector database for AI applications",
  "private": false,
  "main": "dist/index.js",
@@ -92,13 +92,13 @@
    }
  },
  "optionalDependencies": {
-    "@lancedb/vectordb-darwin-x64": "0.14.1-beta.4",
-    "@lancedb/vectordb-darwin-arm64": "0.14.1-beta.4",
-    "@lancedb/vectordb-linux-x64-gnu": "0.14.1-beta.4",
-    "@lancedb/vectordb-linux-arm64-gnu": "0.14.1-beta.4",
-    "@lancedb/vectordb-linux-x64-musl": "0.14.1-beta.4",
-    "@lancedb/vectordb-linux-arm64-musl": "0.14.1-beta.4",
-    "@lancedb/vectordb-win32-x64-msvc": "0.14.1-beta.4",
-    "@lancedb/vectordb-win32-arm64-msvc": "0.14.1-beta.4"
+    "@lancedb/vectordb-darwin-x64": "0.14.1-beta.6",
+    "@lancedb/vectordb-darwin-arm64": "0.14.1-beta.6",
+    "@lancedb/vectordb-linux-x64-gnu": "0.14.1-beta.6",
+    "@lancedb/vectordb-linux-arm64-gnu": "0.14.1-beta.6",
+    "@lancedb/vectordb-linux-x64-musl": "0.14.1-beta.6",
+    "@lancedb/vectordb-linux-arm64-musl": "0.14.1-beta.6",
+    "@lancedb/vectordb-win32-x64-msvc": "0.14.1-beta.6",
+    "@lancedb/vectordb-win32-arm64-msvc": "0.14.1-beta.6"
  }
 }
--- a/nodejs/Cargo.toml
+++ b/nodejs/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.14.1-beta.4"
+version = "0.14.1-beta.6"
 license.workspace = true
 description.workspace = true
 repository.workspace = true
--- a/nodejs/npm/darwin-arm64/package.json
+++ b/nodejs/npm/darwin-arm64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
--- a/nodejs/npm/darwin-x64/package.json
+++ b/nodejs/npm/darwin-x64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-x64",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["darwin"],
 	"cpu": ["x64"],
 	"main": "lancedb.darwin-x64.node",
--- a/nodejs/npm/linux-arm64-gnu/package.json
+++ b/nodejs/npm/linux-arm64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
--- a/nodejs/npm/linux-arm64-musl/package.json
+++ b/nodejs/npm/linux-arm64-musl/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-musl",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-musl.node",
--- a/nodejs/npm/linux-x64-gnu/package.json
+++ b/nodejs/npm/linux-x64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
--- a/nodejs/npm/linux-x64-musl/package.json
+++ b/nodejs/npm/linux-x64-musl/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-musl",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-musl.node",
--- a/nodejs/npm/win32-arm64-msvc/package.json
+++ b/nodejs/npm/win32-arm64-msvc/package.json
@@ -1,6 +1,6 @@
 {
  "name": "@lancedb/lancedb-win32-arm64-msvc",
-  "version": "0.14.1-beta.4",
+  "version": "0.14.1-beta.6",
  "os": [
    "win32"
  ],
--- a/nodejs/npm/win32-x64-msvc/package.json
+++ b/nodejs/npm/win32-x64-msvc/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.14.1-beta.4",
+	"version": "0.14.1-beta.6",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
--- a/nodejs/package-lock.json
+++ b/nodejs/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.14.0",
+  "version": "0.14.1-beta.6",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.14.0",
+      "version": "0.14.1-beta.6",
      "cpu": [
        "x64",
        "arm64"
--- a/nodejs/package.json
+++ b/nodejs/package.json
@@ -11,7 +11,7 @@
    "ann"
  ],
  "private": false,
-  "version": "0.14.1-beta.4",
+  "version": "0.14.1-beta.6",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
--- a/nodejs/src/util.rs
+++ b/nodejs/src/util.rs
@@ -5,8 +5,9 @@ pub fn parse_distance_type(distance_type: impl AsRef<str>) -> napi::Result<Dista
        "l2" => Ok(DistanceType::L2),
        "cosine" => Ok(DistanceType::Cosine),
        "dot" => Ok(DistanceType::Dot),
+        "hamming" => Ok(DistanceType::Hamming),
        _ => Err(napi::Error::from_reason(format!(
-            "Invalid distance type '{}'.  Must be one of l2, cosine, or dot",
+            "Invalid distance type '{}'.  Must be one of l2, cosine, dot, or hamming",
            distance_type.as_ref()
        ))),
    }
--- a/python/.bumpversion.toml
+++ b/python/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.17.1-beta.5"
+current_version = "0.17.1"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/python/Cargo.toml
+++ b/python/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.17.1-beta.5"
+version = "0.17.1"
 edition.workspace = true
 description = "Python bindings for LanceDB"
 license.workspace = true
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -3,7 +3,7 @@ name = "lancedb"
 # version in Cargo.toml
 dependencies = [
    "deprecation",
-    "pylance==0.21.0b4",
+    "pylance==0.21.0b5",
    "tqdm>=4.27.0",
    "pydantic>=1.10",
    "packaging",
--- a/python/python/lancedb/db.py
+++ b/python/python/lancedb/db.py
@@ -18,12 +18,12 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Dict, Iterable, List, Literal, Optional, Union

 from lancedb.embeddings.registry import EmbeddingFunctionRegistry
-from overrides import EnforceOverrides, override
+from overrides import EnforceOverrides, override  # type: ignore

 from lancedb.common import data_to_reader, sanitize_uri, validate_schema
 from lancedb.background_loop import LOOP

-from ._lancedb import connect as lancedb_connect
+from ._lancedb import connect as lancedb_connect  # type: ignore
 from .table import (
    AsyncTable,
    LanceTable,
@@ -503,13 +503,7 @@ class LanceDBConnection(DBConnection):
        ignore_missing: bool, default False
            If True, ignore if the table does not exist.
        """
-        try:
-            LOOP.run(self._conn.drop_table(name))
-        except ValueError as e:
-            if not ignore_missing:
-                raise e
-            if f"Table '{name}' was not found" not in str(e):
-                raise e
+        LOOP.run(self._conn.drop_table(name, ignore_missing=ignore_missing))

    @override
    def drop_database(self):
@@ -886,15 +880,23 @@ class AsyncConnection(object):
        """
        await self._inner.rename_table(old_name, new_name)

-    async def drop_table(self, name: str):
+    async def drop_table(self, name: str, *, ignore_missing: bool = False):
        """Drop a table from the database.

        Parameters
        ----------
        name: str
            The name of the table.
+        ignore_missing: bool, default False
+            If True, ignore if the table does not exist.
        """
+        try:
            await self._inner.drop_table(name)
+        except ValueError as e:
+            if not ignore_missing:
+                raise e
+            if f"Table '{name}' was not found" not in str(e):
+                raise e

    async def drop_database(self):
        """
--- a/python/python/lancedb/index.py
+++ b/python/python/lancedb/index.py
@@ -355,6 +355,97 @@ class HnswSq:
    ef_construction: int = 300


+@dataclass
+class IvfFlat:
+    """Describes an IVF Flat Index
+
+    This index stores raw vectors.
+    These vectors are grouped into partitions of similar vectors.
+    Each partition keeps track of a centroid which is
+    the average value of all vectors in the group.
+
+    Attributes
+    ----------
+    distance_type: str, default "L2"
+        The distance metric used to train the index
+
+        This is used when training the index to calculate the IVF partitions
+        (vectors are grouped in partitions with similar vectors according to this
+        distance type) and to calculate a subvector's code during quantization.
+
+        The distance type used to train an index MUST match the distance type used
+        to search the index.  Failure to do so will yield inaccurate results.
+
+        The following distance types are available:
+
+        "l2" - Euclidean distance. This is a very common distance metric that
+        accounts for both magnitude and direction when determining the distance
+        between vectors. L2 distance has a range of [0, ∞).
+
+        "cosine" - Cosine distance.  Cosine distance is a distance metric
+        calculated from the cosine similarity between two vectors. Cosine
+        similarity is a measure of similarity between two non-zero vectors of an
+        inner product space. It is defined to equal the cosine of the angle
+        between them.  Unlike L2, the cosine distance is not affected by the
+        magnitude of the vectors.  Cosine distance has a range of [0, 2].
+
+        Note: the cosine distance is undefined when one (or both) of the vectors
+        are all zeros (there is no direction).  These vectors are invalid and may
+        never be returned from a vector search.
+
+        "dot" - Dot product. Dot distance is the dot product of two vectors. Dot
+        distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
+        L2 norm is 1), then dot distance is equivalent to the cosine distance.
+
+        "hamming" - Hamming distance. Hamming distance is a distance metric
+        calculated as the number of positions at which the corresponding bits are
+        different. Hamming distance has a range of [0, vector dimension].
+
+    num_partitions: int, default sqrt(num_rows)
+        The number of IVF partitions to create.
+
+        This value should generally scale with the number of rows in the dataset.
+        By default the number of partitions is the square root of the number of
+        rows.
+
+        If this value is too large then the first part of the search (picking the
+        right partition) will be slow.  If this value is too small then the second
+        part of the search (searching within a partition) will be slow.
+
+    max_iterations: int, default 50
+        Max iteration to train kmeans.
+
+        When training an IVF PQ index we use kmeans to calculate the partitions.
+        This parameter controls how many iterations of kmeans to run.
+
+        Increasing this might improve the quality of the index but in most cases
+        these extra iterations have diminishing returns.
+
+        The default value is 50.
+    sample_rate: int, default 256
+        The rate used to calculate the number of training vectors for kmeans.
+
+        When an IVF PQ index is trained, we need to calculate partitions.  These
+        are groups of vectors that are similar to each other.  To do this we use an
+        algorithm called kmeans.
+
+        Running kmeans on a large dataset can be slow.  To speed this up we run
+        kmeans on a random sample of the data.  This parameter controls the size of
+        the sample.  The total number of vectors used to train the index is
+        `sample_rate * num_partitions`.
+
+        Increasing this value might improve the quality of the index but in most
+        cases the default should be sufficient.
+
+        The default value is 256.
+    """
+
+    distance_type: Literal["l2", "cosine", "dot", "hamming"] = "l2"
+    num_partitions: Optional[int] = None
+    max_iterations: int = 50
+    sample_rate: int = 256
+
+
@dataclass
 class IvfPq:
    """Describes an IVF PQ Index
@@ -477,4 +568,4 @@ class IvfPq:
    sample_rate: int = 256


-__all__ = ["BTree", "IvfPq", "HnswPq", "HnswSq", "IndexConfig"]
+__all__ = ["BTree", "IvfFlat", "IvfPq", "HnswPq", "HnswSq", "IndexConfig"]
--- a/python/python/lancedb/query.py
+++ b/python/python/lancedb/query.py
@@ -126,6 +126,9 @@ class Query(pydantic.BaseModel):

    ef: Optional[int] = None

+    # Default is true. Set to false to enforce a brute force search.
+    use_index: bool = True
+

 class LanceQueryBuilder(ABC):
    """An abstract query builder. Subclasses are defined for vector search,
@@ -253,6 +256,7 @@ class LanceQueryBuilder(ABC):
        self._vector = None
        self._text = None
        self._ef = None
+        self._use_index = True

    @deprecation.deprecated(
        deprecated_in="0.3.1",
@@ -511,6 +515,7 @@ class LanceQueryBuilder(ABC):
                "metric": self._metric,
                "nprobes": self._nprobes,
                "refine_factor": self._refine_factor,
+                "use_index": self._use_index,
            },
            prefilter=self._prefilter,
            filter=self._str_query,
@@ -729,6 +734,7 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
            offset=self._offset,
            fast_search=self._fast_search,
            ef=self._ef,
+            use_index=self._use_index,
        )
        result_set = self._table._execute_query(query, batch_size)
        if self._reranker is not None:
@@ -802,6 +808,24 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
        self._str_query = query_string if query_string is not None else self._str_query
        return self

+    def bypass_vector_index(self) -> LanceVectorQueryBuilder:
+        """
+        If this is called then any vector index is skipped
+
+        An exhaustive (flat) search will be performed.  The query vector will
+        be compared to every vector in the table.  At high scales this can be
+        expensive.  However, this is often still useful.  For example, skipping
+        the vector index can give you ground truth results which you can use to
+        calculate your recall to select an appropriate value for nprobes.
+
+        Returns
+        -------
+        LanceVectorQueryBuilder
+            The LanceVectorQueryBuilder object.
+        """
+        self._use_index = False
+        return self
+

 class LanceFtsQueryBuilder(LanceQueryBuilder):
    """A builder for full text search for LanceDB."""
@@ -1108,6 +1132,8 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
            self._vector_query.refine_factor(self._refine_factor)
        if self._ef:
            self._vector_query.ef(self._ef)
+        if not self._use_index:
+            self._vector_query.bypass_vector_index()

        with ThreadPoolExecutor() as executor:
            fts_future = executor.submit(self._fts_query.with_row_id(True).to_arrow)
@@ -1323,6 +1349,24 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
        self._text = text
        return self

+    def bypass_vector_index(self) -> LanceHybridQueryBuilder:
+        """
+        If this is called then any vector index is skipped
+
+        An exhaustive (flat) search will be performed.  The query vector will
+        be compared to every vector in the table.  At high scales this can be
+        expensive.  However, this is often still useful.  For example, skipping
+        the vector index can give you ground truth results which you can use to
+        calculate your recall to select an appropriate value for nprobes.
+
+        Returns
+        -------
+        LanceHybridQueryBuilder
+            The LanceHybridQueryBuilder object.
+        """
+        self._use_index = False
+        return self
+

 class AsyncQueryBase(object):
    def __init__(self, inner: Union[LanceQuery | LanceVectorQuery]):
--- a/python/python/lancedb/remote/table.py
+++ b/python/python/lancedb/remote/table.py
@@ -15,6 +15,7 @@ from datetime import timedelta
 import logging
 from functools import cached_property
 from typing import Dict, Iterable, List, Optional, Union, Literal
+import warnings

 from lancedb._lancedb import IndexConfig
 from lancedb.embeddings.base import EmbeddingFunctionConfig
@@ -481,16 +482,28 @@ class RemoteTable(Table):
        )

    def cleanup_old_versions(self, *_):
-        """cleanup_old_versions() is not supported on the LanceDB cloud"""
-        raise NotImplementedError(
-            "cleanup_old_versions() is not supported on the LanceDB cloud"
+        """
+        cleanup_old_versions() is a no-op on LanceDB Cloud.
+
+        Tables are automatically cleaned up and optimized.
+        """
+        warnings.warn(
+            "cleanup_old_versions() is a no-op on LanceDB Cloud. "
+            "Tables are automatically cleaned up and optimized."
        )
+        pass

    def compact_files(self, *_):
-        """compact_files() is not supported on the LanceDB cloud"""
-        raise NotImplementedError(
-            "compact_files() is not supported on the LanceDB cloud"
+        """
+        compact_files() is a no-op on LanceDB Cloud.
+
+        Tables are automatically compacted and optimized.
+        """
+        warnings.warn(
+            "compact_files() is a no-op on LanceDB Cloud. "
+            "Tables are automatically compacted and optimized."
        )
+        pass

    def optimize(
        self,
@@ -498,12 +511,16 @@ class RemoteTable(Table):
        cleanup_older_than: Optional[timedelta] = None,
        delete_unverified: bool = False,
    ):
-        """optimize() is not supported on the LanceDB cloud.
-        Indices are optimized automatically."""
-        raise NotImplementedError(
-            "optimize() is not supported on the LanceDB cloud. "
+        """
+        optimize() is a no-op on LanceDB Cloud.
+
+        Indices are optimized automatically.
+        """
+        warnings.warn(
+            "optimize() is a no-op on LanceDB Cloud. "
            "Indices are optimized automatically."
        )
+        pass

    def count_rows(self, filter: Optional[str] = None) -> int:
        return LOOP.run(self._table.count_rows(filter))
--- a/python/python/lancedb/table.py
+++ b/python/python/lancedb/table.py
@@ -34,7 +34,7 @@ from lance.dependencies import _check_for_hugging_face

 from .common import DATA, VEC, VECTOR_COLUMN_NAME
 from .embeddings import EmbeddingFunctionConfig, EmbeddingFunctionRegistry
-from .index import BTree, IvfPq, Bitmap, LabelList, HnswPq, HnswSq, FTS
+from .index import BTree, IvfFlat, IvfPq, Bitmap, LabelList, HnswPq, HnswSq, FTS
 from .merge import LanceMergeInsertBuilder
 from .pydantic import LanceModel, model_to_dict
 from .query import (
@@ -433,7 +433,9 @@ class Table(ABC):
        accelerator: Optional[str] = None,
        index_cache_size: Optional[int] = None,
        *,
-        index_type: Literal["IVF_PQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"] = "IVF_PQ",
+        index_type: Literal[
+            "IVF_FLAT", "IVF_PQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"
+        ] = "IVF_PQ",
        num_bits: int = 8,
        max_iterations: int = 50,
        sample_rate: int = 256,
@@ -446,8 +448,9 @@ class Table(ABC):
        ----------
        metric: str, default "L2"
            The distance metric to use when creating the index.
-            Valid values are "L2", "cosine", or "dot".
+            Valid values are "L2", "cosine", "dot", or "hamming".
            L2 is euclidean distance.
+            Hamming is available only for binary vectors.
        num_partitions: int, default 256
            The number of IVF partitions to use when creating the index.
            Default is 256.
@@ -917,9 +920,6 @@ class Table(ABC):
        """
        Clean up old versions of the table, freeing disk space.

-        Note: This function is not available in LanceDb Cloud (since LanceDb
-        Cloud manages cleanup for you automatically)
-
        Parameters
        ----------
        older_than: timedelta, default None
@@ -936,21 +936,38 @@ class Table(ABC):
        CleanupStats
            The stats of the cleanup operation, including how many bytes were
            freed.
+
+        See Also
+        --------
+        [Table.optimize][lancedb.table.Table.optimize]: A more comprehensive
+            optimization operation that includes cleanup as well as other operations.
+
+        Notes
+        -----
+        This function is not available in LanceDb Cloud (since LanceDB
+        Cloud manages cleanup for you automatically)
        """

    @abstractmethod
    def compact_files(self, *args, **kwargs):
        """
        Run the compaction process on the table.
-
-        Note: This function is not available in LanceDb Cloud (since LanceDb
-        Cloud manages compaction for you automatically)
-
        This can be run after making several small appends to optimize the table
        for faster reads.

-        Arguments are passed onto :meth:`lance.dataset.DatasetOptimizer.compact_files`.
+        Arguments are passed onto Lance's
+        [compact_files][lance.dataset.DatasetOptimizer.compact_files].
        For most cases, the default should be fine.
+
+        See Also
+        --------
+        [Table.optimize][lancedb.table.Table.optimize]: A more comprehensive
+            optimization operation that includes cleanup as well as other operations.
+
+        Notes
+        -----
+        This function is not available in LanceDB Cloud (since LanceDB
+        Cloud manages compaction for you automatically)
        """

    @abstractmethod
@@ -1394,7 +1411,9 @@ class LanceTable(Table):
        accelerator: Optional[str] = None,
        index_cache_size: Optional[int] = None,
        num_bits: int = 8,
-        index_type: Literal["IVF_PQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"] = "IVF_PQ",
+        index_type: Literal[
+            "IVF_FLAT", "IVF_PQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"
+        ] = "IVF_PQ",
        max_iterations: int = 50,
        sample_rate: int = 256,
        m: int = 20,
@@ -1418,6 +1437,13 @@ class LanceTable(Table):
            )
            self.checkout_latest()
            return
+        elif index_type == "IVF_FLAT":
+            config = IvfFlat(
+                distance_type=metric,
+                num_partitions=num_partitions,
+                max_iterations=max_iterations,
+                sample_rate=sample_rate,
+            )
        elif index_type == "IVF_PQ":
            config = IvfPq(
                distance_type=metric,
@@ -2605,7 +2631,7 @@ class AsyncTable:
        *,
        replace: Optional[bool] = None,
        config: Optional[
-            Union[IvfPq, HnswPq, HnswSq, BTree, Bitmap, LabelList, FTS]
+            Union[IvfFlat, IvfPq, HnswPq, HnswSq, BTree, Bitmap, LabelList, FTS]
        ] = None,
    ):
        """Create an index to speed up queries
@@ -2634,7 +2660,7 @@ class AsyncTable:
        """
        if config is not None:
            if not isinstance(
-                config, (IvfPq, HnswPq, HnswSq, BTree, Bitmap, LabelList, FTS)
+                config, (IvfFlat, IvfPq, HnswPq, HnswSq, BTree, Bitmap, LabelList, FTS)
            ):
                raise TypeError(
                    "config must be an instance of IvfPq, HnswPq, HnswSq, BTree,"
@@ -2798,6 +2824,8 @@ class AsyncTable:
                async_query = async_query.column(query.vector_column)
            if query.ef:
                async_query = async_query.ef(query.ef)
+            if not query.use_index:
+                async_query = async_query.bypass_vector_index()

        if not query.prefilter:
            async_query = async_query.postfilter()
--- a/python/python/tests/docs/test_binary_vector.py
+++ b/python/python/tests/docs/test_binary_vector.py
@@ -0,0 +1,44 @@
+import shutil
+
+# --8<-- [start:imports]
+import lancedb
+import numpy as np
+import pytest
+# --8<-- [end:imports]
+
+shutil.rmtree("data/binary_lancedb", ignore_errors=True)
+
+
+def test_binary_vector():
+    # --8<-- [start:sync_binary_vector]
+    db = lancedb.connect("data/binary_lancedb")
+    data = [
+        {
+            "id": i,
+            "vector": np.random.randint(0, 256, size=16),
+        }
+        for i in range(1024)
+    ]
+    tbl = db.create_table("my_binary_vectors", data=data)
+    query = np.random.randint(0, 256, size=16)
+    tbl.search(query).to_arrow()
+    # --8<-- [end:sync_binary_vector]
+    db.drop_table("my_binary_vectors")
+
+
+@pytest.mark.asyncio
+async def test_binary_vector_async():
+    # --8<-- [start:async_binary_vector]
+    db = await lancedb.connect_async("data/binary_lancedb")
+    data = [
+        {
+            "id": i,
+            "vector": np.random.randint(0, 256, size=16),
+        }
+        for i in range(1024)
+    ]
+    tbl = await db.create_table("my_binary_vectors", data=data)
+    query = np.random.randint(0, 256, size=16)
+    await tbl.query().nearest_to(query).to_arrow()
+    # --8<-- [end:async_binary_vector]
+    await db.drop_table("my_binary_vectors")
--- a/python/python/tests/test_db.py
+++ b/python/python/tests/test_db.py
@@ -508,6 +508,32 @@ def test_delete_table(tmp_db: lancedb.DBConnection):
    tmp_db.drop_table("does_not_exist", ignore_missing=True)


+@pytest.mark.asyncio
+async def test_delete_table_async(tmp_db: lancedb.DBConnection):
+    data = pd.DataFrame(
+        {
+            "vector": [[3.1, 4.1], [5.9, 26.5]],
+            "item": ["foo", "bar"],
+            "price": [10.0, 20.0],
+        }
+    )
+
+    tmp_db.create_table("test", data=data)
+
+    with pytest.raises(Exception):
+        tmp_db.create_table("test", data=data)
+
+    assert tmp_db.table_names() == ["test"]
+
+    tmp_db.drop_table("test")
+    assert tmp_db.table_names() == []
+
+    tmp_db.create_table("test", data=data)
+    assert tmp_db.table_names() == ["test"]
+
+    tmp_db.drop_table("does_not_exist", ignore_missing=True)
+
+
 def test_drop_database(tmp_db: lancedb.DBConnection):
    data = pd.DataFrame(
        {
@@ -681,3 +707,25 @@ def test_create_table_with_invalid_names(tmp_db: lancedb.DBConnection):
    with pytest.raises(ValueError):
        tmp_db.create_table("foo$$bar", data)
    tmp_db.create_table("foo.bar", data)
+
+
+def test_bypass_vector_index_sync(tmp_db: lancedb.DBConnection):
+    data = [{"vector": np.random.rand(32)} for _ in range(512)]
+    sample_key = data[100]["vector"]
+    table = tmp_db.create_table(
+        "test",
+        data,
+    )
+
+    table.create_index(
+        num_partitions=2,
+        num_sub_vectors=2,
+    )
+
+    plan_with_index = table.search(sample_key).explain_plan(verbose=True)
+    assert "ANN" in plan_with_index
+
+    plan_without_index = (
+        table.search(sample_key).bypass_vector_index().explain_plan(verbose=True)
+    )
+    assert "KNN" in plan_without_index
--- a/python/python/tests/test_index.py
+++ b/python/python/tests/test_index.py
@@ -8,7 +8,7 @@ import pyarrow as pa
 import pytest
 import pytest_asyncio
 from lancedb import AsyncConnection, AsyncTable, connect_async
-from lancedb.index import BTree, IvfPq, Bitmap, LabelList, HnswPq, HnswSq
+from lancedb.index import BTree, IvfFlat, IvfPq, Bitmap, LabelList, HnswPq, HnswSq


@pytest_asyncio.fixture
@@ -42,6 +42,27 @@ async def some_table(db_async):
    )


+@pytest_asyncio.fixture
+async def binary_table(db_async):
+    data = [
+        {
+            "id": i,
+            "vector": [i] * 128,
+        }
+        for i in range(NROWS)
+    ]
+    return await db_async.create_table(
+        "binary_table",
+        data,
+        schema=pa.schema(
+            [
+                pa.field("id", pa.int64()),
+                pa.field("vector", pa.list_(pa.uint8(), 128)),
+            ]
+        ),
+    )
+
+
@pytest.mark.asyncio
 async def test_create_scalar_index(some_table: AsyncTable):
    # Can create
@@ -143,3 +164,27 @@ async def test_create_hnswsq_index(some_table: AsyncTable):
    await some_table.create_index("vector", config=HnswSq(num_partitions=10))
    indices = await some_table.list_indices()
    assert len(indices) == 1
+
+
+@pytest.mark.asyncio
+async def test_create_index_with_binary_vectors(binary_table: AsyncTable):
+    await binary_table.create_index(
+        "vector", config=IvfFlat(distance_type="hamming", num_partitions=10)
+    )
+    indices = await binary_table.list_indices()
+    assert len(indices) == 1
+    assert indices[0].index_type == "IvfFlat"
+    assert indices[0].columns == ["vector"]
+    assert indices[0].name == "vector_idx"
+
+    stats = await binary_table.index_stats("vector_idx")
+    assert stats.index_type == "IVF_FLAT"
+    assert stats.distance_type == "hamming"
+    assert stats.num_indexed_rows == await binary_table.count_rows()
+    assert stats.num_unindexed_rows == 0
+    assert stats.num_indices == 1
+
+    # the dataset contains vectors with all values from 0 to 255
+    for v in range(256):
+        res = await binary_table.query().nearest_to([v] * 128).to_arrow()
+        assert res["id"][0].as_py() == v
--- a/python/src/index.rs
+++ b/python/src/index.rs
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+use lancedb::index::vector::IvfFlatIndexBuilder;
 use lancedb::index::{
    scalar::{BTreeIndexBuilder, FtsIndexBuilder, TokenizerConfig},
    vector::{IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder},
@@ -59,6 +60,18 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
                opts.tokenizer_configs = inner_opts;
                Ok(LanceDbIndex::FTS(opts))
            },
+            "IvfFlat" => {
+                let params = source.extract::<IvfFlatParams>()?;
+                let distance_type = parse_distance_type(params.distance_type)?;
+                let mut ivf_flat_builder = IvfFlatIndexBuilder::default()
+                    .distance_type(distance_type)
+                    .max_iterations(params.max_iterations)
+                    .sample_rate(params.sample_rate);
+                if let Some(num_partitions) = params.num_partitions {
+                    ivf_flat_builder = ivf_flat_builder.num_partitions(num_partitions);
+                }
+                Ok(LanceDbIndex::IvfFlat(ivf_flat_builder))
+            },
            "IvfPq" => {
                let params = source.extract::<IvfPqParams>()?;
                let distance_type = parse_distance_type(params.distance_type)?;
@@ -129,6 +142,14 @@ struct FtsParams {
    ascii_folding: bool,
 }

+#[derive(FromPyObject)]
+struct IvfFlatParams {
+    distance_type: String,
+    num_partitions: Option<u32>,
+    max_iterations: u32,
+    sample_rate: u32,
+}
+
 #[derive(FromPyObject)]
 struct IvfPqParams {
    distance_type: String,
--- a/python/src/util.rs
+++ b/python/src/util.rs
@@ -43,8 +43,9 @@ pub fn parse_distance_type(distance_type: impl AsRef<str>) -> PyResult<DistanceT
        "l2" => Ok(DistanceType::L2),
        "cosine" => Ok(DistanceType::Cosine),
        "dot" => Ok(DistanceType::Dot),
+        "hamming" => Ok(DistanceType::Hamming),
        _ => Err(PyValueError::new_err(format!(
-            "Invalid distance type '{}'.  Must be one of l2, cosine, or dot",
+            "Invalid distance type '{}'.  Must be one of l2, cosine, dot, or hamming",
            distance_type.as_ref()
        ))),
    }
--- a/rust-toolchain.toml
+++ b/rust-toolchain.toml
@@ -1,2 +1,2 @@
 [toolchain]
-channel = "1.80.0"
+channel = "1.83.0"
--- a/rust/ffi/node/Cargo.toml
+++ b/rust/ffi/node/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-node"
-version = "0.14.1-beta.4"
+version = "0.14.1-beta.6"
 description = "Serverless, low-latency vector database for AI applications"
 license.workspace = true
 edition.workspace = true
--- a/rust/lancedb/Cargo.toml
+++ b/rust/lancedb/Cargo.toml
@@ -1,13 +1,13 @@
 [package]
 name = "lancedb"
-version = "0.14.1-beta.4"
+version = "0.14.1-beta.6"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
 repository.workspace = true
 keywords.workspace = true
 categories.workspace = true
-rust-version = "1.75"
+rust-version.workspace = true

 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 [dependencies]
--- a/rust/lancedb/src/index.rs
+++ b/rust/lancedb/src/index.rs
@@ -17,6 +17,7 @@ use std::sync::Arc;
 use scalar::FtsIndexBuilder;
 use serde::Deserialize;
 use serde_with::skip_serializing_none;
+use vector::IvfFlatIndexBuilder;

 use crate::{table::TableInternal, DistanceType, Error, Result};

@@ -56,6 +57,9 @@ pub enum Index {
    /// Full text search index using bm25.
    FTS(FtsIndexBuilder),

+    /// IVF index
+    IvfFlat(IvfFlatIndexBuilder),
+
    /// IVF index with Product Quantization
    IvfPq(IvfPqIndexBuilder),

@@ -106,6 +110,8 @@ impl IndexBuilder {
 #[derive(Debug, Clone, PartialEq, Deserialize)]
 pub enum IndexType {
    // Vector
+    #[serde(alias = "IVF_FLAT")]
+    IvfFlat,
    #[serde(alias = "IVF_PQ")]
    IvfPq,
    #[serde(alias = "IVF_HNSW_PQ")]
@@ -127,6 +133,7 @@ pub enum IndexType {
 impl std::fmt::Display for IndexType {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
+            Self::IvfFlat => write!(f, "IVF_FLAT"),
            Self::IvfPq => write!(f, "IVF_PQ"),
            Self::IvfHnswPq => write!(f, "IVF_HNSW_PQ"),
            Self::IvfHnswSq => write!(f, "IVF_HNSW_SQ"),
@@ -147,6 +154,7 @@ impl std::str::FromStr for IndexType {
            "BITMAP" => Ok(Self::Bitmap),
            "LABEL_LIST" | "LABELLIST" => Ok(Self::LabelList),
            "FTS" | "INVERTED" => Ok(Self::FTS),
+            "IVF_FLAT" => Ok(Self::IvfFlat),
            "IVF_PQ" => Ok(Self::IvfPq),
            "IVF_HNSW_PQ" => Ok(Self::IvfHnswPq),
            "IVF_HNSW_SQ" => Ok(Self::IvfHnswSq),
--- a/rust/lancedb/src/index/vector.rs
+++ b/rust/lancedb/src/index/vector.rs
@@ -162,6 +162,43 @@ macro_rules! impl_hnsw_params_setter {
    };
 }

+/// Builder for an IVF Flat index.
+///
+/// This index stores raw vectors. These vectors are grouped into partitions of similar vectors.
+/// Each partition keeps track of a centroid which is the average value of all vectors in the group.
+///
+/// During a query the centroids are compared with the query vector to find the closest partitions.
+/// The raw vectors in these partitions are then searched to find the closest vectors.
+///
+/// The partitioning process is called IVF and the `num_partitions` parameter controls how many groups to create.
+///
+/// Note that training an IVF Flat index on a large dataset is a slow operation and currently is also a memory intensive operation.
+#[derive(Debug, Clone)]
+pub struct IvfFlatIndexBuilder {
+    pub(crate) distance_type: DistanceType,
+
+    // IVF
+    pub(crate) num_partitions: Option<u32>,
+    pub(crate) sample_rate: u32,
+    pub(crate) max_iterations: u32,
+}
+
+impl Default for IvfFlatIndexBuilder {
+    fn default() -> Self {
+        Self {
+            distance_type: DistanceType::L2,
+            num_partitions: None,
+            sample_rate: 256,
+            max_iterations: 50,
+        }
+    }
+}
+
+impl IvfFlatIndexBuilder {
+    impl_distance_type_setter!();
+    impl_ivf_params_setter!();
+}
+
 /// Builder for an IVF PQ index.
 ///
 /// This index stores a compressed (quantized) copy of every vector.  These vectors
--- a/rust/lancedb/src/query.rs
+++ b/rust/lancedb/src/query.rs
@@ -339,7 +339,7 @@ pub trait QueryBase {
    fn limit(self, limit: usize) -> Self;

    /// Set the offset of the query.
-
+    ///
    /// By default, it fetches starting with the first row.
    /// This method can be used to skip the first `offset` rows.
    fn offset(self, offset: usize) -> Self;
--- a/rust/lancedb/src/table.rs
+++ b/rust/lancedb/src/table.rs
@@ -18,9 +18,9 @@ use std::path::Path;
 use std::sync::Arc;

 use arrow::array::AsArray;
-use arrow::datatypes::Float32Type;
+use arrow::datatypes::{Float32Type, UInt8Type};
 use arrow_array::{RecordBatchIterator, RecordBatchReader};
-use arrow_schema::{Field, Schema, SchemaRef};
+use arrow_schema::{DataType, Field, Schema, SchemaRef};
 use async_trait::async_trait;
 use datafusion_physical_plan::display::DisplayableExecutionPlan;
 use datafusion_physical_plan::projection::ProjectionExec;
@@ -58,8 +58,8 @@ use crate::embeddings::{EmbeddingDefinition, EmbeddingRegistry, MaybeEmbedded, M
 use crate::error::{Error, Result};
 use crate::index::scalar::FtsIndexBuilder;
 use crate::index::vector::{
-    suggested_num_partitions_for_hnsw, IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder,
-    IvfPqIndexBuilder, VectorIndex,
+    suggested_num_partitions_for_hnsw, IvfFlatIndexBuilder, IvfHnswPqIndexBuilder,
+    IvfHnswSqIndexBuilder, IvfPqIndexBuilder, VectorIndex,
 };
 use crate::index::IndexStatistics;
 use crate::index::{
@@ -1306,6 +1306,44 @@ impl NativeTable {
            .collect())
    }

+    async fn create_ivf_flat_index(
+        &self,
+        index: IvfFlatIndexBuilder,
+        field: &Field,
+        replace: bool,
+    ) -> Result<()> {
+        if !supported_vector_data_type(field.data_type()) {
+            return Err(Error::InvalidInput {
+                message: format!(
+                    "An IVF Flat index cannot be created on the column `{}` which has data type {}",
+                    field.name(),
+                    field.data_type()
+                ),
+            });
+        }
+
+        let num_partitions = if let Some(n) = index.num_partitions {
+            n
+        } else {
+            suggested_num_partitions(self.count_rows(None).await?)
+        };
+        let mut dataset = self.dataset.get_mut().await?;
+        let lance_idx_params = lance::index::vector::VectorIndexParams::ivf_flat(
+            num_partitions as usize,
+            index.distance_type.into(),
+        );
+        dataset
+            .create_index(
+                &[field.name()],
+                IndexType::Vector,
+                None,
+                &lance_idx_params,
+                replace,
+            )
+            .await?;
+        Ok(())
+    }
+
    async fn create_ivf_pq_index(
        &self,
        index: IvfPqIndexBuilder,
@@ -1778,6 +1816,10 @@ impl TableInternal for NativeTable {
            Index::Bitmap(_) => self.create_bitmap_index(field, opts).await,
            Index::LabelList(_) => self.create_label_list_index(field, opts).await,
            Index::FTS(fts_opts) => self.create_fts_index(field, fts_opts, opts.replace).await,
+            Index::IvfFlat(ivf_flat) => {
+                self.create_ivf_flat_index(ivf_flat, field, opts.replace)
+                    .await
+            }
            Index::IvfPq(ivf_pq) => self.create_ivf_pq_index(ivf_pq, field, opts.replace).await,
            Index::IvfHnswPq(ivf_hnsw_pq) => {
                self.create_ivf_hnsw_pq_index(ivf_hnsw_pq, field, opts.replace)
@@ -1848,8 +1890,14 @@ impl TableInternal for NativeTable {
                message: format!("Column {} not found in dataset schema", column),
            })?;

-            if let arrow_schema::DataType::FixedSizeList(f, dim) = field.data_type() {
-                if !f.data_type().is_floating() {
+            let mut is_binary = false;
+            if let arrow_schema::DataType::FixedSizeList(element, dim) = field.data_type() {
+                match element.data_type() {
+                    e_type if e_type.is_floating() => {}
+                    e_type if *e_type == DataType::UInt8 => {
+                        is_binary = true;
+                    }
+                    _ => {
                        return Err(Error::InvalidInput {
                            message: format!(
                                "The data type of the vector column '{}' is not a floating point type",
@@ -1857,6 +1905,7 @@ impl TableInternal for NativeTable {
                            ),
                        });
                    }
+                }
                if dim != query_vector.len() as i32 {
                    return Err(Error::InvalidInput {
                    message: format!(
@@ -1870,6 +1919,15 @@ impl TableInternal for NativeTable {
                }
            }

+            if is_binary {
+                let query_vector = arrow::compute::cast(&query_vector, &DataType::UInt8)?;
+                let query_vector = query_vector.as_primitive::<UInt8Type>();
+                scanner.nearest(
+                    &column,
+                    query_vector,
+                    query.base.limit.unwrap_or(DEFAULT_TOP_K),
+                )?;
+            } else {
                let query_vector = query_vector.as_primitive::<Float32Type>();
                scanner.nearest(
                    &column,
@@ -1877,6 +1935,7 @@ impl TableInternal for NativeTable {
                    query.base.limit.unwrap_or(DEFAULT_TOP_K),
                )?;
            }
+        }
        scanner.limit(
            query.base.limit.map(|limit| limit as i64),
            query.base.offset.map(|offset| offset as i64),
--- a/rust/lancedb/src/utils.rs
+++ b/rust/lancedb/src/utils.rs
@@ -110,7 +110,7 @@ pub(crate) fn default_vector_column(schema: &Schema, dim: Option<i32>) -> Result
        .iter()
        .filter_map(|field| match field.data_type() {
            arrow_schema::DataType::FixedSizeList(f, d)
-                if f.data_type().is_floating()
+                if (f.data_type().is_floating() || f.data_type() == &DataType::UInt8)
                    && dim.map(|expect| *d == expect).unwrap_or(true) =>
            {
                Some(field.name())
@@ -171,7 +171,9 @@ pub fn supported_fts_data_type(dtype: &DataType) -> bool {

 pub fn supported_vector_data_type(dtype: &DataType) -> bool {
    match dtype {
-        DataType::FixedSizeList(inner, _) => DataType::is_floating(inner.data_type()),
+        DataType::FixedSizeList(inner, _) => {
+            DataType::is_floating(inner.data_type()) || *inner.data_type() == DataType::UInt8
+        }
        _ => false,
    }
 }
Author	SHA1	Message	Date
Lance Release	27404c8623	Bump version: 0.17.1-beta.7 → 0.17.1	2024-12-24 18:37:28 +00:00
Lance Release	f181c7e77f	Bump version: 0.17.1-beta.6 → 0.17.1-beta.7	2024-12-24 18:37:27 +00:00
BubbleCal	e70fd4fecc	feat: support IVF_FLAT, binary vectors and hamming distance (#1955 ) binary vectors and hamming distance can work on only IVF_FLAT, so introduce them all in this PR. --------- Signed-off-by: BubbleCal <bubble-cal@outlook.com>	2024-12-24 10:36:20 -08:00
verma nakul	ac0068b80e	feat(python): add `ignore_missing` to the async drop_table() method (#1953 ) - feat(db): add `ignore_missing` to async `drop_table` method Fixes #1951 --------- Co-authored-by: Will Jones <willjones127@gmail.com>	2024-12-24 10:33:47 -08:00
Hezi Zisman	ebac960571	feat(python): add `bypass_vector_index` to sync api (#1947 ) Hi lancedb team, This PR adds the `bypass_vector_index` logic to the sync API, as described in [Issue #535](https://github.com/lancedb/lancedb/issues/535). (Closes #535). Iv'e implemented it only for the regular vector search. If you think it should also be supported for FTS, Hybrid, or Empty queries and for the cloud solution, please let me know, and I’ll be happy to extend it. Since there’s no `CONTRIBUTING.md` or contribution guidelines, I opted for the simplest implementation to get this started. Looking forward to your feedback! Thanks! --------- Co-authored-by: Will Jones <willjones127@gmail.com>	2024-12-24 10:33:26 -08:00
Lance Release	59b57055e7	Updating package-lock.json	2024-12-19 19:40:28 +00:00
Lance Release	591c8de8fc	Updating package-lock.json	2024-12-19 19:40:13 +00:00
Lance Release	f835ff310f	Bump version: 0.14.1-beta.5 → 0.14.1-beta.6	2024-12-19 19:39:41 +00:00
Lance Release	cf8c2edaf4	Bump version: 0.17.1-beta.5 → 0.17.1-beta.6	2024-12-19 19:39:08 +00:00
Will Jones	61a714a459	docs: improve optimization docs (#1957 ) * Add `See Also` section to `cleanup_old_files` and `compact_files` so they know it's linked to `optimize`. * Fixes link to `compact_files` arguments * Improves formatting of note.	2024-12-19 10:55:11 -08:00
Will Jones	5ddd84cec0	feat: upgrade lance to 0.21.0-beta.5 (#1961 )	2024-12-19 10:54:59 -08:00
Will Jones	27ef0bb0a2	ci(rust): check MSRV and upgrade toolchain (#1960 ) * Upgrades our toolchain file to v1.83.0, since many dependencies now have MSRV of 1.81.0 * Reverts Rust changes from #1946 that were working around this in a dumb way * Adding an MSRV check * Reduce MSRV back to 1.78.0	2024-12-19 08:43:25 -08:00
Will Jones	25402ba6ec	chore: update lockfiles (#1946 )	2024-12-18 08:43:33 -08:00
Lance Release	37c359ed40	Updating package-lock.json	2024-12-13 22:38:04 +00:00
Lance Release	06cdf00987	Bump version: 0.14.1-beta.4 → 0.14.1-beta.5	2024-12-13 22:37:41 +00:00