From 32b21e1d2099736224be9f9afe69c31bff89e983 Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 17:20:58 -0700 Subject: [PATCH 1/7] wip ann indexes doc --- docs/src/ann_indexes.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/src/ann_indexes.md diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md new file mode 100644 index 000000000..2dbc1884c --- /dev/null +++ b/docs/src/ann_indexes.md @@ -0,0 +1,37 @@ +# ANN (Approximate Nearest Neighbor) Indexes + +In order to make vector search faster, you can create an index over your vector data. Vector indexes are faster but +not as accurate as exhaustive search. LanceDB provide many parameters to fine tune the size of the index, the speed +of queries and the accuracy for results + +## Creating an ANN Index + +This is how you can create a vector index: + +```python +import lancedb +import numpy as np +uri = "~/.lancedb" +db = lancedb.connect(uri) + +# Create 10,000 sample vectors +data = [{"vector": row, "item": f"item {i}"} + for i, row in enumerate(np.random.random((10_000, 768)).astype('float32'))] + +# Add the vectors to a table +tbl = db.create_table("my_vectors", data=data) + +# Create and train the index - you need to have enough data in the table for an effective training step +tbl.create_index(num_partitions=256,num_sub_vectors=96) +``` + +Since `create_index` has a training, it can take a couple of minutes to finish for large tables. You can control index +creation by providing the following parameters: +- **num_partitions**: The number of partitions of the index. A higher number leads to better search quality, but it makes index +generation slower. +- **num_partitions**: The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes +search more accurate, but also makes the index larger and slower to build. + +## Querying an ANN Index + +TODO \ No newline at end of file From 4f7f33f7b7952aa316b827141bcd35565defc8b5 Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 17:23:06 -0700 Subject: [PATCH 2/7] wip ann indexes doc --- docs/src/ann_indexes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index 2dbc1884c..2de6afd2c 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -29,7 +29,7 @@ Since `create_index` has a training, it can take a couple of minutes to finish f creation by providing the following parameters: - **num_partitions**: The number of partitions of the index. A higher number leads to better search quality, but it makes index generation slower. -- **num_partitions**: The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes +- **num_sub_vectors**: The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes search more accurate, but also makes the index larger and slower to build. ## Querying an ANN Index From 8bf4d169e229b1e331d9f8985c5e0c98636f9f95 Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 17:23:31 -0700 Subject: [PATCH 3/7] wip ann indexes doc --- docs/src/ann_indexes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index 2de6afd2c..c20faa46d 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -2,7 +2,7 @@ In order to make vector search faster, you can create an index over your vector data. Vector indexes are faster but not as accurate as exhaustive search. LanceDB provide many parameters to fine tune the size of the index, the speed -of queries and the accuracy for results +of queries and the accuracy of results. ## Creating an ANN Index From ce34d055af0414239f27949396e81af88370e76e Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 19:26:27 -0700 Subject: [PATCH 4/7] search docs --- docs/src/ann_indexes.md | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index c20faa46d..e36cd0707 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -1,12 +1,12 @@ # ANN (Approximate Nearest Neighbor) Indexes -In order to make vector search faster, you can create an index over your vector data. Vector indexes are faster but -not as accurate as exhaustive search. LanceDB provide many parameters to fine tune the size of the index, the speed -of queries and the accuracy of results. +You can create an index over your vector data to make search faster. Vector indexes are faster but less + accurate than exhaustive search. LanceDB provides many parameters to fine-tune the index's size, the speed of +queries, and the accuracy of results. ## Creating an ANN Index -This is how you can create a vector index: +Creating indexes is done via the [create_index](https://lancedb.github.io/lancedb/python/#lancedb.table.LanceTable.create_index) function. ```python import lancedb @@ -25,13 +25,35 @@ tbl = db.create_table("my_vectors", data=data) tbl.create_index(num_partitions=256,num_sub_vectors=96) ``` -Since `create_index` has a training, it can take a couple of minutes to finish for large tables. You can control index +Since `create_index` has a training step, it can take a few minutes to finish for large tables. You can control the index creation by providing the following parameters: -- **num_partitions**: The number of partitions of the index. A higher number leads to better search quality, but it makes index + +- **num_partitions**: The number of partitions of the index. A higher number leads to faster queries, but it makes index generation slower. - **num_sub_vectors**: The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes search more accurate, but also makes the index larger and slower to build. ## Querying an ANN Index -TODO \ No newline at end of file +Querying vector indexes is done via the [search](https://lancedb.github.io/lancedb/python/#lancedb.table.LanceTable.search) function. + +There are a couple of parameters that can be used to fine-tune the search: + +- **limit**: The amount of results that will be returned +- **nprobes**: The number of probes used. A higher number makes search more accurate but also slower. +- **refine_factor**: Refine the results by reading extra elements and re-ranking them in memory. A higher number makes +search more accurate but also slower. + +```python +tbl.search(np.random.random((768))) \ + .limit(2) \ + .nprobes(20) \ + .refine_factor(20) \ + .to_df() + + vector item score +0 [0.44949695, 0.8444449, 0.06281311, 0.23338133... item 1141 103.575333 +1 [0.48587373, 0.269207, 0.15095535, 0.65531915,... item 3953 108.393867 +``` + +The search will return the data requested in addition to the score of each item. The score is the distance between the query vector and the element. A lower number means that the result is more relevant. From b19ce1018494da1528c33f245dfa3e4b7eb289e9 Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 19:28:40 -0700 Subject: [PATCH 5/7] review comments --- docs/src/ann_indexes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index e36cd0707..e9fd5682a 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -22,7 +22,7 @@ data = [{"vector": row, "item": f"item {i}"} tbl = db.create_table("my_vectors", data=data) # Create and train the index - you need to have enough data in the table for an effective training step -tbl.create_index(num_partitions=256,num_sub_vectors=96) +tbl.create_index(num_partitions=256, num_sub_vectors=96) ``` Since `create_index` has a training step, it can take a few minutes to finish for large tables. You can control the index From 3eac75e61ae5e7ce40d652a0189c5365508d758b Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 20:23:18 -0700 Subject: [PATCH 6/7] review comments --- docs/src/ann_indexes.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index e9fd5682a..6d81d33ef 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -1,8 +1,8 @@ # ANN (Approximate Nearest Neighbor) Indexes -You can create an index over your vector data to make search faster. Vector indexes are faster but less - accurate than exhaustive search. LanceDB provides many parameters to fine-tune the index's size, the speed of -queries, and the accuracy of results. +You can create an index over your vector data to make search faster. Vector indexes are faster but less accurate than exhaustive search. LanceDB provides many parameters to fine-tune the index's size, the speed of queries, and the accuracy of results. + +Currently, LanceDB does not automatically create the ANN index. In the future we will look to improve this experience and automate index creation and configuration. ## Creating an ANN Index @@ -28,9 +28,10 @@ tbl.create_index(num_partitions=256, num_sub_vectors=96) Since `create_index` has a training step, it can take a few minutes to finish for large tables. You can control the index creation by providing the following parameters: -- **num_partitions**: The number of partitions of the index. A higher number leads to faster queries, but it makes index -generation slower. -- **num_sub_vectors**: The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes +- **num_partitions** (default: 256): The number of partitions of the index. The number of partitions should be configured so each partition has 3-5K vectors. For example, a table +with ~1M vectors should use 256 partitions. You can specify arbitrary number of partitions but powers of 2 is most conventional. +A higher number leads to faster queries, but it makes index generation slower. +- **num_sub_vectors** (default: 96): The number of subvectors (M) that will be created during Product Quantization (PQ). A larger number makes search more accurate, but also makes the index larger and slower to build. ## Querying an ANN Index @@ -39,9 +40,9 @@ Querying vector indexes is done via the [search](https://lancedb.github.io/lance There are a couple of parameters that can be used to fine-tune the search: -- **limit**: The amount of results that will be returned -- **nprobes**: The number of probes used. A higher number makes search more accurate but also slower. -- **refine_factor**: Refine the results by reading extra elements and re-ranking them in memory. A higher number makes +- **limit** (default: 10): The amount of results that will be returned +- **nprobes** (default: 20): The number of probes used. A higher number makes search more accurate but also slower. +- **refine_factor** (default: None): Refine the results by reading extra elements and re-ranking them in memory. A higher number makes search more accurate but also slower. ```python From 3e79b4d9cb8575bca7ad4fee7c5e1cec6b0844a3 Mon Sep 17 00:00:00 2001 From: gsilvestrin Date: Wed, 19 Apr 2023 20:33:23 -0700 Subject: [PATCH 7/7] review comments --- docs/src/ann_indexes.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/src/ann_indexes.md b/docs/src/ann_indexes.md index 6d81d33ef..96b93e2b8 100644 --- a/docs/src/ann_indexes.md +++ b/docs/src/ann_indexes.md @@ -58,3 +58,23 @@ tbl.search(np.random.random((768))) \ ``` The search will return the data requested in addition to the score of each item. The score is the distance between the query vector and the element. A lower number means that the result is more relevant. + +### Filtering (where clause) + +You can further filter the elements returned by a search using a where clause. + +```python +tbl.search(np.random.random((768))).where("item != 'item 1141'").to_df() +``` + +### Projections (select clause) + +You can select the columns returned by the query using a select clause. + +```python +tbl.search(np.random.random((768))).select(["vector"]).to_df() + vector score +0 [0.30928212, 0.022668175, 0.1756372, 0.4911822... 93.971092 +1 [0.2525465, 0.01723831, 0.261568, 0.002007689,... 95.173485 +... +```