mirror of
https://github.com/lancedb/lancedb.git
synced 2025-12-23 13:29:57 +00:00
Compare commits
12 Commits
python-v0.
...
qian@saas-
| Author | SHA1 | Date | |
|---|---|---|---|
|
8e25e0c7f0 | ||
|
|
5f989e86d2 | ||
|
1336cce0dc | ||
|
6c83b6a513 | ||
|
|
6bec4bec51 | ||
|
|
23d30dfc78 | ||
|
94c8c50f96 | ||
|
|
72765d8e1a | ||
|
|
a2a8f9615e | ||
|
b085d9aaa1 | ||
|
6eb662de9b | ||
|
|
2bb2bb581a |
@@ -1,5 +1,5 @@
|
||||
[bumpversion]
|
||||
current_version = 0.3.8
|
||||
current_version = 0.3.9
|
||||
commit = True
|
||||
message = Bump version: {current_version} → {new_version}
|
||||
tag = True
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Builds the macOS artifacts (node binaries).
|
||||
# Usage: ./ci/build_macos_artifacts.sh [target]
|
||||
# Targets supported: x86_64-apple-darwin aarch64-apple-darwin
|
||||
set -e
|
||||
|
||||
prebuild_rust() {
|
||||
# Building here for the sake of easier debugging.
|
||||
|
||||
@@ -80,6 +80,7 @@ nav:
|
||||
- Ingest Embedding Functions: embeddings/embedding_functions.md
|
||||
- Available Functions: embeddings/default_embedding_functions.md
|
||||
- Create Custom Embedding Functions: embeddings/api.md
|
||||
- Example - Calculate CLIP Embeddings with Roboflow Inference: examples/image_embeddings_roboflow.md
|
||||
- Example - Multi-lingual semantic search: notebooks/multi_lingual_example.ipynb
|
||||
- Example - MultiModal CLIP Embeddings: notebooks/DisappearingEmbeddingFunction.ipynb
|
||||
- 🔍 Python full-text search: fts.md
|
||||
@@ -145,7 +146,8 @@ nav:
|
||||
- Serverless Chatbot from any website: examples/serverless_website_chatbot.md
|
||||
- TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md
|
||||
- API references:
|
||||
- Python API: python/python.md
|
||||
- OSS Python API: python/python.md
|
||||
- SaaS Python API: python/saas-python.md
|
||||
- Javascript API: javascript/modules.md
|
||||
- LanceDB Cloud↗: https://noteforms.com/forms/lancedb-mailing-list-cloud-kty1o5?notionforms=1&utm_source=notionforms
|
||||
|
||||
|
||||
165
docs/src/examples/image_embeddings_roboflow.md
Normal file
165
docs/src/examples/image_embeddings_roboflow.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# How to Load Image Embeddings into LanceDB
|
||||
|
||||
With the rise of Large Multimodal Models (LMMs) such as [GPT-4 Vision](https://blog.roboflow.com/gpt-4-vision/), the need for storing image embeddings is growing. The most effective way to store text and image embeddings is in a vector database such as LanceDB. Vector databases are a special kind of data store that enables efficient search over stored embeddings.
|
||||
|
||||
[CLIP](https://blog.roboflow.com/openai-clip/), a multimodal model developed by OpenAI, is commonly used to calculate image embeddings. These embeddings can then be used with a vector database to build a semantic search engine that you can query using images or text. For example, you could use LanceDB and CLIP embeddings to build a search engine for a database of folders.
|
||||
|
||||
In this guide, we are going to show you how to use Roboflow Inference to load image embeddings into LanceDB. Without further ado, let’s get started!
|
||||
|
||||
## Step #1: Install Roboflow Inference
|
||||
|
||||
[Roboflow Inference](https://inference.roboflow.com) enables you to run state-of-the-art computer vision models with minimal configuration. Inference supports a range of models, from fine-tuned object detection, classification, and segmentation models to foundation models like CLIP. We will use Inference to calculate CLIP image embeddings.
|
||||
|
||||
Inference provides a HTTP API through which you can run vision models.
|
||||
|
||||
Inference powers the Roboflow hosted API, and is available as an open source utility. In this guide, we are going to run Inference locally, which enables you to calculate CLIP embeddings on your own hardware. We will also show you how to use the hosted Roboflow CLIP API, which is ideal if you need to scale and do not want to manage a system for calculating embeddings.
|
||||
|
||||
To get started, first install the Inference CLI:
|
||||
|
||||
```
|
||||
pip install inference-cli
|
||||
```
|
||||
|
||||
Next, install Docker. Refer to the official Docker installation instructions for your operating system to get Docker set up. Once Docker is ready, you can start Inference using the following command:
|
||||
|
||||
```
|
||||
inference server start
|
||||
```
|
||||
|
||||
An Inference server will start running at ‘http://localhost:9001’.
|
||||
|
||||
## Step #2: Set Up a LanceDB Vector Database
|
||||
|
||||
Now that we have Inference running, we can set up a LanceDB vector database. You can run LanceDB in JavaScript and Python. For this guide, we will use the Python API. But, you can take the HTTP requests we make below and change them to JavaScript if required.
|
||||
|
||||
For this guide, we are going to search the [COCO 128 dataset](https://universe.roboflow.com/team-roboflow/coco-128), which contains a wide range of objects. The variability in objects present in this dataset makes it a good dataset to demonstrate the capabilities of vector search. If you want to use this dataset, you can download [COCO 128 from Roboflow Universe](https://universe.roboflow.com/team-roboflow/coco-128). With that said, you can search whatever folder of images you want.
|
||||
|
||||
Once you have a dataset ready, install LanceDB with the following command:
|
||||
|
||||
```
|
||||
pip install lancedb
|
||||
```
|
||||
|
||||
We also need to install a specific commit of `tantivy`, a dependency of the LanceDB full text search engine we will use later in this guide:
|
||||
|
||||
```
|
||||
pip install tantivy@git+https://github.com/quickwit-oss/tantivy-py#164adc87e1a033117001cf70e38c82a53014d985
|
||||
```
|
||||
|
||||
Create a new Python file and add the following code:
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
import requests
|
||||
|
||||
import lancedb
|
||||
|
||||
db = lancedb.connect("./embeddings")
|
||||
|
||||
IMAGE_DIR = "images/"
|
||||
API_KEY = os.environ.get("ROBOFLOW_API_KEY")
|
||||
SERVER_URL = "http://localhost:9001"
|
||||
|
||||
results = []
|
||||
|
||||
for i, image in enumerate(os.listdir(IMAGE_DIR)):
|
||||
infer_clip_payload = {
|
||||
#Images can be provided as urls or as base64 encoded strings
|
||||
"image": {
|
||||
"type": "base64",
|
||||
"value": base64.b64encode(open(IMAGE_DIR + image, "rb").read()).decode("utf-8"),
|
||||
},
|
||||
}
|
||||
|
||||
res = requests.post(
|
||||
f"{SERVER_URL}/clip/embed_image?api_key={API_KEY}",
|
||||
json=infer_clip_payload,
|
||||
)
|
||||
|
||||
embeddings = res.json()['embeddings']
|
||||
|
||||
print("Calculated embedding for image: ", image)
|
||||
|
||||
image = {"vector": embeddings[0], "name": os.path.join(IMAGE_DIR, image)}
|
||||
|
||||
results.append(image)
|
||||
|
||||
tbl = db.create_table("images", data=results)
|
||||
|
||||
tbl.create_fts_index("name")
|
||||
```
|
||||
|
||||
To use the code above, you will need a Roboflow API key. [Learn how to retrieve a Roboflow API key](https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key). Run the following command to set up your API key in your environment:
|
||||
|
||||
```
|
||||
export ROBOFLOW_API_KEY=""
|
||||
```
|
||||
|
||||
Replace the `IMAGE_DIR` value with the folder in which you are storing the images for which you want to calculate embeddings. If you want to use the Roboflow CLIP API to calculate embeddings, replace the `SERVER_URL` value with `https://infer.roboflow.com`.
|
||||
|
||||
Run the script above to create a new LanceDB database. This database will be stored on your local machine. The database will be called `embeddings` and the table will be called `images`.
|
||||
|
||||
The script above calculates all embeddings for a folder then creates a new table. To add additional images, use the following code:
|
||||
|
||||
```python
|
||||
def make_batches():
|
||||
for i in range(5):
|
||||
yield [
|
||||
{"vector": [3.1, 4.1], "name": "image1.png"},
|
||||
{"vector": [5.9, 26.5], "name": "image2.png"}
|
||||
]
|
||||
|
||||
tbl = db.open_table("images")
|
||||
tbl.add(make_batches())
|
||||
```
|
||||
|
||||
Replacing the `make_batches()` function with code to load embeddings for images.
|
||||
|
||||
## Step #3: Run a Search Query
|
||||
|
||||
We are now ready to run a search query. To run a search query, we need a text embedding that represents a text query. We can use this embedding to search our LanceDB database for an entry.
|
||||
|
||||
Let’s calculate a text embedding for the query “cat”, then run a search query:
|
||||
|
||||
```python
|
||||
infer_clip_payload = {
|
||||
"text": "cat",
|
||||
}
|
||||
|
||||
res = requests.post(
|
||||
f"{SERVER_URL}/clip/embed_text?api_key={API_KEY}",
|
||||
json=infer_clip_payload,
|
||||
)
|
||||
|
||||
embeddings = res.json()['embeddings']
|
||||
|
||||
df = tbl.search(embeddings[0]).limit(3).to_list()
|
||||
|
||||
print("Results:")
|
||||
|
||||
for i in df:
|
||||
print(i["name"])
|
||||
```
|
||||
|
||||
This code will search for the three images most closely related to the prompt “cat”. The names of the most similar three images will be printed to the console. Here are the three top results:
|
||||
|
||||
```
|
||||
dataset/images/train/000000000650_jpg.rf.1b74ba165c5a3513a3211d4a80b69e1c.jpg
|
||||
dataset/images/train/000000000138_jpg.rf.af439ef1c55dd8a4e4b142d186b9c957.jpg
|
||||
dataset/images/train/000000000165_jpg.rf.eae14d5509bf0c9ceccddbb53a5f0c66.jpg
|
||||
```
|
||||
|
||||
Let’s open the top image:
|
||||
|
||||
|
||||
|
||||
The top image was a cat. Our search was successful.
|
||||
|
||||
## Conclusion
|
||||
|
||||
LanceDB is a vector database that you can use to store and efficiently search your image embeddings. You can use Roboflow Inference, a scalable computer vision inference server, to calculate CLIP embeddings that you can store in LanceDB.
|
||||
|
||||
You can use Inference and LanceDB together to build a range of applications with image embeddings, from a media search engine to a retrieval-augmented generation pipeline for use with LMMs.
|
||||
|
||||
To learn more about Inference and its capabilities, refer to the Inference documentation.
|
||||
18
docs/src/python/saas-python.md
Normal file
18
docs/src/python/saas-python.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# LanceDB Python API Reference
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install lancedb
|
||||
```
|
||||
|
||||
## Connection
|
||||
|
||||
::: lancedb.connect
|
||||
|
||||
::: lancedb.remote.db.RemoteDBConnection
|
||||
|
||||
## Table
|
||||
|
||||
::: lancedb.remote.table.RemoteTable
|
||||
|
||||
86
node/package-lock.json
generated
86
node/package-lock.json
generated
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "vectordb",
|
||||
"version": "0.3.8",
|
||||
"version": "0.3.9",
|
||||
"lockfileVersion": 2,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "vectordb",
|
||||
"version": "0.3.8",
|
||||
"version": "0.3.9",
|
||||
"cpu": [
|
||||
"x64",
|
||||
"arm64"
|
||||
@@ -53,11 +53,11 @@
|
||||
"uuid": "^9.0.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@lancedb/vectordb-darwin-arm64": "0.3.8",
|
||||
"@lancedb/vectordb-darwin-x64": "0.3.8",
|
||||
"@lancedb/vectordb-linux-arm64-gnu": "0.3.8",
|
||||
"@lancedb/vectordb-linux-x64-gnu": "0.3.8",
|
||||
"@lancedb/vectordb-win32-x64-msvc": "0.3.8"
|
||||
"@lancedb/vectordb-darwin-arm64": "0.3.9",
|
||||
"@lancedb/vectordb-darwin-x64": "0.3.9",
|
||||
"@lancedb/vectordb-linux-arm64-gnu": "0.3.9",
|
||||
"@lancedb/vectordb-linux-x64-gnu": "0.3.9",
|
||||
"@lancedb/vectordb-win32-x64-msvc": "0.3.9"
|
||||
}
|
||||
},
|
||||
"node_modules/@apache-arrow/ts": {
|
||||
@@ -316,6 +316,54 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.4.10"
|
||||
}
|
||||
},
|
||||
"node_modules/@lancedb/vectordb-darwin-x64": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.3.9.tgz",
|
||||
"integrity": "sha512-4xXQoPheyIl1P5kRoKmZtaAHFrYdL9pw5yq+r6ewIx0TCemN4LSvzSUTqM5nZl3QPU8FeL0CGD8Gt2gMU0HQ2A==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@lancedb/vectordb-linux-arm64-gnu": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.3.9.tgz",
|
||||
"integrity": "sha512-WIxCZKnLeSlz0PGURtKSX6hJ4CYE2o5P+IFmmuWOWB1uNapQu6zOpea6rNxcRFHUA0IJdO02lVxVfn2hDX4SMg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@lancedb/vectordb-linux-x64-gnu": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.3.9.tgz",
|
||||
"integrity": "sha512-bQbcV9adKzYbJLNzDjk9OYsMnT2IjmieLfb4IQ1hj5IUoWfbg80Bd0+gZUnrmrhG6fe56TIriFZYQR9i7TSE9Q==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@lancedb/vectordb-win32-x64-msvc": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.3.9.tgz",
|
||||
"integrity": "sha512-7EXI7P1QvAfgJNPWWBMDOkoJ696gSBAClcyEJNYg0JV21jVFZRwJVI3bZXflesWduFi/mTuzPkFFA68us1u19A==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
]
|
||||
},
|
||||
"node_modules/@neon-rs/cli": {
|
||||
"version": "0.0.160",
|
||||
"resolved": "https://registry.npmjs.org/@neon-rs/cli/-/cli-0.0.160.tgz",
|
||||
@@ -4808,6 +4856,30 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.4.10"
|
||||
}
|
||||
},
|
||||
"@lancedb/vectordb-darwin-x64": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.3.9.tgz",
|
||||
"integrity": "sha512-4xXQoPheyIl1P5kRoKmZtaAHFrYdL9pw5yq+r6ewIx0TCemN4LSvzSUTqM5nZl3QPU8FeL0CGD8Gt2gMU0HQ2A==",
|
||||
"optional": true
|
||||
},
|
||||
"@lancedb/vectordb-linux-arm64-gnu": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.3.9.tgz",
|
||||
"integrity": "sha512-WIxCZKnLeSlz0PGURtKSX6hJ4CYE2o5P+IFmmuWOWB1uNapQu6zOpea6rNxcRFHUA0IJdO02lVxVfn2hDX4SMg==",
|
||||
"optional": true
|
||||
},
|
||||
"@lancedb/vectordb-linux-x64-gnu": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.3.9.tgz",
|
||||
"integrity": "sha512-bQbcV9adKzYbJLNzDjk9OYsMnT2IjmieLfb4IQ1hj5IUoWfbg80Bd0+gZUnrmrhG6fe56TIriFZYQR9i7TSE9Q==",
|
||||
"optional": true
|
||||
},
|
||||
"@lancedb/vectordb-win32-x64-msvc": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.3.9.tgz",
|
||||
"integrity": "sha512-7EXI7P1QvAfgJNPWWBMDOkoJ696gSBAClcyEJNYg0JV21jVFZRwJVI3bZXflesWduFi/mTuzPkFFA68us1u19A==",
|
||||
"optional": true
|
||||
},
|
||||
"@neon-rs/cli": {
|
||||
"version": "0.0.160",
|
||||
"resolved": "https://registry.npmjs.org/@neon-rs/cli/-/cli-0.0.160.tgz",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "vectordb",
|
||||
"version": "0.3.8",
|
||||
"version": "0.3.9",
|
||||
"description": " Serverless, low-latency vector database for AI applications",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
@@ -81,10 +81,10 @@
|
||||
}
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@lancedb/vectordb-darwin-arm64": "0.3.8",
|
||||
"@lancedb/vectordb-darwin-x64": "0.3.8",
|
||||
"@lancedb/vectordb-linux-arm64-gnu": "0.3.8",
|
||||
"@lancedb/vectordb-linux-x64-gnu": "0.3.8",
|
||||
"@lancedb/vectordb-win32-x64-msvc": "0.3.8"
|
||||
"@lancedb/vectordb-darwin-arm64": "0.3.9",
|
||||
"@lancedb/vectordb-darwin-x64": "0.3.9",
|
||||
"@lancedb/vectordb-linux-arm64-gnu": "0.3.9",
|
||||
"@lancedb/vectordb-linux-x64-gnu": "0.3.9",
|
||||
"@lancedb/vectordb-win32-x64-msvc": "0.3.9"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,7 @@ export class Query<T = number[]> {
|
||||
private _select?: string[]
|
||||
private _filter?: string
|
||||
private _metricType?: MetricType
|
||||
private _prefilter: boolean
|
||||
protected readonly _embeddings?: EmbeddingFunction<T>
|
||||
|
||||
constructor (query: T, tbl?: any, embeddings?: EmbeddingFunction<T>) {
|
||||
@@ -44,6 +45,7 @@ export class Query<T = number[]> {
|
||||
this._filter = undefined
|
||||
this._metricType = undefined
|
||||
this._embeddings = embeddings
|
||||
this._prefilter = false
|
||||
}
|
||||
|
||||
/***
|
||||
@@ -102,6 +104,11 @@ export class Query<T = number[]> {
|
||||
return this
|
||||
}
|
||||
|
||||
prefilter (value: boolean): Query<T> {
|
||||
this._prefilter = value
|
||||
return this
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute the query and return the results as an Array of Objects
|
||||
*/
|
||||
|
||||
@@ -38,6 +38,7 @@ export class HttpLancedbClient {
|
||||
vector: number[],
|
||||
k: number,
|
||||
nprobes: number,
|
||||
prefilter: boolean,
|
||||
refineFactor?: number,
|
||||
columns?: string[],
|
||||
filter?: string
|
||||
@@ -50,7 +51,8 @@ export class HttpLancedbClient {
|
||||
nprobes,
|
||||
refineFactor,
|
||||
columns,
|
||||
filter
|
||||
filter,
|
||||
prefilter
|
||||
},
|
||||
{
|
||||
headers: {
|
||||
|
||||
@@ -154,6 +154,7 @@ export class RemoteQuery<T = number[]> extends Query<T> {
|
||||
queryVector,
|
||||
(this as any)._limit,
|
||||
(this as any)._nprobes,
|
||||
(this as any)._prefilter,
|
||||
(this as any)._refineFactor,
|
||||
(this as any)._select,
|
||||
(this as any)._filter
|
||||
|
||||
@@ -102,6 +102,20 @@ describe('LanceDB client', function () {
|
||||
assertResults(results)
|
||||
})
|
||||
|
||||
it('should correctly process prefilter/postfilter', async function () {
|
||||
const uri = await createTestDB(16, 300)
|
||||
const con = await lancedb.connect(uri)
|
||||
const table = await con.openTable('vectors')
|
||||
await table.createIndex({ type: 'ivf_pq', column: 'vector', num_partitions: 2, max_iters: 2, num_sub_vectors: 2 })
|
||||
// post filter should return less than the limit
|
||||
let results = await table.search(new Array(16).fill(0.1)).limit(10).filter('id >= 10').prefilter(false).execute()
|
||||
assert.isTrue(results.length < 10)
|
||||
|
||||
// pre filter should return exactly the limit
|
||||
results = await table.search(new Array(16).fill(0.1)).limit(10).filter('id >= 10').prefilter(true).execute()
|
||||
assert.isTrue(results.length === 10)
|
||||
})
|
||||
|
||||
it('select only a subset of columns', async function () {
|
||||
const uri = await createTestDB()
|
||||
const con = await lancedb.connect(uri)
|
||||
|
||||
@@ -27,7 +27,7 @@ def connect(
|
||||
uri: URI,
|
||||
*,
|
||||
api_key: Optional[str] = None,
|
||||
region: str = "us-west-2",
|
||||
region: str = "us-east-1",
|
||||
host_override: Optional[str] = None,
|
||||
) -> DBConnection:
|
||||
"""Connect to a LanceDB database.
|
||||
@@ -39,7 +39,7 @@ def connect(
|
||||
api_key: str, optional
|
||||
If presented, connect to LanceDB cloud.
|
||||
Otherwise, connect to a database on file system or cloud storage.
|
||||
region: str, default "us-west-2"
|
||||
region: str, default "us-east-1"
|
||||
The region to use for LanceDB Cloud.
|
||||
host_override: str, optional
|
||||
The override url for LanceDB Cloud.
|
||||
|
||||
@@ -28,6 +28,7 @@ from ..pydantic import LanceModel
|
||||
from ..table import Table, _sanitize_data
|
||||
from .arrow import to_ipc_binary
|
||||
from .client import ARROW_STREAM_CONTENT_TYPE, RestfulLanceDBClient
|
||||
from .errors import LanceDBClientError
|
||||
|
||||
|
||||
class RemoteDBConnection(DBConnection):
|
||||
@@ -58,13 +59,17 @@ class RemoteDBConnection(DBConnection):
|
||||
return f"RemoveConnect(name={self.db_name})"
|
||||
|
||||
@override
|
||||
def table_names(self, page_token: Optional[str] = None, limit=10) -> Iterable[str]:
|
||||
def table_names(
|
||||
self, page_token: Optional[str] = None, limit: int = 10
|
||||
) -> Iterable[str]:
|
||||
"""List the names of all tables in the database.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
page_token: str
|
||||
The last token to start the new page.
|
||||
limit: int, default 10
|
||||
The maximum number of tables to return for each page.
|
||||
|
||||
Returns
|
||||
-------
|
||||
@@ -101,11 +106,12 @@ class RemoteDBConnection(DBConnection):
|
||||
self._loop.run_until_complete(
|
||||
self._client.post(f"/v1/table/{name}/describe/")
|
||||
)
|
||||
except Exception:
|
||||
logging.error(
|
||||
"Table {name} does not exist."
|
||||
"Please first call db.create_table({name}, data)"
|
||||
)
|
||||
except LanceDBClientError as err:
|
||||
if str(err).startswith("Not found"):
|
||||
logging.error(
|
||||
f"Table {name} does not exist. "
|
||||
f"Please first call db.create_table({name}, data)"
|
||||
)
|
||||
return RemoteTable(self, name)
|
||||
|
||||
@override
|
||||
@@ -118,6 +124,97 @@ class RemoteDBConnection(DBConnection):
|
||||
fill_value: float = 0.0,
|
||||
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
|
||||
) -> Table:
|
||||
"""Create a [Table][lancedb.table.Table] in the database.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
name: str
|
||||
The name of the table.
|
||||
data: The data to initialize the table, *optional*
|
||||
User must provide at least one of `data` or `schema`.
|
||||
Acceptable types are:
|
||||
|
||||
- dict or list-of-dict
|
||||
|
||||
- pandas.DataFrame
|
||||
|
||||
- pyarrow.Table or pyarrow.RecordBatch
|
||||
schema: The schema of the table, *optional*
|
||||
Acceptable types are:
|
||||
|
||||
- pyarrow.Schema
|
||||
|
||||
- [LanceModel][lancedb.pydantic.LanceModel]
|
||||
on_bad_vectors: str, default "error"
|
||||
What to do if any of the vectors are not the same size or contains NaNs.
|
||||
One of "error", "drop", "fill".
|
||||
fill_value: float
|
||||
The value to use when filling vectors. Only used if on_bad_vectors="fill".
|
||||
|
||||
Returns
|
||||
-------
|
||||
LanceTable
|
||||
A reference to the newly created table.
|
||||
|
||||
!!! note
|
||||
|
||||
The vector index won't be created by default.
|
||||
To create the index, call the `create_index` method on the table.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Can create with list of tuples or dictionaries:
|
||||
|
||||
>>> import lancedb
|
||||
>>> db = lancedb.connect("db://test-project-8f45eb")
|
||||
>>> data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
|
||||
... {"vector": [0.2, 1.8], "lat": 40.1, "long": -74.1}]
|
||||
>>> db.create_table("my_table", data)
|
||||
LanceTable(my_table)
|
||||
|
||||
You can also pass a pandas DataFrame:
|
||||
|
||||
>>> import pandas as pd
|
||||
>>> data = pd.DataFrame({
|
||||
... "vector": [[1.1, 1.2], [0.2, 1.8]],
|
||||
... "lat": [45.5, 40.1],
|
||||
... "long": [-122.7, -74.1]
|
||||
... })
|
||||
>>> db.create_table("table2", data)
|
||||
LanceTable(table2)
|
||||
|
||||
>>> custom_schema = pa.schema([
|
||||
... pa.field("vector", pa.list_(pa.float32(), 2)),
|
||||
... pa.field("lat", pa.float32()),
|
||||
... pa.field("long", pa.float32())
|
||||
... ])
|
||||
>>> db.create_table("table3", data, schema = custom_schema)
|
||||
LanceTable(table3)
|
||||
|
||||
It is also possible to create an table from `[Iterable[pa.RecordBatch]]`:
|
||||
|
||||
>>> import pyarrow as pa
|
||||
>>> def make_batches():
|
||||
... for i in range(5):
|
||||
... yield pa.RecordBatch.from_arrays(
|
||||
... [
|
||||
... pa.array([[3.1, 4.1], [5.9, 26.5]],
|
||||
... pa.list_(pa.float32(), 2)),
|
||||
... pa.array(["foo", "bar"]),
|
||||
... pa.array([10.0, 20.0]),
|
||||
... ],
|
||||
... ["vector", "item", "price"],
|
||||
... )
|
||||
>>> schema=pa.schema([
|
||||
... pa.field("vector", pa.list_(pa.float32(), 2)),
|
||||
... pa.field("item", pa.utf8()),
|
||||
... pa.field("price", pa.float32()),
|
||||
... ])
|
||||
>>> db.create_table("table4", make_batches(), schema=schema)
|
||||
LanceTable(table4)
|
||||
|
||||
"""
|
||||
if data is None and schema is None:
|
||||
raise ValueError("Either data or schema must be provided.")
|
||||
if embedding_functions is not None:
|
||||
|
||||
@@ -37,7 +37,10 @@ class RemoteTable(Table):
|
||||
|
||||
@cached_property
|
||||
def schema(self) -> pa.Schema:
|
||||
"""Return the schema of the table."""
|
||||
"""The [Arrow Schema](https://arrow.apache.org/docs/python/api/datatypes.html#)
|
||||
of this Table
|
||||
|
||||
"""
|
||||
resp = self._conn._loop.run_until_complete(
|
||||
self._conn._client.post(f"/v1/table/{self._name}/describe/")
|
||||
)
|
||||
@@ -53,24 +56,17 @@ class RemoteTable(Table):
|
||||
return resp["version"]
|
||||
|
||||
def to_arrow(self) -> pa.Table:
|
||||
"""Return the table as an Arrow table."""
|
||||
"""to_arrow() is not supported on the LanceDB cloud"""
|
||||
raise NotImplementedError("to_arrow() is not supported on the LanceDB cloud")
|
||||
|
||||
def to_pandas(self):
|
||||
"""Return the table as a Pandas DataFrame.
|
||||
|
||||
Intercept `to_arrow()` for better error message.
|
||||
"""
|
||||
"""to_pandas() is not supported on the LanceDB cloud"""
|
||||
return NotImplementedError("to_pandas() is not supported on the LanceDB cloud")
|
||||
|
||||
def create_index(
|
||||
self,
|
||||
metric="L2",
|
||||
num_partitions=256,
|
||||
num_sub_vectors=96,
|
||||
vector_column_name: str = VECTOR_COLUMN_NAME,
|
||||
replace: bool = True,
|
||||
accelerator: Optional[str] = None,
|
||||
index_cache_size: Optional[int] = None,
|
||||
):
|
||||
"""Create an index on the table.
|
||||
@@ -81,39 +77,28 @@ class RemoteTable(Table):
|
||||
----------
|
||||
metric : str
|
||||
The metric to use for the index. Default is "L2".
|
||||
num_partitions : int
|
||||
The number of partitions to use for the index. Default is 256.
|
||||
num_sub_vectors : int
|
||||
The number of sub-vectors to use for the index. Default is 96.
|
||||
vector_column_name : str
|
||||
The name of the vector column. Default is "vector".
|
||||
replace : bool
|
||||
Whether to replace the existing index. Default is True.
|
||||
accelerator : str, optional
|
||||
If set, use the given accelerator to create the index.
|
||||
Default is None. Currently not supported.
|
||||
index_cache_size : int, optional
|
||||
The size of the index cache in number of entries. Default value is 256.
|
||||
|
||||
Examples
|
||||
--------
|
||||
import lancedb
|
||||
import uuid
|
||||
from lancedb.schema import vector
|
||||
conn = lancedb.connect("db://...", api_key="...", region="...")
|
||||
table_name = uuid.uuid4().hex
|
||||
schema = pa.schema(
|
||||
[
|
||||
pa.field("id", pa.uint32(), False),
|
||||
pa.field("vector", vector(128), False),
|
||||
pa.field("s", pa.string(), False),
|
||||
]
|
||||
)
|
||||
table = conn.create_table(
|
||||
table_name,
|
||||
schema=schema,
|
||||
)
|
||||
table.create_index()
|
||||
>>> import lancedb
|
||||
>>> import uuid
|
||||
>>> from lancedb.schema import vector
|
||||
>>> conn = lancedb.connect("db://...", api_key="...", region="...")
|
||||
>>> table_name = uuid.uuid4().hex
|
||||
>>> schema = pa.schema(
|
||||
... [
|
||||
... pa.field("id", pa.uint32(), False),
|
||||
... pa.field("vector", vector(128), False),
|
||||
... pa.field("s", pa.string(), False),
|
||||
... ]
|
||||
... )
|
||||
>>> table = conn.create_table(
|
||||
>>> table_name,
|
||||
>>> schema=schema,
|
||||
>>> )
|
||||
>>> table.create_index("L2", "vector")
|
||||
"""
|
||||
index_type = "vector"
|
||||
|
||||
@@ -135,6 +120,28 @@ class RemoteTable(Table):
|
||||
on_bad_vectors: str = "error",
|
||||
fill_value: float = 0.0,
|
||||
) -> int:
|
||||
"""Add more data to the [Table](Table). It has the same API signature as the OSS version.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
data: DATA
|
||||
The data to insert into the table. Acceptable types are:
|
||||
|
||||
- dict or list-of-dict
|
||||
|
||||
- pandas.DataFrame
|
||||
|
||||
- pyarrow.Table or pyarrow.RecordBatch
|
||||
mode: str
|
||||
The mode to use when writing the data. Valid values are
|
||||
"append" and "overwrite".
|
||||
on_bad_vectors: str, default "error"
|
||||
What to do if any of the vectors are not the same size or contains NaNs.
|
||||
One of "error", "drop", "fill".
|
||||
fill_value: float, default 0.
|
||||
The value to use when filling vectors. Only used if on_bad_vectors="fill".
|
||||
|
||||
"""
|
||||
data = _sanitize_data(
|
||||
data,
|
||||
self.schema,
|
||||
@@ -158,6 +165,58 @@ class RemoteTable(Table):
|
||||
def search(
|
||||
self, query: Union[VEC, str], vector_column_name: str = VECTOR_COLUMN_NAME
|
||||
) -> LanceVectorQueryBuilder:
|
||||
"""Create a search query to find the nearest neighbors
|
||||
of the given query vector. We currently support [vector search][search]
|
||||
|
||||
All query options are defined in [Query][lancedb.query.Query].
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> import lancedb
|
||||
>>> db = lancedb.connect("db://...", api_key="...", region="...")
|
||||
>>> data = [
|
||||
... {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]},
|
||||
... {"original_width": 2000, "caption": "foo", "vector": [0.5, 3.4, 1.3]},
|
||||
... {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]}
|
||||
... ]
|
||||
>>> table = db.create_table("my_table", data)
|
||||
>>> query = [0.4, 1.4, 2.4]
|
||||
>>> (table.search(query, vector_column_name="vector")
|
||||
... .where("original_width > 1000", prefilter=True)
|
||||
... .select(["caption", "original_width"])
|
||||
... .limit(2)
|
||||
... .to_pandas())
|
||||
caption original_width vector _distance
|
||||
0 foo 2000 [0.5, 3.4, 1.3] 5.220000
|
||||
1 test 3000 [0.3, 6.2, 2.6] 23.089996
|
||||
|
||||
Parameters
|
||||
----------
|
||||
query: list/np.ndarray/str/PIL.Image.Image, default None
|
||||
The targetted vector to search for.
|
||||
|
||||
- *default None*.
|
||||
Acceptable types are: list, np.ndarray, PIL.Image.Image
|
||||
|
||||
- If None then the select/where/limit clauses are applied to filter
|
||||
the table
|
||||
vector_column_name: str
|
||||
The name of the vector column to search.
|
||||
*default "vector"*
|
||||
|
||||
Returns
|
||||
-------
|
||||
LanceQueryBuilder
|
||||
A query builder object representing the query.
|
||||
Once executed, the query returns
|
||||
|
||||
- selected columns
|
||||
|
||||
- the vector
|
||||
|
||||
- and also the "_distance" column which is the distance between the query
|
||||
vector and the returned vector.
|
||||
"""
|
||||
return LanceVectorQueryBuilder(self, query, vector_column_name)
|
||||
|
||||
def _execute_query(self, query: Query) -> pa.Table:
|
||||
@@ -165,7 +224,53 @@ class RemoteTable(Table):
|
||||
return self._conn._loop.run_until_complete(result).to_arrow()
|
||||
|
||||
def delete(self, predicate: str):
|
||||
"""Delete rows from the table."""
|
||||
"""Delete rows from the table.
|
||||
|
||||
This can be used to delete a single row, many rows, all rows, or
|
||||
sometimes no rows (if your predicate matches nothing).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
predicate: str
|
||||
The SQL where clause to use when deleting rows.
|
||||
|
||||
- For example, 'x = 2' or 'x IN (1, 2, 3)'.
|
||||
|
||||
The filter must not be empty, or it will error.
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> import lancedb
|
||||
>>> data = [
|
||||
... {"x": 1, "vector": [1, 2]},
|
||||
... {"x": 2, "vector": [3, 4]},
|
||||
... {"x": 3, "vector": [5, 6]}
|
||||
... ]
|
||||
>>> db = lancedb.connect("db://...", api_key="...", region="...")
|
||||
>>> table = db.create_table("my_table", data)
|
||||
>>> table.search([10,10]).to_pandas()
|
||||
x vector _distance
|
||||
0 3 [5.0, 6.0] 41.0
|
||||
1 2 [3.0, 4.0] 85.0
|
||||
2 1 [1.0, 2.0] 145.0
|
||||
>>> table.delete("x = 2")
|
||||
>>> table.search([10,10]).to_pandas()
|
||||
x vector _distance
|
||||
0 3 [5.0, 6.0] 41.0
|
||||
1 1 [1.0, 2.0] 145.0
|
||||
|
||||
If you have a list of values to delete, you can combine them into a
|
||||
stringified list and use the `IN` operator:
|
||||
|
||||
>>> to_remove = [1, 3]
|
||||
>>> to_remove = ", ".join([str(v) for v in to_remove])
|
||||
>>> to_remove
|
||||
'1, 3'
|
||||
>>> table.delete(f"x IN ({to_remove})")
|
||||
>>> table.search([10,10]).to_pandas()
|
||||
x vector _distance
|
||||
0 2 [3.0, 4.0] 85.0
|
||||
"""
|
||||
payload = {"predicate": predicate}
|
||||
self._conn._loop.run_until_complete(
|
||||
self._conn._client.post(f"/v1/table/{self._name}/delete/", data=payload)
|
||||
|
||||
@@ -26,6 +26,9 @@ class FakeLanceDBClient:
|
||||
t = pa.schema([]).empty_table()
|
||||
return VectorQueryResult(t)
|
||||
|
||||
async def post(self, path: str):
|
||||
pass
|
||||
|
||||
|
||||
def test_remote_db():
|
||||
conn = lancedb.connect("db://client-will-be-injected", api_key="fake")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "vectordb-node"
|
||||
version = "0.3.8"
|
||||
version = "0.3.9"
|
||||
description = "Serverless, low-latency vector database for AI applications"
|
||||
license = "Apache-2.0"
|
||||
edition = "2018"
|
||||
|
||||
@@ -48,6 +48,8 @@ impl JsQuery {
|
||||
.map(|s| s.value(&mut cx))
|
||||
.map(|s| MetricType::try_from(s.as_str()).unwrap());
|
||||
|
||||
let prefilter = query_obj.get::<JsBoolean, _, _>(&mut cx, "_prefilter")?.value(&mut cx);
|
||||
|
||||
let is_electron = cx
|
||||
.argument::<JsBoolean>(1)
|
||||
.or_throw(&mut cx)?
|
||||
@@ -69,7 +71,8 @@ impl JsQuery {
|
||||
.nprobes(nprobes)
|
||||
.filter(filter)
|
||||
.metric_type(metric_type)
|
||||
.select(select);
|
||||
.select(select)
|
||||
.prefilter(prefilter);
|
||||
let record_batch_stream = builder.execute();
|
||||
let results = record_batch_stream
|
||||
.and_then(|stream| {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "vectordb"
|
||||
version = "0.3.8"
|
||||
version = "0.3.9"
|
||||
edition = "2021"
|
||||
description = "LanceDB: A serverless, low-latency vector database for AI applications"
|
||||
license = "Apache-2.0"
|
||||
|
||||
@@ -32,6 +32,7 @@ pub struct Query {
|
||||
pub refine_factor: Option<u32>,
|
||||
pub metric_type: Option<MetricType>,
|
||||
pub use_index: bool,
|
||||
pub prefilter: bool,
|
||||
}
|
||||
|
||||
impl Query {
|
||||
@@ -56,6 +57,7 @@ impl Query {
|
||||
use_index: true,
|
||||
filter: None,
|
||||
select: None,
|
||||
prefilter: false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,6 +76,8 @@ impl Query {
|
||||
)?;
|
||||
scanner.nprobs(self.nprobes);
|
||||
scanner.use_index(self.use_index);
|
||||
scanner.prefilter(self.prefilter);
|
||||
|
||||
self.select.as_ref().map(|p| scanner.project(p.as_slice()));
|
||||
self.filter.as_ref().map(|f| scanner.filter(f));
|
||||
self.refine_factor.map(|rf| scanner.refine(rf));
|
||||
@@ -158,6 +162,11 @@ impl Query {
|
||||
self.select = columns;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn prefilter(mut self, prefilter: bool) -> Query {
|
||||
self.prefilter = prefilter;
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -167,7 +176,9 @@ mod tests {
|
||||
use super::*;
|
||||
use arrow_array::{Float32Array, RecordBatch, RecordBatchIterator, RecordBatchReader};
|
||||
use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema};
|
||||
use futures::StreamExt;
|
||||
use lance::dataset::Dataset;
|
||||
use lance_testing::datagen::{BatchGenerator, IncrementingInt32, RandomVector};
|
||||
|
||||
use crate::query::Query;
|
||||
|
||||
@@ -200,13 +211,43 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_execute() {
|
||||
let batches = make_test_batches();
|
||||
let ds = Dataset::write(batches, "memory://foo", None).await.unwrap();
|
||||
let batches = make_non_empty_batches();
|
||||
let ds = Arc::new(Dataset::write(batches, "memory://foo", None).await.unwrap());
|
||||
|
||||
let vector = Float32Array::from_iter_values([0.1; 128]);
|
||||
let query = Query::new(Arc::new(ds), vector.clone());
|
||||
let result = query.execute().await;
|
||||
assert_eq!(result.is_ok(), true);
|
||||
let vector = Float32Array::from_iter_values([0.1; 4]);
|
||||
|
||||
let query = Query::new(ds.clone(), vector.clone());
|
||||
let result = query
|
||||
.limit(10)
|
||||
.filter(Some("id % 2 == 0".to_string()))
|
||||
.execute()
|
||||
.await;
|
||||
let mut stream = result.expect("should have result");
|
||||
// should only have one batch
|
||||
while let Some(batch) = stream.next().await {
|
||||
// post filter should have removed some rows
|
||||
assert!(batch.expect("should be Ok").num_rows() < 10);
|
||||
}
|
||||
|
||||
let query = Query::new(ds, vector.clone());
|
||||
let result = query
|
||||
.limit(10)
|
||||
.filter(Some("id % 2 == 0".to_string()))
|
||||
.prefilter(true)
|
||||
.execute()
|
||||
.await;
|
||||
let mut stream = result.expect("should have result");
|
||||
// should only have one batch
|
||||
while let Some(batch) = stream.next().await {
|
||||
// pre filter should return 10 rows
|
||||
assert!(batch.expect("should be Ok").num_rows() == 10);
|
||||
}
|
||||
}
|
||||
|
||||
fn make_non_empty_batches() -> impl RecordBatchReader + Send + 'static {
|
||||
let vec = Box::new(RandomVector::new().named("vector".to_string()));
|
||||
let id = Box::new(IncrementingInt32::new().named("id".to_string()));
|
||||
BatchGenerator::new().col(vec).col(id).batch(512)
|
||||
}
|
||||
|
||||
fn make_test_batches() -> impl RecordBatchReader + Send + 'static {
|
||||
|
||||
Reference in New Issue
Block a user