rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-01-09 05:12:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

js docs, modal example, doc notebook integration, update doc styles (#131)

Browse Source
This commit is contained in:
Jai
2023-06-02 15:24:16 -07:00
committed by GitHub
parent fbd0bc7740
commit 8af5f19cc1
29 changed files with 1780 additions and 143 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
docs/src/ann_indexes.md
View File

@@ -12,24 +12,38 @@ In the future we will look to automatically create and configure the ANN index.
## Creating an ANN Index
Creating indexes is done via the [create_index](https://lancedb.github.io/lancedb/python/#lancedb.table.LanceTable.create_index) method.
=== "Python"
Creating indexes is done via the [create_index](https://lancedb.github.io/lancedb/python/#lancedb.table.LanceTable.create_index) method.
```python
import lancedb
import numpy as np
uri = "~/.lancedb"
db = lancedb.connect(uri)
```python
import lancedb
import numpy as np
uri = "~/.lancedb"
db = lancedb.connect(uri)
# Create 10,000 sample vectors
data = [{"vector": row, "item": f"item {i}"}
for i, row in enumerate(np.random.random((10_000, 768)).astype('float32'))]
# Create 10,000 sample vectors
data = [{"vector": row, "item": f"item {i}"}
for i, row in enumerate(np.random.random((10_000, 768)).astype('float32'))]
# Add the vectors to a table
tbl = db.create_table("my_vectors", data=data)
# Add the vectors to a table
tbl = db.create_table("my_vectors", data=data)
# Create and train the index - you need to have enough data in the table for an effective training step
tbl.create_index(num_partitions=256, num_sub_vectors=96)
```
# Create and train the index - you need to have enough data in the table for an effective training step
tbl.create_index(num_partitions=256, num_sub_vectors=96)
```
=== "Javascript"
```javascript
const vectordb = require('vectordb')
const db = await vectordb.connect('data/sample-lancedb')
let data = []
for (let i = 0; i < 10_000; i++) {
data.push({vector: Array(1536).fill(i), id: `${i}`, content: "", longId: `${i}`},)
}
const table = await db.createTable('vectors', data)
await table.create_index({ type: 'ivf_pq', column: 'vector', num_partitions: 256, num_sub_vectors: 96 })
```
Since `create_index` has a training step, it can take a few minutes to finish for large tables. You can control the index
creation by providing the following parameters:
@@ -57,18 +71,28 @@ There are a couple of parameters that can be used to fine-tune the search:
e.g., for 1M vectors divided into 256 partitions, if you're looking for top 20, then refine_factor=200 reranks the whole partition.<br/>
Note: refine_factor is only applicable if an ANN index is present. If specified on a table without an ANN index, it is ignored.
```python
tbl.search(np.random.random((768))) \
.limit(2) \
.nprobes(20) \
.refine_factor(10) \
.to_df()
=== "Python"
```python
tbl.search(np.random.random((768))) \
.limit(2) \
.nprobes(20) \
.refine_factor(10) \
.to_df()
vector item score
0 [0.44949695, 0.8444449, 0.06281311, 0.23338133... item 1141 103.575333
1 [0.48587373, 0.269207, 0.15095535, 0.65531915,... item 3953 108.393867
```
0 [0.44949695, 0.8444449, 0.06281311, 0.23338133... item 1141 103.575333
1 [0.48587373, 0.269207, 0.15095535, 0.65531915,... item 3953 108.393867
```
=== "Javascript"
```javascript
const results = await table
.search(Array(1536).fill(1.2))
.limit(2)
.nprobes(20)
.refineFactor(10)
.execute()
```
The search will return the data requested in addition to the score of each item.
@@ -78,18 +102,31 @@ The search will return the data requested in addition to the score of each item.
You can further filter the elements returned by a search using a where clause.
```python
tbl.search(np.random.random((768))).where("item != 'item 1141'").to_df()
```
=== "Python"
```python
tbl.search(np.random.random((768))).where("item != 'item 1141'").to_df()
```
=== "Javascript"
```javascript
const results = await table
.search(Array(1536).fill(1.2))
.where("item != 'item 1141'")
.execute()
```
### Projections (select clause)
You can select the columns returned by the query using a select clause.
```python
tbl.search(np.random.random((768))).select(["vector"]).to_df()
vector score
0 [0.30928212, 0.022668175, 0.1756372, 0.4911822... 93.971092
1 [0.2525465, 0.01723831, 0.261568, 0.002007689,... 95.173485
...
```
=== "Python"
```python
tbl.search(np.random.random((768))).select(["vector"]).to_df()
vector score
0 [0.30928212, 0.022668175, 0.1756372, 0.4911822... 93.971092
1 [0.2525465, 0.01723831, 0.261568, 0.002007689,... 95.173485
...
```
=== "Javascript"
Projections are not currently supported in the Javascript SDK.

BIN
docs/src/assets/lancedb_embedded_explanation.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 190 KiB

BIN
docs/src/assets/lancedb_local_data_explanation.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 101 KiB

BIN
docs/src/assets/logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.7 KiB

152
docs/src/basic.md
View File

@@ -1,74 +1,142 @@
# Basic LanceDB Functionality
We'll cover the basics of using LanceDB on your local machine in this section.
??? info "LanceDB runs embedded on your backend application, so there is no need to run a separate server."
<img src="../assets/lancedb_embedded_explanation.png" width="650px" />
## Installation
=== "Python"
```shell
pip install lancedb
```
=== "Javascript"
```shell
npm install vectordb
```
## How to connect to a database
In local mode, LanceDB stores data in a directory on your local machine. To connect to a local database, you can use the following code:
```python
import lancedb
uri = "~/.lancedb"
db = lancedb.connect(uri)
```
=== "Python"
```python
import lancedb
uri = "~/.lancedb"
db = lancedb.connect(uri)
```
LanceDB will create the directory if it doesn't exist (including parent directories).
LanceDB will create the directory if it doesn't exist (including parent directories).
If you need a reminder of the uri, use the `db.uri` property.
If you need a reminder of the uri, use the `db.uri` property.
=== "Javascript"
```javascript
const lancedb = require("vectordb");
const uri = "~./lancedb";
const db = await lancedb.connect(uri);
```
LanceDB will create the directory if it doesn't exist (including parent directories).
If you need a reminder of the uri, you can call `db.uri()`.
## How to create a table
To create a table, you can use the following code:
```python
tbl = db.create_table("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
```
=== "Python"
```python
tbl = db.create_table("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
```
Under the hood, LanceDB is converting the input data into an Apache Arrow table
and persisting it to disk in [Lance format](github.com/eto-ai/lance).
If the table already exists, LanceDB will raise an error by default.
If you want to overwrite the table, you can pass in `mode="overwrite"`
to the `create_table` method.
If the table already exists, LanceDB will raise an error by default.
If you want to overwrite the table, you can pass in `mode="overwrite"`
to the `create_table` method.
You can also pass in a pandas DataFrame directly:
```python
import pandas as pd
df = pd.DataFrame([{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
tbl = db.create_table("table_from_df", data=df)
```
You can also pass in a pandas DataFrame directly:
```python
import pandas as pd
df = pd.DataFrame([{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
tbl = db.create_table("table_from_df", data=df)
```
=== "Javascript"
```javascript
const tb = await db.createTable("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
```
!!! warning
If the table already exists, LanceDB will raise an error by default.
If you want to overwrite the table, you can pass in `mode="overwrite"`
to the `createTable` function.
??? info "Under the hood, LanceDB is converting the input data into an Apache Arrow table and persisting it to disk in [Lance format](https://www.github.com/lancedb/lance)."
## How to open an existing table
Once created, you can open a table using the following code:
```python
tbl = db.open_table("my_table")
```
If you forget the name of your table, you can always get a listing of all table names:
=== "Python"
```python
tbl = db.open_table("my_table")
```
```python
db.table_names()
```
If you forget the name of your table, you can always get a listing of all table names:
```python
print(db.table_names())
```
=== "Javascript"
```javascript
const tbl = await db.openTable("my_table");
```
If you forget the name of your table, you can always get a listing of all table names:
```javascript
console.log(db.tableNames());
```
## How to add data to a table
After a table has been created, you can always add more data to it using
```python
df = pd.DataFrame([{"vector": [1.3, 1.4], "item": "fizz", "price": 100.0},
{"vector": [9.5, 56.2], "item": "buzz", "price": 200.0}])
tbl.add(df)
```
=== "Python"
```python
df = pd.DataFrame([{"vector": [1.3, 1.4], "item": "fizz", "price": 100.0},
{"vector": [9.5, 56.2], "item": "buzz", "price": 200.0}])
tbl.add(df)
```
=== "Javascript"
```javascript
await tbl.add([vector: [1.3, 1.4], item: "fizz", price: 100.0},
{vector: [9.5, 56.2], item: "buzz", price: 200.0}])
```
## How to search for (approximate) nearest neighbors
Once you've embedded the query, you can find its nearest neighbors using the following code:
```python
tbl.search([100, 100]).limit(2).to_df()
```
=== "Python"
```python
tbl.search([100, 100]).limit(2).to_df()
```
This returns a pandas DataFrame with the results.
This returns a pandas DataFrame with the results.
=== "Javascript"
```javascript
const query = await tbl.search([100, 100]).limit(2).execute();
```
## What's next

121
docs/src/embedding.md
View File

@@ -25,55 +25,88 @@ def embed_func(batch):
return [model.encode(sentence) for sentence in batch]
```
Please note that currently HuggingFace is only supported in the Python SDK.
### OpenAI example
You can also use an external API like OpenAI to generate embeddings
```python
import openai
import os
=== "Python"
```python
import openai
import os
# Configuring the environment variable OPENAI_API_KEY
if "OPENAI_API_KEY" not in os.environ:
# OR set the key here as a variable
openai.api_key = "sk-..."
# Configuring the environment variable OPENAI_API_KEY
if "OPENAI_API_KEY" not in os.environ:
# OR set the key here as a variable
openai.api_key = "sk-..."
# verify that the API key is working
assert len(openai.Model.list()["data"]) > 0
# verify that the API key is working
assert len(openai.Model.list()["data"]) > 0
def embed_func(c):
rs = openai.Embedding.create(input=c, engine="text-embedding-ada-002")
return [record["embedding"] for record in rs["data"]]
```
def embed_func(c):
rs = openai.Embedding.create(input=c, engine="text-embedding-ada-002")
return [record["embedding"] for record in rs["data"]]
```
=== "Javascript"
```javascript
const lancedb = require("vectordb");
// You need to provide an OpenAI API key
const apiKey = "sk-..."
// The embedding function will create embeddings for the 'text' column
const embedding = new lancedb.OpenAIEmbeddingFunction('text', apiKey)
```
## Applying an embedding function
Using an embedding function, you can apply it to raw data
to generate embeddings for each row.
=== "Python"
Using an embedding function, you can apply it to raw data
to generate embeddings for each row.
Say if you have a pandas DataFrame with a `text` column that you want to be embedded,
you can use the [with_embeddings](https://lancedb.github.io/lancedb/python/#lancedb.embeddings.with_embeddings)
function to generate embeddings and add create a combined pyarrow table:
Say if you have a pandas DataFrame with a `text` column that you want to be embedded,
you can use the [with_embeddings](https://lancedb.github.io/lancedb/python/#lancedb.embeddings.with_embeddings)
function to generate embeddings and add create a combined pyarrow table:
```python
import pandas as pd
from lancedb.embeddings import with_embeddings
df = pd.DataFrame([{"text": "pepperoni"},
{"text": "pineapple"}])
data = with_embeddings(embed_func, df)
```python
import pandas as pd
from lancedb.embeddings import with_embeddings
# The output is used to create / append to a table
# db.create_table("my_table", data=data)
```
df = pd.DataFrame([{"text": "pepperoni"},
{"text": "pineapple"}])
data = with_embeddings(embed_func, df)
If your data is in a different column, you can specify the `column` kwarg to `with_embeddings`.
# The output is used to create / append to a table
# db.create_table("my_table", data=data)
```
By default, LanceDB calls the function with batches of 1000 rows. This can be configured
using the `batch_size` parameter to `with_embeddings`.
If your data is in a different column, you can specify the `column` kwarg to `with_embeddings`.
By default, LanceDB calls the function with batches of 1000 rows. This can be configured
using the `batch_size` parameter to `with_embeddings`.
LanceDB automatically wraps the function with retry and rate-limit logic to ensure the OpenAI
API call is reliable.
=== "Javascript"
Using an embedding function, you can apply it to raw data
to generate embeddings for each row.
You can just pass the embedding function created previously and LanceDB will automatically generate
embededings for your data.
```javascript
const db = await lancedb.connect("/tmp/lancedb");
const data = [
{ text: 'pepperoni' },
{ text: 'pineapple' }
]
const table = await db.createTable('vectors', data, embedding)
```
LanceDB automatically wraps the function with retry and rate-limit logic to ensure the OpenAI
API call is reliable.
## Searching with an embedding function
@@ -81,13 +114,25 @@ At inference time, you also need the same embedding function to embed your query
It's important that you use the same model / function otherwise the embedding vectors don't
belong in the same latent space and your results will be nonsensical.
```python
query = "What's the best pizza topping?"
query_vector = embed_func([query])[0]
tbl.search(query_vector).limit(10).to_df()
```
=== "Python"
```python
query = "What's the best pizza topping?"
query_vector = embed_func([query])[0]
tbl.search(query_vector).limit(10).to_df()
```
The above snippet returns a pandas DataFrame with the 10 closest vectors to the query.
=== "Javascript"
```javascript
const results = await table
.search('What's the best pizza topping?')
.limit(10)
.execute()
```
The above snippet returns an array of records with the 10 closest vectors to the query.
The above snippet returns a pandas DataFrame with the 10 closest vectors to the query.
## Roadmap

90
docs/src/examples/modal.py Normal file
View File

@@ -0,0 +1,90 @@
import sys
from modal import Secret, Stub, Image
import lancedb
import re
import pickle
from pathlib import Path
from langchain.document_loaders import UnstructuredHTMLLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import LanceDB
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA
lancedb_image = Image.debian_slim().pip_install("lancedb", "langchain", "openai", "tiktoken")
stub = Stub(
name="example-langchain-lancedb",
image=lancedb_image,
secrets=[Secret.from_name("my-openai-secret")],
)
docsearch = None
docs_path = Path("docs.pkl")
db_path = Path("lancedb")
doc_cache = []
def get_document_title(document):
m = str(document.metadata["source"])
title = re.findall("pandas.documentation(.*).html", m)
if title[0] is not None:
return(title[0])
return ''
def store_docs():
docs = []
if not docs_path.exists():
for p in Path("./pandas.documentation").rglob("*.html"):
if p.is_dir():
continue
loader = UnstructuredHTMLLoader(p)
raw_document = loader.load()
m = {}
m["title"] = get_document_title(raw_document[0])
m["version"] = "2.0rc0"
raw_document[0].metadata = raw_document[0].metadata | m
raw_document[0].metadata["source"] = str(raw_document[0].metadata["source"])
docs = docs + raw_document
with docs_path.open("wb") as fh:
pickle.dump(docs, fh)
else:
with docs_path.open("rb") as fh:
docs = pickle.load(fh)
doc_cache = docs
def qanda_langchain(query):
store_docs()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
documents = text_splitter.split_documents(doc_cache)
embeddings = OpenAIEmbeddings()
db = lancedb.connect(db_path)
table = db.create_table("pandas_docs", data=[
{"vector": embeddings.embed_query("Hello World")}
], mode="overwrite")
docsearch = LanceDB.from_documents(documents, embeddings, connection=table)
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())
return qa.run(query)
@stub.function()
def cli(query: str, show_sources: bool = False):
answer = qanda_langchain(query)
# Terminal codes for pretty-printing.
bold, end = "\033[1m", "\033[0m"
print(f"🦜 {bold}ANSWER:{end}")
print(answer)
if show_sources:
print(f"🔗 {bold}SOURCES:{end}")
for text in sources:
print(text)
print("----")

49
docs/src/index.md
View File

@@ -1,6 +1,6 @@
# Welcome to LanceDB's Documentation
LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrivial, filtering and management of embeddings.
LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrevial, filtering and management of embeddings.
The key features of LanceDB include:
@@ -8,37 +8,52 @@ The key features of LanceDB include:
* Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).
* Native Python and Javascript/Typescript support (coming soon).
* Native Python and Javascript/Typescript support.
* Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.
* Ecosystem integrations with [LangChain 🦜️🔗](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lanecdb.html), [LlamaIndex 🦙](https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html), Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
LanceDB's core is written in Rust 🦀 and is built using Lance, an open-source columnar format designed for performant ML workloads.
LanceDB's core is written in Rust 🦀 and is built using <a href="https://github.com/eto-ai/lance">Lance</a>, an open-source columnar format designed for performant ML workloads.
## Quick Start
## Installation
=== "Python"
```shell
pip install lancedb
```
```shell
pip install lancedb
```
```python
import lancedb
## Quickstart
uri = "/tmp/lancedb"
db = lancedb.connect(uri)
table = db.create_table("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
result = table.search([100, 100]).limit(2).to_df()
```
```python
import lancedb
=== "Javascript"
```shell
npm install vectordb
```
db = lancedb.connect(".")
table = db.create_table("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
result = table.search([100, 100]).limit(2).to_df()
```
```javascript
const lancedb = require("vectordb");
const uri = "/tmp/lancedb";
const db = await lancedb.connect(uri);
const table = await db.createTable("my_table",
[{ id: 1, vector: [3.1, 4.1], item: "foo", price: 10.0 },
{ id: 2, vector: [5.9, 26.5], item: "bar", price: 20.0 }])
const results = await table.search([100, 100]).limit(2).execute();
```
## Complete Demos
We will be adding completed demo apps built using LanceDB.
- [YouTube Transcript Search](../../notebooks/youtube_transcript_search.ipynb)
- [YouTube Transcript Search](notebooks/youtube_transcript_search.ipynb)
## Documentation Quick Links

1
docs/src/javascript/.nojekyll Normal file
View File

@@ -0,0 +1 @@
TypeDoc added this file to prevent GitHub Pages from using Jekyll. You can turn off this behavior by setting the `githubPages` option to false.

51
docs/src/javascript/README.md Normal file
View File

@@ -0,0 +1,51 @@
vectordb / [Exports](modules.md)
# LanceDB
A JavaScript / Node.js library for [LanceDB](https://github.com/lancedb/lancedb).
## Installation
```bash
npm install vectordb
```
## Usage
### Basic Example
```javascript
const lancedb = require('vectordb');
const db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>');
const table = await db.openTable('my_table');
const query = await table.search([0.1, 0.3]).setLimit(20).execute();
console.log(results);
```
The [examples](./examples) folder contains complete examples.
## Development
The LanceDB javascript is built with npm:
```bash
npm run tsc
```
Run the tests with
```bash
npm test
```
To run the linter and have it automatically fix all errors
```bash
npm run lint -- --fix
```
To build documentation
```bash
npx typedoc --plugin typedoc-plugin-markdown --out ../docs/src/javascript src/index.ts
```

211
docs/src/javascript/classes/Connection.md Normal file
View File

@@ -0,0 +1,211 @@
[vectordb](../README.md) / [Exports](../modules.md) / Connection
# Class: Connection
A connection to a LanceDB database.
## Table of contents
### Constructors
- [constructor](Connection.md#constructor)
### Properties
- [\_db](Connection.md#_db)
- [\_uri](Connection.md#_uri)
### Accessors
- [uri](Connection.md#uri)
### Methods
- [createTable](Connection.md#createtable)
- [createTableArrow](Connection.md#createtablearrow)
- [openTable](Connection.md#opentable)
- [tableNames](Connection.md#tablenames)
## Constructors
### constructor
• **new Connection**(`db`, `uri`)
#### Parameters
| Name | Type |
| :------ | :------ |
| `db` | `any` |
| `uri` | `string` |
#### Defined in
[index.ts:46](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L46)
## Properties
### \_db
• `Private` `Readonly` **\_db**: `any`
#### Defined in
[index.ts:44](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L44)
___
### \_uri
• `Private` `Readonly` **\_uri**: `string`
#### Defined in
[index.ts:43](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L43)
## Accessors
### uri
• `get` **uri**(): `string`
#### Returns
`string`
#### Defined in
[index.ts:51](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L51)
## Methods
### createTable
▸ **createTable**(`name`, `data`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
Creates a new Table and initialize it with new data.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `name` | `string` | The name of the table. |
| `data` | `Record`<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the Table |
#### Returns
`Promise`<[`Table`](Table.md)<`number`[]\>\>
#### Defined in
[index.ts:91](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L91)
▸ **createTable**<`T`\>(`name`, `data`, `embeddings`): `Promise`<[`Table`](Table.md)<`T`\>\>
Creates a new Table and initialize it with new data.
#### Type parameters
| Name |
| :------ |
| `T` |
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `name` | `string` | The name of the table. |
| `data` | `Record`<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the Table |
| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use on this Table |
#### Returns
`Promise`<[`Table`](Table.md)<`T`\>\>
#### Defined in
[index.ts:99](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L99)
___
### createTableArrow
▸ **createTableArrow**(`name`, `table`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
#### Parameters
| Name | Type |
| :------ | :------ |
| `name` | `string` |
| `table` | `Table`<`any`\> |
#### Returns
`Promise`<[`Table`](Table.md)<`number`[]\>\>
#### Defined in
[index.ts:109](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L109)
___
### openTable
▸ **openTable**(`name`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
Open a table in the database.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `name` | `string` | The name of the table. |
#### Returns
`Promise`<[`Table`](Table.md)<`number`[]\>\>
#### Defined in
[index.ts:67](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L67)
▸ **openTable**<`T`\>(`name`, `embeddings`): `Promise`<[`Table`](Table.md)<`T`\>\>
Open a table in the database.
#### Type parameters
| Name |
| :------ |
| `T` |
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `name` | `string` | The name of the table. |
| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use on this Table |
#### Returns
`Promise`<[`Table`](Table.md)<`T`\>\>
#### Defined in
[index.ts:74](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L74)
___
### tableNames
▸ **tableNames**(): `Promise`<`string`[]\>
Get the names of all tables in the database.
#### Returns
`Promise`<`string`[]\>
#### Defined in
[index.ts:58](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L58)

105
docs/src/javascript/classes/OpenAIEmbeddingFunction.md Normal file
View File

@@ -0,0 +1,105 @@
[vectordb](../README.md) / [Exports](../modules.md) / OpenAIEmbeddingFunction
# Class: OpenAIEmbeddingFunction
An embedding function that automatically creates vector representation for a given column.
## Implements
- [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`string`\>
## Table of contents
### Constructors
- [constructor](OpenAIEmbeddingFunction.md#constructor)
### Properties
- [\_modelName](OpenAIEmbeddingFunction.md#_modelname)
- [\_openai](OpenAIEmbeddingFunction.md#_openai)
- [sourceColumn](OpenAIEmbeddingFunction.md#sourcecolumn)
### Methods
- [embed](OpenAIEmbeddingFunction.md#embed)
## Constructors
### constructor
• **new OpenAIEmbeddingFunction**(`sourceColumn`, `openAIKey`, `modelName?`)
#### Parameters
| Name | Type | Default value |
| :------ | :------ | :------ |
| `sourceColumn` | `string` | `undefined` |
| `openAIKey` | `string` | `undefined` |
| `modelName` | `string` | `'text-embedding-ada-002'` |
#### Defined in
[embedding/openai.ts:21](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L21)
## Properties
### \_modelName
• `Private` `Readonly` **\_modelName**: `string`
#### Defined in
[embedding/openai.ts:19](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L19)
___
### \_openai
• `Private` `Readonly` **\_openai**: `any`
#### Defined in
[embedding/openai.ts:18](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L18)
___
### sourceColumn
• **sourceColumn**: `string`
The name of the column that will be used as input for the Embedding Function.
#### Implementation of
[EmbeddingFunction](../interfaces/EmbeddingFunction.md).[sourceColumn](../interfaces/EmbeddingFunction.md#sourcecolumn)
#### Defined in
[embedding/openai.ts:50](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L50)
## Methods
### embed
▸ **embed**(`data`): `Promise`<`number`[][]\>
Creates a vector representation for the given values.
#### Parameters
| Name | Type |
| :------ | :------ |
| `data` | `string`[] |
#### Returns
`Promise`<`number`[][]\>
#### Implementation of
[EmbeddingFunction](../interfaces/EmbeddingFunction.md).[embed](../interfaces/EmbeddingFunction.md#embed)
#### Defined in
[embedding/openai.ts:38](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L38)

299
docs/src/javascript/classes/Query.md Normal file
View File

@@ -0,0 +1,299 @@
[vectordb](../README.md) / [Exports](../modules.md) / Query
# Class: Query<T\>
A builder for nearest neighbor queries for LanceDB.
## Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `number`[] |
## Table of contents
### Constructors
- [constructor](Query.md#constructor)
### Properties
- [\_columns](Query.md#_columns)
- [\_embeddings](Query.md#_embeddings)
- [\_filter](Query.md#_filter)
- [\_limit](Query.md#_limit)
- [\_metricType](Query.md#_metrictype)
- [\_nprobes](Query.md#_nprobes)
- [\_query](Query.md#_query)
- [\_queryVector](Query.md#_queryvector)
- [\_refineFactor](Query.md#_refinefactor)
- [\_tbl](Query.md#_tbl)
### Methods
- [execute](Query.md#execute)
- [filter](Query.md#filter)
- [limit](Query.md#limit)
- [metricType](Query.md#metrictype)
- [nprobes](Query.md#nprobes)
- [refineFactor](Query.md#refinefactor)
## Constructors
### constructor
• **new Query**<`T`\>(`tbl`, `query`, `embeddings?`)
#### Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `number`[] |
#### Parameters
| Name | Type |
| :------ | :------ |
| `tbl` | `any` |
| `query` | `T` |
| `embeddings?` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> |
#### Defined in
[index.ts:241](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L241)
## Properties
### \_columns
• `Private` `Optional` `Readonly` **\_columns**: `string`[]
#### Defined in
[index.ts:236](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L236)
___
### \_embeddings
• `Private` `Optional` `Readonly` **\_embeddings**: [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\>
#### Defined in
[index.ts:239](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L239)
___
### \_filter
• `Private` `Optional` **\_filter**: `string`
#### Defined in
[index.ts:237](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L237)
___
### \_limit
• `Private` **\_limit**: `number`
#### Defined in
[index.ts:233](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L233)
___
### \_metricType
• `Private` `Optional` **\_metricType**: [`MetricType`](../enums/MetricType.md)
#### Defined in
[index.ts:238](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L238)
___
### \_nprobes
• `Private` **\_nprobes**: `number`
#### Defined in
[index.ts:235](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L235)
___
### \_query
• `Private` `Readonly` **\_query**: `T`
#### Defined in
[index.ts:231](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L231)
___
### \_queryVector
• `Private` `Optional` **\_queryVector**: `number`[]
#### Defined in
[index.ts:232](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L232)
___
### \_refineFactor
• `Private` `Optional` **\_refineFactor**: `number`
#### Defined in
[index.ts:234](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L234)
___
### \_tbl
• `Private` `Readonly` **\_tbl**: `any`
#### Defined in
[index.ts:230](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L230)
## Methods
### execute
▸ **execute**<`T`\>(): `Promise`<`T`[]\>
Execute the query and return the results as an Array of Objects
#### Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `Record`<`string`, `unknown`\> |
#### Returns
`Promise`<`T`[]\>
#### Defined in
[index.ts:301](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L301)
___
### filter
▸ **filter**(`value`): [`Query`](Query.md)<`T`\>
A filter statement to be applied to this query.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `value` | `string` | A filter in the same format used by a sql WHERE clause. |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:284](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L284)
___
### limit
▸ **limit**(`value`): [`Query`](Query.md)<`T`\>
Sets the number of results that will be returned
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `value` | `number` | number of results |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:257](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L257)
___
### metricType
▸ **metricType**(`value`): [`Query`](Query.md)<`T`\>
The MetricType used for this Query.
**`See`**
MetricType for the different options
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `value` | [`MetricType`](../enums/MetricType.md) | The metric to the. |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:293](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L293)
___
### nprobes
▸ **nprobes**(`value`): [`Query`](Query.md)<`T`\>
The number of probes used. A higher number makes search more accurate but also slower.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `value` | `number` | The number of probes used. |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:275](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L275)
___
### refineFactor
▸ **refineFactor**(`value`): [`Query`](Query.md)<`T`\>
Refine the results by reading extra elements and re-ranking them in memory.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `value` | `number` | refine factor to use in this query. |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:266](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L266)

215
docs/src/javascript/classes/Table.md Normal file
View File

@@ -0,0 +1,215 @@
[vectordb](../README.md) / [Exports](../modules.md) / Table
# Class: Table<T\>
## Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `number`[] |
## Table of contents
### Constructors
- [constructor](Table.md#constructor)
### Properties
- [\_embeddings](Table.md#_embeddings)
- [\_name](Table.md#_name)
- [\_tbl](Table.md#_tbl)
### Accessors
- [name](Table.md#name)
### Methods
- [add](Table.md#add)
- [create\_index](Table.md#create_index)
- [overwrite](Table.md#overwrite)
- [search](Table.md#search)
## Constructors
### constructor
• **new Table**<`T`\>(`tbl`, `name`)
#### Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `number`[] |
#### Parameters
| Name | Type |
| :------ | :------ |
| `tbl` | `any` |
| `name` | `string` |
#### Defined in
[index.ts:121](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L121)
• **new Table**<`T`\>(`tbl`, `name`, `embeddings`)
#### Type parameters
| Name | Type |
| :------ | :------ |
| `T` | `number`[] |
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `tbl` | `any` | |
| `name` | `string` | |
| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use when interacting with this table |
#### Defined in
[index.ts:127](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L127)
## Properties
### \_embeddings
• `Private` `Optional` `Readonly` **\_embeddings**: [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\>
#### Defined in
[index.ts:119](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L119)
___
### \_name
• `Private` `Readonly` **\_name**: `string`
#### Defined in
[index.ts:118](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L118)
___
### \_tbl
• `Private` `Readonly` **\_tbl**: `any`
#### Defined in
[index.ts:117](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L117)
## Accessors
### name
• `get` **name**(): `string`
#### Returns
`string`
#### Defined in
[index.ts:134](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L134)
## Methods
### add
▸ **add**(`data`): `Promise`<`number`\>
Insert records into this Table.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `data` | `Record`<`string`, `unknown`\>[] | Records to be inserted into the Table |
#### Returns
`Promise`<`number`\>
The number of rows added to the table
#### Defined in
[index.ts:152](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L152)
___
### create\_index
▸ **create_index**(`indexParams`): `Promise`<`any`\>
Create an ANN index on this Table vector index.
**`See`**
VectorIndexParams.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `indexParams` | `IvfPQIndexConfig` | The parameters of this Index, |
#### Returns
`Promise`<`any`\>
#### Defined in
[index.ts:171](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L171)
___
### overwrite
▸ **overwrite**(`data`): `Promise`<`number`\>
Insert records into this Table, replacing its contents.
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `data` | `Record`<`string`, `unknown`\>[] | Records to be inserted into the Table |
#### Returns
`Promise`<`number`\>
The number of rows added to the table
#### Defined in
[index.ts:162](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L162)
___
### search
▸ **search**(`query`): [`Query`](Query.md)<`T`\>
Creates a search query to find the nearest neighbors of the given search term
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `query` | `T` | The query search term |
#### Returns
[`Query`](Query.md)<`T`\>
#### Defined in
[index.ts:142](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L142)

36
docs/src/javascript/enums/MetricType.md Normal file
View File

@@ -0,0 +1,36 @@
[vectordb](../README.md) / [Exports](../modules.md) / MetricType
# Enumeration: MetricType
Distance metrics type.
## Table of contents
### Enumeration Members
- [Cosine](MetricType.md#cosine)
- [L2](MetricType.md#l2)
## Enumeration Members
### Cosine
• **Cosine** = ``"cosine"``
Cosine distance
#### Defined in
[index.ts:341](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L341)
___
### L2
• **L2** = ``"l2"``
Euclidean distance
#### Defined in
[index.ts:336](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L336)

30
docs/src/javascript/enums/WriteMode.md Normal file
View File

@@ -0,0 +1,30 @@
[vectordb](../README.md) / [Exports](../modules.md) / WriteMode
# Enumeration: WriteMode
## Table of contents
### Enumeration Members
- [Append](WriteMode.md#append)
- [Overwrite](WriteMode.md#overwrite)
## Enumeration Members
### Append
• **Append** = ``"append"``
#### Defined in
[index.ts:326](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L326)
___
### Overwrite
• **Overwrite** = ``"overwrite"``
#### Defined in
[index.ts:325](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L325)

60
docs/src/javascript/interfaces/EmbeddingFunction.md Normal file
View File

@@ -0,0 +1,60 @@
[vectordb](../README.md) / [Exports](../modules.md) / EmbeddingFunction
# Interface: EmbeddingFunction<T\>
An embedding function that automatically creates vector representation for a given column.
## Type parameters
| Name |
| :------ |
| `T` |
## Implemented by
- [`OpenAIEmbeddingFunction`](../classes/OpenAIEmbeddingFunction.md)
## Table of contents
### Properties
- [embed](EmbeddingFunction.md#embed)
- [sourceColumn](EmbeddingFunction.md#sourcecolumn)
## Properties
### embed
• **embed**: (`data`: `T`[]) => `Promise`<`number`[][]\>
#### Type declaration
▸ (`data`): `Promise`<`number`[][]\>
Creates a vector representation for the given values.
##### Parameters
| Name | Type |
| :------ | :------ |
| `data` | `T`[] |
##### Returns
`Promise`<`number`[][]\>
#### Defined in
[embedding/embedding_function.ts:27](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/embedding_function.ts#L27)
___
### sourceColumn
• **sourceColumn**: `string`
The name of the column that will be used as input for the Embedding Function.
#### Defined in
[embedding/embedding_function.ts:22](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/embedding_function.ts#L22)

61
docs/src/javascript/modules.md Normal file
View File

@@ -0,0 +1,61 @@
[vectordb](README.md) / Exports
# vectordb
## Table of contents
### Enumerations
- [MetricType](enums/MetricType.md)
- [WriteMode](enums/WriteMode.md)
### Classes
- [Connection](classes/Connection.md)
- [OpenAIEmbeddingFunction](classes/OpenAIEmbeddingFunction.md)
- [Query](classes/Query.md)
- [Table](classes/Table.md)
### Interfaces
- [EmbeddingFunction](interfaces/EmbeddingFunction.md)
### Type Aliases
- [VectorIndexParams](modules.md#vectorindexparams)
### Functions
- [connect](modules.md#connect)
## Type Aliases
### VectorIndexParams
Ƭ **VectorIndexParams**: `IvfPQIndexConfig`
#### Defined in
[index.ts:224](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L224)
## Functions
### connect
▸ **connect**(`uri`): `Promise`<[`Connection`](classes/Connection.md)\>
Connect to a LanceDB instance at the given URI
#### Parameters
| Name | Type | Description |
| :------ | :------ | :------ |
| `uri` | `string` | The uri of the database. |
#### Returns
`Promise`<[`Connection`](classes/Connection.md)\>
#### Defined in
[index.ts:34](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L34)

357
docs/src/notebooks/code_qa_bot.ipynb Normal file
View File

@@ -0,0 +1,357 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "13cb272e",
"metadata": {},
"source": [
"# Code documentation Q&A bot example with LangChain\n",
"\n",
"This Q&A bot will allow you to query your own documentation easily using questions. We'll also demonstrate the use of LangChain and LanceDB using the OpenAI API. \n",
"\n",
"In this example we'll use Pandas 2.0 documentation, but, this could be replaced for your own docs as well"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "66638d6c",
"metadata": {},
"outputs": [],
"source": [
"!pip install --quiet openai langchain\n",
"!pip install --quiet -U lancedb"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d1cdcac3",
"metadata": {},
"source": [
"First, let's get some setup out of the way. As we're using the OpenAI API, ensure that you've set your key (and organization if needed):"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "58ee1868",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"import os\n",
"\n",
"# Configuring the environment variable OPENAI_API_KEY\n",
"if \"OPENAI_API_KEY\" not in os.environ:\n",
" # OR set the key here as a variable\n",
" openai.api_key = \"sk-...\"\n",
" \n",
"assert len(openai.Model.list()[\"data\"]) > 0"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "34f524d3",
"metadata": {},
"source": [
"# Loading in our code documentation, generating embeddings and storing our documents in LanceDB\n",
"\n",
"We're going to use the power of LangChain to help us create our Q&A bot. It comes with several APIs that can make our development much easier as well as a LanceDB integration for vectorstore."
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "b55d22f1",
"metadata": {},
"outputs": [],
"source": [
"import lancedb\n",
"import re\n",
"import pickle\n",
"from pathlib import Path\n",
"\n",
"from langchain.document_loaders import UnstructuredHTMLLoader\n",
"from langchain.embeddings import OpenAIEmbeddings\n",
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
"from langchain.vectorstores import LanceDB\n",
"from langchain.llms import OpenAI\n",
"from langchain.chains import RetrievalQA"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6ccf9b2b",
"metadata": {},
"source": [
"You can download the Pandas documentation from https://pandas.pydata.org/docs/. To make sure we're not littering our repo with docs, we won't include it in the LanceDB repo, so download this and store it locally first."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ae42496c",
"metadata": {},
"source": [
"We'll create a simple helper function that can help to extract metadata, so we can use this downstream when we're wanting to query with filters. In this case, we want to keep the lineage of the uri or path for each document that we process:"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "d171d062",
"metadata": {},
"outputs": [],
"source": [
"def get_document_title(document):\n",
" m = str(document.metadata[\"source\"])\n",
" title = re.findall(\"pandas.documentation(.*).html\", m)\n",
" if title[0] is not None:\n",
" return(title[0])\n",
" return ''"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "130162ad",
"metadata": {},
"source": [
"# Pre-processing and loading the documentation\n",
"\n",
"Next, let's pre-process and load the documentation. To make sure we don't need to do this repeatedly if we were updating code, we're caching it using pickle so we can retrieve it again (this could take a few minutes to run the first time yyou do it). We'll also add some more metadata to the docs here such as the title and version of the code:"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "33bfe7d8",
"metadata": {},
"outputs": [],
"source": [
"docs_path = Path(\"docs.pkl\")\n",
"docs = []\n",
"\n",
"if not docs_path.exists():\n",
" for p in Path(\"./pandas.documentation\").rglob(\"*.html\"):\n",
" if p.is_dir():\n",
" continue\n",
" loader = UnstructuredHTMLLoader(p)\n",
" raw_document = loader.load()\n",
" \n",
" m = {}\n",
" m[\"title\"] = get_document_title(raw_document[0])\n",
" m[\"version\"] = \"2.0rc0\"\n",
" raw_document[0].metadata = raw_document[0].metadata | m\n",
" raw_document[0].metadata[\"source\"] = str(raw_document[0].metadata[\"source\"])\n",
" docs = docs + raw_document\n",
"\n",
" with docs_path.open(\"wb\") as fh:\n",
" pickle.dump(docs, fh)\n",
"else:\n",
" with docs_path.open(\"rb\") as fh:\n",
" docs = pickle.load(fh)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c3852dd3",
"metadata": {},
"source": [
"# Generating emebeddings from our docs\n",
"\n",
"Now that we have our raw documents loaded, we need to pre-process them to generate embeddings:"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "82230563",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size=1000,\n",
" chunk_overlap=200,\n",
")\n",
"documents = text_splitter.split_documents(docs)\n",
"embeddings = OpenAIEmbeddings()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "43e68215",
"metadata": {},
"source": [
"# Storing and querying with LanceDB\n",
"\n",
"Let's connect to LanceDB so we can store our documents. We'll create a Table to store them in:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "74780a58",
"metadata": {},
"outputs": [],
"source": [
"db = lancedb.connect('/tmp/lancedb')\n",
"table = db.create_table(\"pandas_docs\", data=[\n",
" {\"vector\": embeddings.embed_query(\"Hello World\"), \"text\": \"Hello World\", \"id\": \"1\"}\n",
"], mode=\"overwrite\")\n",
"docsearch = LanceDB.from_documents(documents, embeddings, connection=table)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3cb1dc5d",
"metadata": {},
"source": [
"Now let's create our RetrievalQA chain using the LanceDB vector store:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "6a5891ad",
"metadata": {},
"outputs": [],
"source": [
"qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type=\"stuff\", retriever=docsearch.as_retriever())"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "28d93b85",
"metadata": {},
"source": [
"And thats it! We're all setup. The next step is to run some queries, let's try a few:"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "70d88316",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' The major differences in pandas 2.0 include installing optional dependencies with pip extras, the ability to use any numpy numeric dtype in an Index, and enhancements, notable bug fixes, backwards incompatible API changes, deprecations, and performance improvements.'"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"What are the major differences in pandas 2.0?\"\n",
"qa.run(query)"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "85a0397c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' 2.0.0rc0'"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"What's the current version of pandas?\"\n",
"qa.run(query)"
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "923f86c6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' Optional dependencies can be installed with pip install \"pandas[all]\" or \"pandas[performance]\". This will install all recommended performance dependencies such as numexpr, bottleneck and numba.'"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"How do I make use of installing optional dependencies?\"\n",
"qa.run(query)"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "02082f83",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\" \\n\\nPandas 2.0 includes a number of API breaking changes, such as increased minimum versions for dependencies, the use of os.linesep for DataFrame.to_csv's line_terminator, and reorganization of the library. See the release notes for a full list of changes.\""
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"What are the backwards incompatible API changes in Pandas 2.0?\"\n",
"qa.run(query)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "75cea547",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

108
docs/src/notebooks/diffusiondb/datagen.py Executable file
View File

@@ -0,0 +1,108 @@
#!/usr/bin/env python
#
# Copyright 2023 LanceDB Developers
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Dataset hf://poloclub/diffusiondb
"""
import io
from argparse import ArgumentParser
from multiprocessing import Pool
import lance
import lancedb
import pyarrow as pa
from datasets import load_dataset
from PIL import Image
from transformers import CLIPModel, CLIPProcessor, CLIPTokenizerFast
MODEL_ID = "openai/clip-vit-base-patch32"
device = "cuda"
tokenizer = CLIPTokenizerFast.from_pretrained(MODEL_ID)
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
schema = pa.schema(
[
pa.field("prompt", pa.string()),
pa.field("seed", pa.uint32()),
pa.field("step", pa.uint16()),
pa.field("cfg", pa.float32()),
pa.field("sampler", pa.string()),
pa.field("width", pa.uint16()),
pa.field("height", pa.uint16()),
pa.field("timestamp", pa.timestamp("s")),
pa.field("image_nsfw", pa.float32()),
pa.field("prompt_nsfw", pa.float32()),
pa.field("vector", pa.list_(pa.float32(), 512)),
pa.field("image", pa.binary()),
]
)
def pil_to_bytes(img) -> list[bytes]:
buf = io.BytesIO()
img.save(buf, format="PNG")
return buf.getvalue()
def generate_clip_embeddings(batch) -> pa.RecordBatch:
image = processor(text=None, images=batch["image"], return_tensors="pt")[
"pixel_values"
].to(device)
img_emb = model.get_image_features(image)
batch["vector"] = img_emb.cpu().tolist()
with Pool() as p:
batch["image_bytes"] = p.map(pil_to_bytes, batch["image"])
return batch
def datagen(args):
"""Generate DiffusionDB dataset, and use CLIP model to generate image embeddings."""
dataset = load_dataset("poloclub/diffusiondb", args.subset)
data = []
for b in dataset.map(
generate_clip_embeddings, batched=True, batch_size=256, remove_columns=["image"]
)["train"]:
b["image"] = b["image_bytes"]
del b["image_bytes"]
data.append(b)
tbl = pa.Table.from_pylist(data, schema=schema)
return tbl
def main():
parser = ArgumentParser()
parser.add_argument(
"-o", "--output", metavar="DIR", help="Output lance directory", required=True
)
parser.add_argument(
"-s",
"--subset",
choices=["2m_all", "2m_first_10k", "2m_first_100k"],
default="2m_first_10k",
help="subset of the hg dataset",
)
args = parser.parse_args()
batches = datagen(args)
lance.write_dataset(batches, args.output)
if __name__ == "__main__":
main()

9
docs/src/notebooks/diffusiondb/requirements.txt Normal file
View File

@@ -0,0 +1,9 @@
datasets
Pillow
lancedb
isort
black
transformers
--index-url https://download.pytorch.org/whl/cu118
torch
torchvision

240
docs/src/notebooks/multimodal_search.ipynb Normal file
View File

@@ -0,0 +1,240 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
]
}
],
"source": [
"!pip install --quiet -U lancedb\n",
"!pip install --quiet gradio transformers torch torchvision"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import PIL\n",
"import duckdb\n",
"import lancedb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## First run setup: Download data and pre-process"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<lance.dataset.LanceDataset at 0x3045db590>"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# remove null prompts\n",
"import lance\n",
"import pyarrow.compute as pc\n",
"\n",
"# download s3://eto-public/datasets/diffusiondb/small_10k.lance to this uri\n",
"data = lance.dataset(\"~/datasets/rawdata.lance\").to_table()\n",
"\n",
"# First data processing and full-text-search index\n",
"db = lancedb.connect(\"~/datasets/demo\")\n",
"tbl = db.create_table(\"diffusiondb\", data.filter(~pc.field(\"prompt\").is_null()))\n",
"tbl = tbl.create_fts_index([\"prompt\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create / Open LanceDB Table"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"db = lancedb.connect(\"~/datasets/demo\")\n",
"tbl = db.open_table(\"diffusiondb\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create CLIP embedding function for the text"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"from transformers import CLIPModel, CLIPProcessor, CLIPTokenizerFast\n",
"\n",
"MODEL_ID = \"openai/clip-vit-base-patch32\"\n",
"\n",
"tokenizer = CLIPTokenizerFast.from_pretrained(MODEL_ID)\n",
"model = CLIPModel.from_pretrained(MODEL_ID)\n",
"processor = CLIPProcessor.from_pretrained(MODEL_ID)\n",
"\n",
"def embed_func(query):\n",
" inputs = tokenizer([query], padding=True, return_tensors=\"pt\")\n",
" text_features = model.get_text_features(**inputs)\n",
" return text_features.detach().numpy()[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search functions for Gradio"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [],
"source": [
"def find_image_vectors(query):\n",
" emb = embed_func(query)\n",
" return _extract(tbl.search(emb).limit(9).to_df())\n",
"\n",
"def find_image_keywords(query):\n",
" return _extract(tbl.search(query).limit(9).to_df())\n",
"\n",
"def find_image_sql(query):\n",
" diffusiondb = tbl.to_lance()\n",
" return _extract(duckdb.query(query).to_df())\n",
"\n",
"def _extract(df):\n",
" image_col = \"image\"\n",
" return [(PIL.Image.open(io.BytesIO(row[image_col])), row[\"prompt\"]) for _, row in df.iterrows()]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup Gradio interface"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on local URL: http://127.0.0.1:7867\n",
"\n",
"To create a public link, set `share=True` in `launch()`.\n"
]
},
{
"data": {
"text/html": [
"<div><iframe src=\"http://127.0.0.1:7867/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": []
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import gradio as gr\n",
"\n",
"\n",
"with gr.Blocks() as demo:\n",
"\n",
" with gr.Row():\n",
" with gr.Tab(\"Embeddings\"):\n",
" vector_query = gr.Textbox(value=\"portraits of a person\", show_label=False)\n",
" b1 = gr.Button(\"Submit\")\n",
" with gr.Tab(\"Keywords\"):\n",
" keyword_query = gr.Textbox(value=\"ninja turtle\", show_label=False)\n",
" b2 = gr.Button(\"Submit\")\n",
" with gr.Tab(\"SQL\"):\n",
" sql_query = gr.Textbox(value=\"SELECT * from diffusiondb WHERE image_nsfw >= 2 LIMIT 9\", show_label=False)\n",
" b3 = gr.Button(\"Submit\")\n",
" with gr.Row():\n",
" gallery = gr.Gallery(\n",
" label=\"Found images\", show_label=False, elem_id=\"gallery\"\n",
" ).style(columns=[3], rows=[3], object_fit=\"contain\", height=\"auto\") \n",
" \n",
" b1.click(find_image_vectors, inputs=vector_query, outputs=gallery)\n",
" b2.click(find_image_keywords, inputs=keyword_query, outputs=gallery)\n",
" b3.click(find_image_sql, inputs=sql_query, outputs=gallery)\n",
" \n",
"demo.launch()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
}
},
"nbformat": 4,
"nbformat_minor": 1
}

683
docs/src/notebooks/youtube_transcript_search.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

0
docs/src/python.md → docs/src/python/python.md
View File

6
docs/src/styles/global.css Normal file
View File

@@ -0,0 +1,6 @@
:root {
--md-primary-fg-color: #625eff;
--md-primary-fg-color--dark: #4338ca;
--md-text-font: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
--md-code-font: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
}