bump version for v0.1.6-python

Better handle empty results from tantivy (#155 )
Closes #154 --------- Co-authored-by: Chang She <chang@lancedb.com>
2025-12-23 21:39:57 +00:00 · 2023-06-05 18:20:15 -07:00 · 2023-06-05 18:18:14 -07:00 · 2023-06-03 13:59:58 -07:00 · 2023-06-03 11:05:35 -07:00 · 2023-06-03 11:05:02 -07:00
47 changed files with 2289 additions and 191 deletions
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -32,7 +32,7 @@ jobs:
      run: |
        pip install -e .
        pip install tantivy@git+https://github.com/quickwit-oss/tantivy-py#164adc87e1a033117001cf70e38c82a53014d985
-        pip install pytest
+        pip install pytest pytest-mock
    - name: Run tests
      run: pytest -x -v --durations=30 tests
  mac:
@@ -55,6 +55,6 @@ jobs:
      run: |
        pip install -e .
        pip install tantivy@git+https://github.com/quickwit-oss/tantivy-py#164adc87e1a033117001cf70e38c82a53014d985
-        pip install pytest
+        pip install pytest pytest-mock
    - name: Run tests
      run: pytest -x -v --durations=30 tests
--- a/.gitignore
+++ b/.gitignore
@@ -15,7 +15,7 @@ site
 python/build
 python/dist

-notebooks/.ipynb_checkpoints
+**/.ipynb_checkpoints

 **/.hypothesis

--- a/README.md
+++ b/README.md
@@ -10,6 +10,10 @@
 <a href="https://discord.gg/zMM32dvNtd">Discord</a> •
 <a href="https://twitter.com/lancedb">Twitter</a>

+</p>
+
+<img max-width="750px" alt="LanceDB Multimodal Search" src="https://github.com/lancedb/lancedb/assets/917119/09c5afc5-7816-4687-bae4-f2ca194426ec">
+
 </p>
 </div>

@@ -23,13 +27,15 @@ The key features of LanceDB include:

 * Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).

+* Support for vector similarity search, full-text search and SQL.
+
 * Native Python and Javascript/Typescript support.

 * Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.

 * Ecosystem integrations with [LangChain 🦜️🔗](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lanecdb.html), [LlamaIndex 🦙](https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html), Apache-Arrow, Pandas, Polars, DuckDB and more on the way.

-LanceDB's core is written in Rust 🦀 and is built using <a href="https://github.com/eto-ai/lance">Lance</a>, an open-source columnar format designed for performant ML workloads.
+LanceDB's core is written in Rust 🦀 and is built using <a href="https://github.com/lancedb/lance">Lance</a>, an open-source columnar format designed for performant ML workloads.

 ## Quick Start

--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -1,10 +1,16 @@
-site_name: LanceDB Documentation
+site_name: LanceDB Docs
+repo_url: https://github.com/lancedb/lancedb
+repo_name: lancedb/lancedb
 docs_dir: src

 theme:
  name: "material"
+  logo: assets/logo.png
  features:
    - content.code.copy
+    - content.tabs.link
+  icon:
+    repo: fontawesome/brands/github

 plugins:
 - search
@@ -14,16 +20,10 @@ plugins:
        paths: [../python]
 - mkdocs-jupyter

-nav:
- Home: index.md
- Basics: basic.md
- Embeddings: embedding.md
- Indexing: ann_indexes.md
- Full-text search: fts.md
- Integrations: integrations.md
- Python API: python.md
-
 markdown_extensions:
+- admonition
+- pymdownx.superfences
+- pymdownx.details
 - pymdownx.highlight:
    anchor_linenums: true
    line_spans: __span
@@ -31,3 +31,25 @@ markdown_extensions:
 - pymdownx.inlinehilite
 - pymdownx.snippets
 - pymdownx.superfences
+- pymdownx.tabbed:
+    alternate_style: true
+
+nav:
+- Home: index.md
+- Basics: basic.md
+- Embeddings: embedding.md
+- Python full-text search: fts.md
+- Python integrations: integrations.md
+- Python examples:
+  - YouTube Transcript Search using OpenAI: notebooks/youtube_transcript_search.ipynb
+  - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
+  - Multimodal search using CLIP: notebooks/multimodal_search.ipynb
+- References:
+  - Vector Search: search.md
+  - Indexing: ann_indexes.md
+- API references:
+  - Python API: python/python.md
+  - Javascript API: javascript/modules.md
+
+extra_css:
+  - styles/global.css
--- a/docs/src/ann_indexes.md
+++ b/docs/src/ann_indexes.md
@@ -12,12 +12,13 @@ In the future we will look to automatically create and configure the ANN index.

 ## Creating an ANN Index

+=== "Python"
     Creating indexes is done via the [create_index](https://lancedb.github.io/lancedb/python/#lancedb.table.LanceTable.create_index) method.

     ```python
     import lancedb
     import numpy as np
-uri = "~/.lancedb"
+     uri = "data/sample-lancedb"
     db = lancedb.connect(uri)

     # Create 10,000 sample vectors
@@ -31,10 +32,23 @@ tbl = db.create_table("my_vectors", data=data)
     tbl.create_index(num_partitions=256, num_sub_vectors=96)
     ```

+=== "Javascript"
+     ```javascript
+     const vectordb = require('vectordb')
+     const db = await vectordb.connect('data/sample-lancedb')
+
+     let data = []
+     for (let i = 0; i < 10_000; i++) {
+         data.push({vector: Array(1536).fill(i), id: `${i}`, content: "", longId: `${i}`},)
+     }
+     const table = await db.createTable('vectors', data)
+     await table.create_index({ type: 'ivf_pq', column: 'vector', num_partitions: 256, num_sub_vectors: 96 })
+     ```
+
 Since `create_index` has a training step, it can take a few minutes to finish for large tables. You can control the index
 creation by providing the following parameters:

- **metric** (default: "L2"): The distance metric to use. By default we use euclidean distance. We also support cosine distance.
+- **metric** (default: "L2"): The distance metric to use. By default we use euclidean distance. We also support "cosine" distance.
 - **num_partitions** (default: 256): The number of partitions of the index. The number of partitions should be configured so each partition has 3-5K vectors. For example, a table
 with ~1M vectors should use 256 partitions. You can specify arbitrary number of partitions but powers of 2 is most conventional.
 A higher number leads to faster queries, but it makes index generation slower.
@@ -57,7 +71,7 @@ There are a couple of parameters that can be used to fine-tune the search:
  e.g., for 1M vectors divided into 256 partitions, if you're looking for top 20, then refine_factor=200 reranks the whole partition.<br/>
  Note: refine_factor is only applicable if an ANN index is present. If specified on a table without an ANN index, it is ignored.

-
+=== "Python"
     ```python
     tbl.search(np.random.random((768))) \
         .limit(2) \
@@ -70,6 +84,16 @@ tbl.search(np.random.random((768))) \
     1  [0.48587373, 0.269207, 0.15095535, 0.65531915,...  item 3953  108.393867
     ```

+=== "Javascript"
+     ```javascript
+     const results = await table
+         .search(Array(768).fill(1.2))
+         .limit(2)
+         .nprobes(20)
+         .refineFactor(10)
+         .execute()
+     ```
+
 The search will return the data requested in addition to the score of each item.

 **Note:** The score is the distance between the query vector and the element. A lower number means that the result is more relevant.
@@ -78,14 +102,24 @@ The search will return the data requested in addition to the score of each item.

 You can further filter the elements returned by a search using a where clause.

+=== "Python"
     ```python
     tbl.search(np.random.random((768))).where("item != 'item 1141'").to_df()
     ```

+=== "Javascript"
+     ```javascript
+     const results = await table
+         .search(Array(1536).fill(1.2))
+         .where("item != 'item 1141'")
+         .execute()
+     ```
+
 ### Projections (select clause)

 You can select the columns returned by the query using a select clause.

+=== "Python"
     ```python
     tbl.search(np.random.random((768))).select(["vector"]).to_df()
     vector      score
@@ -93,3 +127,11 @@ tbl.search(np.random.random((768))).select(["vector"]).to_df()
     1  [0.2525465, 0.01723831, 0.261568, 0.002007689,...  95.173485
     ...
     ```
+
+=== "Javascript"
+     ```javascript
+     const results = await table
+         .search(Array(1536).fill(1.2))
+         .select(["id"])
+         .execute()
+     ```
--- a/docs/src/assets/lancedb_embedded_explanation.png
+++ b/docs/src/assets/lancedb_embedded_explanation.png
--- a/docs/src/assets/lancedb_local_data_explanation.png
+++ b/docs/src/assets/lancedb_local_data_explanation.png
--- a/docs/src/assets/logo.png
+++ b/docs/src/assets/logo.png
--- a/docs/src/basic.md
+++ b/docs/src/basic.md
@@ -1,8 +1,26 @@
 # Basic LanceDB Functionality

+We'll cover the basics of using LanceDB on your local machine in this section.
+
+??? info "LanceDB runs embedded on your backend application, so there is no need to run a separate server."
+
+      <img src="../assets/lancedb_embedded_explanation.png" width="650px" />
+
+## Installation
+
+=== "Python"
+      ```shell
+      pip install lancedb
+      ```
+
+=== "Javascript"
+      ```shell
+      npm install vectordb
+      ```
+
 ## How to connect to a database

-In local mode, LanceDB stores data in a directory on your local machine. To connect to a local database, you can use the following code:
+=== "Python"
      ```python
      import lancedb
      uri = "~/.lancedb"
@@ -13,18 +31,27 @@ LanceDB will create the directory if it doesn't exist (including parent director

      If you need a reminder of the uri, use the `db.uri` property.

+=== "Javascript"
+      ```javascript
+      const lancedb = require("vectordb");
+
+      const uri = "~./lancedb";
+      const db = await lancedb.connect(uri);
+      ```
+      
+      LanceDB will create the directory if it doesn't exist (including parent directories).
+
+      If you need a reminder of the uri, you can call `db.uri()`.
+
 ## How to create a table

-To create a table, you can use the following code:
+=== "Python"
      ```python
      tbl = db.create_table("my_table",
                        data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
                              {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
      ```

-Under the hood, LanceDB is converting the input data into an Apache Arrow table
-and persisting it to disk in [Lance format](github.com/eto-ai/lance).
-
      If the table already exists, LanceDB will raise an error by default.
      If you want to overwrite the table, you can pass in `mode="overwrite"`
      to the `create_table` method.
@@ -37,9 +64,26 @@ df = pd.DataFrame([{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
      tbl = db.create_table("table_from_df", data=df)
      ```

+=== "Javascript"
+      ```javascript
+      const tb = await db.createTable("my_table",
+                        data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
+                              {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
+      ```
+      
+!!! warning
+
+      If the table already exists, LanceDB will raise an error by default.
+      If you want to overwrite the table, you can pass in `mode="overwrite"`
+      to the `createTable` function.
+
+??? info "Under the hood, LanceDB is converting the input data into an Apache Arrow table and persisting it to disk in [Lance format](https://www.github.com/lancedb/lance)."
+
 ## How to open an existing table

 Once created, you can open a table using the following code:
+
+=== "Python"
      ```python
      tbl = db.open_table("my_table")
      ```
@@ -47,29 +91,53 @@ tbl = db.open_table("my_table")
      If you forget the name of your table, you can always get a listing of all table names:

      ```python
-db.table_names()
+      print(db.table_names())
+      ```
+
+=== "Javascript"
+      ```javascript
+      const tbl = await db.openTable("my_table");
+      ```
+
+      If you forget the name of your table, you can always get a listing of all table names:
+
+      ```javascript
+      console.log(db.tableNames());
      ```

 ## How to add data to a table

 After a table has been created, you can always add more data to it using

+=== "Python"
      ```python
      df = pd.DataFrame([{"vector": [1.3, 1.4], "item": "fizz", "price": 100.0},
                        {"vector": [9.5, 56.2], "item": "buzz", "price": 200.0}])
      tbl.add(df)
      ```

+=== "Javascript"
+      ```javascript
+      await tbl.add([vector: [1.3, 1.4], item: "fizz", price: 100.0},
+              {vector: [9.5, 56.2], item: "buzz", price: 200.0}])
+      ```
+
 ## How to search for (approximate) nearest neighbors

 Once you've embedded the query, you can find its nearest neighbors using the following code:

+=== "Python"
      ```python
      tbl.search([100, 100]).limit(2).to_df()
      ```

      This returns a pandas DataFrame with the results.

+=== "Javascript"
+      ```javascript
+      const query = await tbl.search([100, 100]).limit(2).execute();
+      ```
+
 ## What's next

 This section covered the very basics of the LanceDB API.
--- a/docs/src/embedding.md
+++ b/docs/src/embedding.md
@@ -25,10 +25,13 @@ def embed_func(batch):
    return [model.encode(sentence) for sentence in batch]
 ```

+Please note that currently HuggingFace is only supported in the Python SDK.
+
 ### OpenAI example

 You can also use an external API like OpenAI to generate embeddings

+=== "Python"
      ```python
        import openai
        import os
@@ -46,8 +49,19 @@ def embed_func(c):
        return [record["embedding"] for record in rs["data"]]
      ```

+=== "Javascript"
+      ```javascript
+        const lancedb = require("vectordb");
+
+        // You need to provide an OpenAI API key
+        const apiKey = "sk-..."
+        // The embedding function will create embeddings for the 'text' column
+        const embedding = new lancedb.OpenAIEmbeddingFunction('text', apiKey)
+      ```
+
 ## Applying an embedding function

+=== "Python"
    Using an embedding function, you can apply it to raw data
    to generate embeddings for each row.

@@ -55,6 +69,7 @@ Say if you have a pandas DataFrame with a `text` column that you want to be embe
    you can use the [with_embeddings](https://lancedb.github.io/lancedb/python/#lancedb.embeddings.with_embeddings)
    function to generate embeddings and add create a combined pyarrow table:

+
     ```python
     import pandas as pd
     from lancedb.embeddings import with_embeddings
@@ -75,12 +90,31 @@ using the `batch_size` parameter to `with_embeddings`.
     LanceDB automatically wraps the function with retry and rate-limit logic to ensure the OpenAI
     API call is reliable.

+=== "Javascript"
+     Using an embedding function, you can apply it to raw data
+     to generate embeddings for each row.
+
+     You can just pass the embedding function created previously and LanceDB will automatically generate
+     embededings for your data.
+
+      ```javascript
+      const db = await lancedb.connect("/tmp/lancedb");
+      const data = [
+        { text: 'pepperoni'  },
+        { text: 'pineapple' }
+      ]
+
+      const table = await db.createTable('vectors', data, embedding)
+      ```
+
+
 ## Searching with an embedding function

 At inference time, you also need the same embedding function to embed your query text.
 It's important that you use the same model / function otherwise the embedding vectors don't
 belong in the same latent space and your results will be nonsensical.

+=== "Python"
     ```python
     query = "What's the best pizza topping?"
     query_vector = embed_func([query])[0]
@@ -89,6 +123,17 @@ tbl.search(query_vector).limit(10).to_df()

     The above snippet returns a pandas DataFrame with the 10 closest vectors to the query.

+=== "Javascript"
+     ```javascript
+      const results = await table
+        .search('What's the best pizza topping?')
+        .limit(10)
+        .execute()
+     ```
+
+     The above snippet returns an array of records with the 10 closest vectors to the query.
+
+
 ## Roadmap

 In the near future, we'll be integrating the embedding functions deeper into LanceDB<br/>.
--- a/docs/src/examples/langchain.md
+++ b/docs/src/examples/langchain.md
@@ -4,4 +4,4 @@

 <img id="splash" width="400" alt="langchain" src="https://user-images.githubusercontent.com/917119/236580868-61a246a9-e587-4c2b-8ae5-6fe5f7b7e81e.png">

-This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/notebooks/code_qa_bot.ipynb)
+This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/docs/src/notebooks/code_qa_bot.ipynb)
--- a/docs/src/examples/modal_langchain.md
+++ b/docs/src/examples/modal_langchain.md
@@ -0,0 +1,166 @@
+# Serverless QA Bot with Modal and LangChain
+
+## use LanceDB's LangChain integration with Modal to run a serverless app
+
+<img id="splash" width="400" alt="modal" src="https://github.com/lancedb/lancedb/assets/917119/7d80a40f-60d7-48a6-972f-dab05000eccf">
+
+We're going to build a QA bot for your documentation using LanceDB's LangChain integration and use Modal for deployment.
+
+Modal is an end-to-end compute platform for model inference, batch jobs, task queues, web apps and more. It's a great way to deploy your LanceDB models and apps.
+
+To get started, ensure that you have created an account and logged into [Modal](https://modal.com/). To follow along, the full source code is available on Github [here](https://github.com/lancedb/lancedb/blob/main/docs/src/examples/modal_langchain.py).
+
+### Setting up Modal
+
+We'll start by specifying our dependencies and creating a new Modal `Stub`:
+
+```python
+lancedb_image = Image.debian_slim().pip_install(
+    "lancedb",
+    "langchain",
+    "openai",
+    "pandas",
+    "tiktoken",
+    "unstructured",
+    "tabulate"
+)
+
+stub = Stub(
+    name="example-langchain-lancedb",
+    image=lancedb_image,
+    secrets=[Secret.from_name("my-openai-secret")],
+)
+```
+
+We're using Modal's Secrets injection to secure our OpenAI key. To set your own, you can access the Modal UI and enter your key.
+
+### Setting up caches for LanceDB and LangChain
+
+Next, we can setup some globals to cache our LanceDB database, as well as our LangChain docsource:
+
+```python
+docsearch = None
+docs_path = Path("docs.pkl")
+db_path = Path("lancedb")
+```
+
+### Downloading our dataset
+
+We're going use a pregenerated dataset, which stores HTML files of the Pandas 2.0 documentation. 
+You could switch this out for your own dataset.
+
+```python
+def download_docs():
+    pandas_docs = requests.get("https://eto-public.s3.us-west-2.amazonaws.com/datasets/pandas_docs/pandas.documentation.zip")
+    with open(Path("pandas.documentation.zip"), "wb") as f:
+        f.write(pandas_docs.content)
+
+    file = zipfile.ZipFile(Path("pandas.documentation.zip"))
+    file.extractall(path=Path("pandas_docs"))
+```
+
+### Pre-processing the dataset and generating metadata
+
+Once we've downloaded it, we want to parse and pre-process them using LangChain, and then vectorize them and store it in LanceDB.
+Let's first create a function that uses LangChains `UnstructuredHTMLLoader` to parse them.
+We can then add our own metadata to it and store it alongside the data, we'll later be able to use this for filtering metadata.
+
+```python
+def store_docs():
+    docs = []
+
+    if not docs_path.exists():
+        for p in Path("pandas_docs/pandas.documentation").rglob("*.html"):
+            if p.is_dir():
+                continue
+            loader = UnstructuredHTMLLoader(p)
+            raw_document = loader.load()
+
+            m = {}
+            m["title"] = get_document_title(raw_document[0])
+            m["version"] = "2.0rc0"
+            raw_document[0].metadata = raw_document[0].metadata | m
+            raw_document[0].metadata["source"] = str(raw_document[0].metadata["source"])
+            docs = docs + raw_document
+
+        with docs_path.open("wb") as fh:
+            pickle.dump(docs, fh)
+    else:
+        with docs_path.open("rb") as fh:
+            docs = pickle.load(fh)
+
+    return docs
+```
+
+### Simple LangChain chain for a QA bot
+
+Now we can create a simple LangChain chain for our QA bot. We'll use the `RecursiveCharacterTextSplitter` to split our documents into chunks, and then use the `OpenAIEmbeddings` to vectorize them.
+
+Lastly, we'll create a LanceDB table and store the vectorized documents in it, then create a `RetrievalQA` model from the chain and return it.
+
+```python
+def qanda_langchain(query):
+    download_docs()
+    docs = store_docs()
+
+    text_splitter = RecursiveCharacterTextSplitter(
+        chunk_size=1000,
+        chunk_overlap=200,
+    )
+    documents = text_splitter.split_documents(docs)
+    embeddings = OpenAIEmbeddings()
+
+    db = lancedb.connect(db_path) 
+    table = db.create_table("pandas_docs", data=[
+        {"vector": embeddings.embed_query("Hello World"), "text": "Hello World", "id": "1"}
+    ], mode="overwrite")
+    docsearch = LanceDB.from_documents(documents, embeddings, connection=table)
+    qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())
+    return qa.run(query)
+```
+
+### Creating our Modal entry points
+
+Now we can create our Modal entry points for our CLI and web endpoint:
+
+```python
+@stub.function()
+@web_endpoint(method="GET")
+def web(query: str):
+    answer = qanda_langchain(query)
+    return {
+        "answer": answer,
+    }
+    
+@stub.function()
+def cli(query: str):
+    answer = qanda_langchain(query)
+    print(answer)
+```
+
+# Testing it out!
+
+Testing the CLI:
+
+```bash
+modal run modal_langchain.py --query "What are the major differences in pandas 2.0?"
+```
+
+Testing the web endpoint:
+
+```bash
+modal serve modal_langchain.py
+```
+
+In the CLI, Modal will provide you a web endpoint. Copy this endpoint URI for the next step.
+Once this is served, then we can hit it with `curl`. 
+
+Note, the first time this runs, it will take a few minutes to download the dataset and vectorize it.
+An actual production example would pre-cache/load the dataset and vectorized documents prior
+
+```bash
+curl --get --data-urlencode "query=What are the major differences in pandas 2.0?" https://your-modal-endpoint-app.modal.run
+
+{"answer":" The major differences in pandas 2.0 include the ability to use any numpy numeric dtype in a Index, installing optional dependencies with pip extras, and enhancements, bug fixes, and performance improvements."}
+```
+
--- a/docs/src/examples/modal_langchain.py
+++ b/docs/src/examples/modal_langchain.py
@@ -0,0 +1,107 @@
+import sys
+from modal import Secret, Stub, Image, web_endpoint
+import lancedb
+import re
+import pickle
+import requests
+import zipfile
+from pathlib import Path
+
+from langchain.document_loaders import UnstructuredHTMLLoader
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.vectorstores import LanceDB
+from langchain.llms import OpenAI
+from langchain.chains import RetrievalQA
+
+lancedb_image = Image.debian_slim().pip_install(
+    "lancedb",
+    "langchain",
+    "openai",
+    "pandas",
+    "tiktoken",
+    "unstructured",
+    "tabulate"
+)
+
+stub = Stub(
+    name="example-langchain-lancedb",
+    image=lancedb_image,
+    secrets=[Secret.from_name("my-openai-secret")],
+)
+
+docsearch = None
+docs_path = Path("docs.pkl")
+db_path = Path("lancedb")
+
+def get_document_title(document):
+    m = str(document.metadata["source"])
+    title = re.findall("pandas.documentation(.*).html", m)
+    if title[0] is not None:
+        return(title[0])
+    return ''
+
+def download_docs():
+    pandas_docs = requests.get("https://eto-public.s3.us-west-2.amazonaws.com/datasets/pandas_docs/pandas.documentation.zip")
+    with open(Path("pandas.documentation.zip"), "wb") as f:
+        f.write(pandas_docs.content)
+
+    file = zipfile.ZipFile(Path("pandas.documentation.zip"))
+    file.extractall(path=Path("pandas_docs"))
+
+def store_docs():
+    docs = []
+
+    if not docs_path.exists():
+        for p in Path("pandas_docs/pandas.documentation").rglob("*.html"):
+            if p.is_dir():
+                continue
+            loader = UnstructuredHTMLLoader(p)
+            raw_document = loader.load()
+
+            m = {}
+            m["title"] = get_document_title(raw_document[0])
+            m["version"] = "2.0rc0"
+            raw_document[0].metadata = raw_document[0].metadata | m
+            raw_document[0].metadata["source"] = str(raw_document[0].metadata["source"])
+            docs = docs + raw_document
+
+        with docs_path.open("wb") as fh:
+            pickle.dump(docs, fh)
+    else:
+        with docs_path.open("rb") as fh:
+            docs = pickle.load(fh)
+
+    return docs
+
+def qanda_langchain(query):
+    download_docs()
+    docs = store_docs()
+
+    text_splitter = RecursiveCharacterTextSplitter(
+        chunk_size=1000,
+        chunk_overlap=200,
+    )
+    documents = text_splitter.split_documents(docs)
+    embeddings = OpenAIEmbeddings()
+
+    db = lancedb.connect(db_path) 
+    table = db.create_table("pandas_docs", data=[
+        {"vector": embeddings.embed_query("Hello World"), "text": "Hello World", "id": "1"}
+    ], mode="overwrite")
+    docsearch = LanceDB.from_documents(documents, embeddings, connection=table)
+    qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())
+    return qa.run(query)
+
+@stub.function()
+@web_endpoint(method="GET")
+def web(query: str):
+    answer = qanda_langchain(query)
+    return {
+        "answer": answer,
+    }
+
+@stub.function()
+def cli(query: str):
+    answer = qanda_langchain(query)
+    print(answer)
--- a/docs/src/examples/multimodal_search.md
+++ b/docs/src/examples/multimodal_search.md
@@ -0,0 +1,7 @@
+# Image multimodal search
+
+## Search through an image dataset using natural language, full text and SQL
+
+<img id="splash" width="400" alt="multimodal search" src="https://github.com/lancedb/lancedb/assets/917119/993a7c9f-be01-449d-942e-1ce1d4ed63af">
+
+This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/docs/src/notebooks/multimodal_search.ipynb)
--- a/docs/src/examples/youtube_transcript_search.md
+++ b/docs/src/examples/youtube_transcript_search.md
@@ -4,4 +4,4 @@

 <img id="splash" width="400" alt="youtube transcript search" src="https://user-images.githubusercontent.com/917119/236965568-def7394d-171c-45f2-939d-8edfeaadd88c.png">

-This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/notebooks/youtube_transcript_search.ipynb)
+This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/docs/src/notebooks/youtube_transcript_search.ipynb)
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -1,6 +1,6 @@
 # Welcome to LanceDB's Documentation

-LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrivial, filtering and management of embeddings.
+LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrevial, filtering and management of embeddings.

 The key features of LanceDB include:

@@ -8,38 +8,52 @@ The key features of LanceDB include:

 * Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).

-* Native Python and Javascript/Typescript support (coming soon).
+* Native Python and Javascript/Typescript support.

 * Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.

 * Ecosystem integrations with [LangChain 🦜️🔗](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lanecdb.html), [LlamaIndex 🦙](https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html), Apache-Arrow, Pandas, Polars, DuckDB and more on the way.

-LanceDB's core is written in Rust 🦀 and is built using Lance, an open-source columnar format designed for performant ML workloads.
+LanceDB's core is written in Rust 🦀 and is built using <a href="https://github.com/lancedb/lance">Lance</a>, an open-source columnar format designed for performant ML workloads.

+## Quick Start

-## Installation
-
+=== "Python"
      ```shell
      pip install lancedb
      ```

-## Quickstart
-
      ```python
      import lancedb

-db = lancedb.connect(".")
+      uri = "/tmp/lancedb"
+      db = lancedb.connect(uri)
      table = db.create_table("my_table",
                              data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
                                    {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
      result = table.search([100, 100]).limit(2).to_df()
      ```

-## Complete Demos
+=== "Javascript"
+      ```shell
+      npm install vectordb
+      ```

-We will be adding completed demo apps built using LanceDB.
- [YouTube Transcript Search](../../notebooks/youtube_transcript_search.ipynb)
+      ```javascript
+      const lancedb = require("vectordb");

+      const uri = "/tmp/lancedb";
+      const db = await lancedb.connect(uri);
+      const table = await db.createTable("my_table", 
+            [{ id: 1, vector: [3.1, 4.1], item: "foo", price: 10.0 },
+            { id: 2, vector: [5.9, 26.5], item: "bar", price: 20.0 }])
+      const results = await table.search([100, 100]).limit(2).execute();
+      ```
+
+## Complete Demos (Python)
+- [YouTube Transcript Search](notebooks/youtube_transcript_search.ipynb)
+- [Documentation QA Bot using LangChain](notebooks/code_qa_bot.ipynb)
+- [Multimodal search using CLIP](notebooks/multimodal_search.ipynb)

 ## Documentation Quick Links
 * [`Basic Operations`](basic.md) - basic functionality of LanceDB.
--- a/docs/src/javascript/.nojekyll
+++ b/docs/src/javascript/.nojekyll
@@ -0,0 +1 @@
+TypeDoc added this file to prevent GitHub Pages from using Jekyll. You can turn off this behavior by setting the `githubPages` option to false.
--- a/docs/src/javascript/README.md
+++ b/docs/src/javascript/README.md
@@ -0,0 +1,51 @@
+vectordb / [Exports](modules.md)
+
+# LanceDB
+
+A JavaScript / Node.js library for [LanceDB](https://github.com/lancedb/lancedb).
+
+## Installation
+
+```bash
+npm install vectordb
+```
+
+## Usage
+
+### Basic Example
+
+```javascript
+const lancedb = require('vectordb');
+const db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>');
+const table = await db.openTable('my_table');
+const query = await table.search([0.1, 0.3]).setLimit(20).execute();
+console.log(results);
+```
+
+The [examples](./examples) folder contains complete examples.
+
+## Development
+
+The LanceDB javascript is built with npm:
+
+```bash
+npm run tsc
+```
+
+Run the tests with
+
+```bash
+npm test
+```
+
+To run the linter and have it automatically fix all errors
+
+```bash
+npm run lint -- --fix
+```
+
+To build documentation
+
+```bash
+npx typedoc --plugin typedoc-plugin-markdown --out ../docs/src/javascript src/index.ts
+```
--- a/docs/src/javascript/classes/Connection.md
+++ b/docs/src/javascript/classes/Connection.md
@@ -0,0 +1,211 @@
+[vectordb](../README.md) / [Exports](../modules.md) / Connection
+
+# Class: Connection
+
+A connection to a LanceDB database.
+
+## Table of contents
+
+### Constructors
+
+- [constructor](Connection.md#constructor)
+
+### Properties
+
+- [\_db](Connection.md#_db)
+- [\_uri](Connection.md#_uri)
+
+### Accessors
+
+- [uri](Connection.md#uri)
+
+### Methods
+
+- [createTable](Connection.md#createtable)
+- [createTableArrow](Connection.md#createtablearrow)
+- [openTable](Connection.md#opentable)
+- [tableNames](Connection.md#tablenames)
+
+## Constructors
+
+### constructor
+
+• **new Connection**(`db`, `uri`)
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `db` | `any` |
+| `uri` | `string` |
+
+#### Defined in
+
+[index.ts:46](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L46)
+
+## Properties
+
+### \_db
+
+• `Private` `Readonly` **\_db**: `any`
+
+#### Defined in
+
+[index.ts:44](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L44)
+
+___
+
+### \_uri
+
+• `Private` `Readonly` **\_uri**: `string`
+
+#### Defined in
+
+[index.ts:43](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L43)
+
+## Accessors
+
+### uri
+
+• `get` **uri**(): `string`
+
+#### Returns
+
+`string`
+
+#### Defined in
+
+[index.ts:51](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L51)
+
+## Methods
+
+### createTable
+
+▸ **createTable**(`name`, `data`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+Creates a new Table and initialize it with new data.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `name` | `string` | The name of the table. |
+| `data` | `Record`<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the Table |
+
+#### Returns
+
+`Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+#### Defined in
+
+[index.ts:91](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L91)
+
+▸ **createTable**<`T`\>(`name`, `data`, `embeddings`): `Promise`<[`Table`](Table.md)<`T`\>\>
+
+Creates a new Table and initialize it with new data.
+
+#### Type parameters
+
+| Name |
+| :------ |
+| `T` |
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `name` | `string` | The name of the table. |
+| `data` | `Record`<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the Table |
+| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use on this Table |
+
+#### Returns
+
+`Promise`<[`Table`](Table.md)<`T`\>\>
+
+#### Defined in
+
+[index.ts:99](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L99)
+
+___
+
+### createTableArrow
+
+▸ **createTableArrow**(`name`, `table`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `name` | `string` |
+| `table` | `Table`<`any`\> |
+
+#### Returns
+
+`Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+#### Defined in
+
+[index.ts:109](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L109)
+
+___
+
+### openTable
+
+▸ **openTable**(`name`): `Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+Open a table in the database.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `name` | `string` | The name of the table. |
+
+#### Returns
+
+`Promise`<[`Table`](Table.md)<`number`[]\>\>
+
+#### Defined in
+
+[index.ts:67](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L67)
+
+▸ **openTable**<`T`\>(`name`, `embeddings`): `Promise`<[`Table`](Table.md)<`T`\>\>
+
+Open a table in the database.
+
+#### Type parameters
+
+| Name |
+| :------ |
+| `T` |
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `name` | `string` | The name of the table. |
+| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use on this Table |
+
+#### Returns
+
+`Promise`<[`Table`](Table.md)<`T`\>\>
+
+#### Defined in
+
+[index.ts:74](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L74)
+
+___
+
+### tableNames
+
+▸ **tableNames**(): `Promise`<`string`[]\>
+
+Get the names of all tables in the database.
+
+#### Returns
+
+`Promise`<`string`[]\>
+
+#### Defined in
+
+[index.ts:58](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L58)
--- a/docs/src/javascript/classes/OpenAIEmbeddingFunction.md
+++ b/docs/src/javascript/classes/OpenAIEmbeddingFunction.md
@@ -0,0 +1,105 @@
+[vectordb](../README.md) / [Exports](../modules.md) / OpenAIEmbeddingFunction
+
+# Class: OpenAIEmbeddingFunction
+
+An embedding function that automatically creates vector representation for a given column.
+
+## Implements
+
+- [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`string`\>
+
+## Table of contents
+
+### Constructors
+
+- [constructor](OpenAIEmbeddingFunction.md#constructor)
+
+### Properties
+
+- [\_modelName](OpenAIEmbeddingFunction.md#_modelname)
+- [\_openai](OpenAIEmbeddingFunction.md#_openai)
+- [sourceColumn](OpenAIEmbeddingFunction.md#sourcecolumn)
+
+### Methods
+
+- [embed](OpenAIEmbeddingFunction.md#embed)
+
+## Constructors
+
+### constructor
+
+• **new OpenAIEmbeddingFunction**(`sourceColumn`, `openAIKey`, `modelName?`)
+
+#### Parameters
+
+| Name | Type | Default value |
+| :------ | :------ | :------ |
+| `sourceColumn` | `string` | `undefined` |
+| `openAIKey` | `string` | `undefined` |
+| `modelName` | `string` | `'text-embedding-ada-002'` |
+
+#### Defined in
+
+[embedding/openai.ts:21](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L21)
+
+## Properties
+
+### \_modelName
+
+• `Private` `Readonly` **\_modelName**: `string`
+
+#### Defined in
+
+[embedding/openai.ts:19](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L19)
+
+___
+
+### \_openai
+
+• `Private` `Readonly` **\_openai**: `any`
+
+#### Defined in
+
+[embedding/openai.ts:18](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L18)
+
+___
+
+### sourceColumn
+
+• **sourceColumn**: `string`
+
+The name of the column that will be used as input for the Embedding Function.
+
+#### Implementation of
+
+[EmbeddingFunction](../interfaces/EmbeddingFunction.md).[sourceColumn](../interfaces/EmbeddingFunction.md#sourcecolumn)
+
+#### Defined in
+
+[embedding/openai.ts:50](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L50)
+
+## Methods
+
+### embed
+
+▸ **embed**(`data`): `Promise`<`number`[][]\>
+
+Creates a vector representation for the given values.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `data` | `string`[] |
+
+#### Returns
+
+`Promise`<`number`[][]\>
+
+#### Implementation of
+
+[EmbeddingFunction](../interfaces/EmbeddingFunction.md).[embed](../interfaces/EmbeddingFunction.md#embed)
+
+#### Defined in
+
+[embedding/openai.ts:38](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/openai.ts#L38)
--- a/docs/src/javascript/classes/Query.md
+++ b/docs/src/javascript/classes/Query.md
@@ -0,0 +1,299 @@
+[vectordb](../README.md) / [Exports](../modules.md) / Query
+
+# Class: Query<T\>
+
+A builder for nearest neighbor queries for LanceDB.
+
+## Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `number`[] |
+
+## Table of contents
+
+### Constructors
+
+- [constructor](Query.md#constructor)
+
+### Properties
+
+- [\_columns](Query.md#_columns)
+- [\_embeddings](Query.md#_embeddings)
+- [\_filter](Query.md#_filter)
+- [\_limit](Query.md#_limit)
+- [\_metricType](Query.md#_metrictype)
+- [\_nprobes](Query.md#_nprobes)
+- [\_query](Query.md#_query)
+- [\_queryVector](Query.md#_queryvector)
+- [\_refineFactor](Query.md#_refinefactor)
+- [\_tbl](Query.md#_tbl)
+
+### Methods
+
+- [execute](Query.md#execute)
+- [filter](Query.md#filter)
+- [limit](Query.md#limit)
+- [metricType](Query.md#metrictype)
+- [nprobes](Query.md#nprobes)
+- [refineFactor](Query.md#refinefactor)
+
+## Constructors
+
+### constructor
+
+• **new Query**<`T`\>(`tbl`, `query`, `embeddings?`)
+
+#### Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `number`[] |
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `tbl` | `any` |
+| `query` | `T` |
+| `embeddings?` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> |
+
+#### Defined in
+
+[index.ts:241](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L241)
+
+## Properties
+
+### \_columns
+
+• `Private` `Optional` `Readonly` **\_columns**: `string`[]
+
+#### Defined in
+
+[index.ts:236](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L236)
+
+___
+
+### \_embeddings
+
+• `Private` `Optional` `Readonly` **\_embeddings**: [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\>
+
+#### Defined in
+
+[index.ts:239](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L239)
+
+___
+
+### \_filter
+
+• `Private` `Optional` **\_filter**: `string`
+
+#### Defined in
+
+[index.ts:237](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L237)
+
+___
+
+### \_limit
+
+• `Private` **\_limit**: `number`
+
+#### Defined in
+
+[index.ts:233](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L233)
+
+___
+
+### \_metricType
+
+• `Private` `Optional` **\_metricType**: [`MetricType`](../enums/MetricType.md)
+
+#### Defined in
+
+[index.ts:238](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L238)
+
+___
+
+### \_nprobes
+
+• `Private` **\_nprobes**: `number`
+
+#### Defined in
+
+[index.ts:235](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L235)
+
+___
+
+### \_query
+
+• `Private` `Readonly` **\_query**: `T`
+
+#### Defined in
+
+[index.ts:231](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L231)
+
+___
+
+### \_queryVector
+
+• `Private` `Optional` **\_queryVector**: `number`[]
+
+#### Defined in
+
+[index.ts:232](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L232)
+
+___
+
+### \_refineFactor
+
+• `Private` `Optional` **\_refineFactor**: `number`
+
+#### Defined in
+
+[index.ts:234](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L234)
+
+___
+
+### \_tbl
+
+• `Private` `Readonly` **\_tbl**: `any`
+
+#### Defined in
+
+[index.ts:230](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L230)
+
+## Methods
+
+### execute
+
+▸ **execute**<`T`\>(): `Promise`<`T`[]\>
+
+Execute the query and return the results as an Array of Objects
+
+#### Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `Record`<`string`, `unknown`\> |
+
+#### Returns
+
+`Promise`<`T`[]\>
+
+#### Defined in
+
+[index.ts:301](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L301)
+
+___
+
+### filter
+
+▸ **filter**(`value`): [`Query`](Query.md)<`T`\>
+
+A filter statement to be applied to this query.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `value` | `string` | A filter in the same format used by a sql WHERE clause. |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:284](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L284)
+
+___
+
+### limit
+
+▸ **limit**(`value`): [`Query`](Query.md)<`T`\>
+
+Sets the number of results that will be returned
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `value` | `number` | number of results |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:257](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L257)
+
+___
+
+### metricType
+
+▸ **metricType**(`value`): [`Query`](Query.md)<`T`\>
+
+The MetricType used for this Query.
+
+**`See`**
+
+MetricType for the different options
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `value` | [`MetricType`](../enums/MetricType.md) | The metric to the. |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:293](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L293)
+
+___
+
+### nprobes
+
+▸ **nprobes**(`value`): [`Query`](Query.md)<`T`\>
+
+The number of probes used. A higher number makes search more accurate but also slower.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `value` | `number` | The number of probes used. |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:275](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L275)
+
+___
+
+### refineFactor
+
+▸ **refineFactor**(`value`): [`Query`](Query.md)<`T`\>
+
+Refine the results by reading extra elements and re-ranking them in memory.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `value` | `number` | refine factor to use in this query. |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:266](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L266)
--- a/docs/src/javascript/classes/Table.md
+++ b/docs/src/javascript/classes/Table.md
@@ -0,0 +1,215 @@
+[vectordb](../README.md) / [Exports](../modules.md) / Table
+
+# Class: Table<T\>
+
+## Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `number`[] |
+
+## Table of contents
+
+### Constructors
+
+- [constructor](Table.md#constructor)
+
+### Properties
+
+- [\_embeddings](Table.md#_embeddings)
+- [\_name](Table.md#_name)
+- [\_tbl](Table.md#_tbl)
+
+### Accessors
+
+- [name](Table.md#name)
+
+### Methods
+
+- [add](Table.md#add)
+- [create\_index](Table.md#create_index)
+- [overwrite](Table.md#overwrite)
+- [search](Table.md#search)
+
+## Constructors
+
+### constructor
+
+• **new Table**<`T`\>(`tbl`, `name`)
+
+#### Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `number`[] |
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `tbl` | `any` |
+| `name` | `string` |
+
+#### Defined in
+
+[index.ts:121](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L121)
+
+• **new Table**<`T`\>(`tbl`, `name`, `embeddings`)
+
+#### Type parameters
+
+| Name | Type |
+| :------ | :------ |
+| `T` | `number`[] |
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `tbl` | `any` |  |
+| `name` | `string` |  |
+| `embeddings` | [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\> | An embedding function to use when interacting with this table |
+
+#### Defined in
+
+[index.ts:127](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L127)
+
+## Properties
+
+### \_embeddings
+
+• `Private` `Optional` `Readonly` **\_embeddings**: [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)<`T`\>
+
+#### Defined in
+
+[index.ts:119](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L119)
+
+___
+
+### \_name
+
+• `Private` `Readonly` **\_name**: `string`
+
+#### Defined in
+
+[index.ts:118](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L118)
+
+___
+
+### \_tbl
+
+• `Private` `Readonly` **\_tbl**: `any`
+
+#### Defined in
+
+[index.ts:117](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L117)
+
+## Accessors
+
+### name
+
+• `get` **name**(): `string`
+
+#### Returns
+
+`string`
+
+#### Defined in
+
+[index.ts:134](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L134)
+
+## Methods
+
+### add
+
+▸ **add**(`data`): `Promise`<`number`\>
+
+Insert records into this Table.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `data` | `Record`<`string`, `unknown`\>[] | Records to be inserted into the Table |
+
+#### Returns
+
+`Promise`<`number`\>
+
+The number of rows added to the table
+
+#### Defined in
+
+[index.ts:152](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L152)
+
+___
+
+### create\_index
+
+▸ **create_index**(`indexParams`): `Promise`<`any`\>
+
+Create an ANN index on this Table vector index.
+
+**`See`**
+
+VectorIndexParams.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `indexParams` | `IvfPQIndexConfig` | The parameters of this Index, |
+
+#### Returns
+
+`Promise`<`any`\>
+
+#### Defined in
+
+[index.ts:171](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L171)
+
+___
+
+### overwrite
+
+▸ **overwrite**(`data`): `Promise`<`number`\>
+
+Insert records into this Table, replacing its contents.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `data` | `Record`<`string`, `unknown`\>[] | Records to be inserted into the Table |
+
+#### Returns
+
+`Promise`<`number`\>
+
+The number of rows added to the table
+
+#### Defined in
+
+[index.ts:162](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L162)
+
+___
+
+### search
+
+▸ **search**(`query`): [`Query`](Query.md)<`T`\>
+
+Creates a search query to find the nearest neighbors of the given search term
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `query` | `T` | The query search term |
+
+#### Returns
+
+[`Query`](Query.md)<`T`\>
+
+#### Defined in
+
+[index.ts:142](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L142)
--- a/docs/src/javascript/enums/MetricType.md
+++ b/docs/src/javascript/enums/MetricType.md
@@ -0,0 +1,36 @@
+[vectordb](../README.md) / [Exports](../modules.md) / MetricType
+
+# Enumeration: MetricType
+
+Distance metrics type.
+
+## Table of contents
+
+### Enumeration Members
+
+- [Cosine](MetricType.md#cosine)
+- [L2](MetricType.md#l2)
+
+## Enumeration Members
+
+### Cosine
+
+• **Cosine** = ``"cosine"``
+
+Cosine distance
+
+#### Defined in
+
+[index.ts:341](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L341)
+
+___
+
+### L2
+
+• **L2** = ``"l2"``
+
+Euclidean distance
+
+#### Defined in
+
+[index.ts:336](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L336)
--- a/docs/src/javascript/enums/WriteMode.md
+++ b/docs/src/javascript/enums/WriteMode.md
@@ -0,0 +1,30 @@
+[vectordb](../README.md) / [Exports](../modules.md) / WriteMode
+
+# Enumeration: WriteMode
+
+## Table of contents
+
+### Enumeration Members
+
+- [Append](WriteMode.md#append)
+- [Overwrite](WriteMode.md#overwrite)
+
+## Enumeration Members
+
+### Append
+
+• **Append** = ``"append"``
+
+#### Defined in
+
+[index.ts:326](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L326)
+
+___
+
+### Overwrite
+
+• **Overwrite** = ``"overwrite"``
+
+#### Defined in
+
+[index.ts:325](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L325)
--- a/docs/src/javascript/interfaces/EmbeddingFunction.md
+++ b/docs/src/javascript/interfaces/EmbeddingFunction.md
@@ -0,0 +1,60 @@
+[vectordb](../README.md) / [Exports](../modules.md) / EmbeddingFunction
+
+# Interface: EmbeddingFunction<T\>
+
+An embedding function that automatically creates vector representation for a given column.
+
+## Type parameters
+
+| Name |
+| :------ |
+| `T` |
+
+## Implemented by
+
+- [`OpenAIEmbeddingFunction`](../classes/OpenAIEmbeddingFunction.md)
+
+## Table of contents
+
+### Properties
+
+- [embed](EmbeddingFunction.md#embed)
+- [sourceColumn](EmbeddingFunction.md#sourcecolumn)
+
+## Properties
+
+### embed
+
+• **embed**: (`data`: `T`[]) => `Promise`<`number`[][]\>
+
+#### Type declaration
+
+▸ (`data`): `Promise`<`number`[][]\>
+
+Creates a vector representation for the given values.
+
+##### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `data` | `T`[] |
+
+##### Returns
+
+`Promise`<`number`[][]\>
+
+#### Defined in
+
+[embedding/embedding_function.ts:27](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/embedding_function.ts#L27)
+
+___
+
+### sourceColumn
+
+• **sourceColumn**: `string`
+
+The name of the column that will be used as input for the Embedding Function.
+
+#### Defined in
+
+[embedding/embedding_function.ts:22](https://github.com/lancedb/lancedb/blob/31dab97/node/src/embedding/embedding_function.ts#L22)
--- a/docs/src/javascript/modules.md
+++ b/docs/src/javascript/modules.md
@@ -0,0 +1,61 @@
+[vectordb](README.md) / Exports
+
+# vectordb
+
+## Table of contents
+
+### Enumerations
+
+- [MetricType](enums/MetricType.md)
+- [WriteMode](enums/WriteMode.md)
+
+### Classes
+
+- [Connection](classes/Connection.md)
+- [OpenAIEmbeddingFunction](classes/OpenAIEmbeddingFunction.md)
+- [Query](classes/Query.md)
+- [Table](classes/Table.md)
+
+### Interfaces
+
+- [EmbeddingFunction](interfaces/EmbeddingFunction.md)
+
+### Type Aliases
+
+- [VectorIndexParams](modules.md#vectorindexparams)
+
+### Functions
+
+- [connect](modules.md#connect)
+
+## Type Aliases
+
+### VectorIndexParams
+
+Ƭ **VectorIndexParams**: `IvfPQIndexConfig`
+
+#### Defined in
+
+[index.ts:224](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L224)
+
+## Functions
+
+### connect
+
+▸ **connect**(`uri`): `Promise`<[`Connection`](classes/Connection.md)\>
+
+Connect to a LanceDB instance at the given URI
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `uri` | `string` | The uri of the database. |
+
+#### Returns
+
+`Promise`<[`Connection`](classes/Connection.md)\>
+
+#### Defined in
+
+[index.ts:34](https://github.com/lancedb/lancedb/blob/31dab97/node/src/index.ts#L34)
--- a/docs/src/notebooks/code_qa_bot.ipynb
+++ b/docs/src/notebooks/code_qa_bot.ipynb
@@ -72,6 +72,8 @@
    "import lancedb\n",
    "import re\n",
    "import pickle\n",
+    "import requests\n",
+    "import zipfile\n",
    "from pathlib import Path\n",
    "\n",
    "from langchain.document_loaders import UnstructuredHTMLLoader\n",
@@ -85,10 +87,25 @@
  {
   "attachments": {},
   "cell_type": "markdown",
-   "id": "6ccf9b2b",
+   "id": "56cc6d50",
   "metadata": {},
   "source": [
-    "You can download the Pandas documentation from https://pandas.pydata.org/docs/. To make sure we're not littering our repo with docs, we won't include it in the LanceDB repo, so download this and store it locally first."
+    "To make this easier, we've downloaded Pandas documentation and stored the raw HTML files for you to download. We'll download them and then use LangChain's HTML document readers to parse them and store them in LanceDB as a vector store, along with relevant metadata."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7da77e75",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pandas_docs = requests.get(\"https://eto-public.s3.us-west-2.amazonaws.com/datasets/pandas_docs/pandas.documentation.zip\")\n",
+    "with open('/tmp/pandas.documentation.zip', 'wb') as f:\n",
+    "    f.write(pandas_docs.content)\n",
+    "\n",
+    "file = zipfile.ZipFile(\"/tmp/pandas.documentation.zip\")\n",
+    "file.extractall(path=\"/tmp/pandas_docs\")"
   ]
  },
  {
@@ -137,7 +154,8 @@
    "docs = []\n",
    "\n",
    "if not docs_path.exists():\n",
-    "    for p in Path(\"./pandas.documentation\").rglob(\"*.html\"):\n",
+    "    for p in Path(\"/tmp/pandas_docs/pandas.documentation\").rglob(\"*.html\"):\n",
+    "        print(p)\n",
    "        if p.is_dir():\n",
    "            continue\n",
    "        loader = UnstructuredHTMLLoader(p)\n",
--- a/docs/src/notebooks/diffusiondb/datagen.py
+++ b/docs/src/notebooks/diffusiondb/datagen.py
--- a/docs/src/notebooks/diffusiondb/requirements.txt
+++ b/docs/src/notebooks/diffusiondb/requirements.txt
--- a/docs/src/notebooks/multimodal_search.ipynb
+++ b/docs/src/notebooks/multimodal_search.ipynb
@@ -25,7 +25,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 60,
+   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -81,7 +81,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 62,
+   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -98,7 +98,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 63,
+   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -125,20 +125,41 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 64,
+   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_image_vectors(query):\n",
    "    emb = embed_func(query)\n",
-    "    return _extract(tbl.search(emb).limit(9).to_df())\n",
+    "    code = (\n",
+    "        \"import lancedb\\n\"\n",
+    "        \"db = lancedb.connect('~/datasets/demo')\\n\"\n",
+    "        \"tbl = db.open_table('diffusiondb')\\n\\n\"\n",
+    "        f\"embedding = embed_func('{query}')\\n\"\n",
+    "        \"tbl.search(embedding).limit(9).to_df()\"\n",
+    "    )\n",
+    "    return (_extract(tbl.search(emb).limit(9).to_df()), code)\n",
    "\n",
    "def find_image_keywords(query):\n",
-    "    return _extract(tbl.search(query).limit(9).to_df())\n",
+    "    code = (\n",
+    "        \"import lancedb\\n\"\n",
+    "        \"db = lancedb.connect('~/datasets/demo')\\n\"\n",
+    "        \"tbl = db.open_table('diffusiondb')\\n\\n\"\n",
+    "        f\"tbl.search('{query}').limit(9).to_df()\"\n",
+    "    )\n",
+    "    return (_extract(tbl.search(query).limit(9).to_df()), code)\n",
    "\n",
    "def find_image_sql(query):\n",
+    "    code = (\n",
+    "        \"import lancedb\\n\"\n",
+    "        \"import duckdb\\n\"\n",
+    "        \"db = lancedb.connect('~/datasets/demo')\\n\"\n",
+    "        \"tbl = db.open_table('diffusiondb')\\n\\n\"\n",
+    "        \"diffusiondb = tbl.to_lance()\\n\"\n",
+    "        f\"duckdb.sql('{query}').to_df()\"\n",
+    "    )    \n",
    "    diffusiondb = tbl.to_lance()\n",
-    "    return _extract(duckdb.query(query).to_df())\n",
+    "    return (_extract(duckdb.sql(query).to_df()), code)\n",
    "\n",
    "def _extract(df):\n",
    "    image_col = \"image\"\n",
@@ -154,14 +175,14 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 65,
+   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
-      "Running on local URL:  http://127.0.0.1:7867\n",
+      "Running on local URL:  http://127.0.0.1:7881\n",
      "\n",
      "To create a public link, set `share=True` in `launch()`.\n"
     ]
@@ -169,7 +190,7 @@
    {
     "data": {
      "text/html": [
-       "<div><iframe src=\"http://127.0.0.1:7867/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+       "<div><iframe src=\"http://127.0.0.1:7881/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
@@ -182,7 +203,7 @@
     "data": {
      "text/plain": []
     },
-     "execution_count": 65,
+     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
@@ -192,7 +213,6 @@
    "\n",
    "\n",
    "with gr.Blocks() as demo:\n",
-    "\n",
    "    with gr.Row():\n",
    "        with gr.Tab(\"Embeddings\"):\n",
    "            vector_query = gr.Textbox(value=\"portraits of a person\", show_label=False)\n",
@@ -204,16 +224,25 @@
    "            sql_query = gr.Textbox(value=\"SELECT * from diffusiondb WHERE image_nsfw >= 2 LIMIT 9\", show_label=False)\n",
    "            b3 = gr.Button(\"Submit\")\n",
    "    with gr.Row():\n",
+    "        code = gr.Code(label=\"Code\", language=\"python\")\n",
+    "    with gr.Row():\n",
    "        gallery = gr.Gallery(\n",
    "                label=\"Found images\", show_label=False, elem_id=\"gallery\"\n",
    "            ).style(columns=[3], rows=[3], object_fit=\"contain\", height=\"auto\")   \n",
    "        \n",
-    "    b1.click(find_image_vectors, inputs=vector_query, outputs=gallery)\n",
-    "    b2.click(find_image_keywords, inputs=keyword_query, outputs=gallery)\n",
-    "    b3.click(find_image_sql, inputs=sql_query, outputs=gallery)\n",
+    "    b1.click(find_image_vectors, inputs=vector_query, outputs=[gallery, code])\n",
+    "    b2.click(find_image_keywords, inputs=keyword_query, outputs=[gallery, code])\n",
+    "    b3.click(find_image_sql, inputs=sql_query, outputs=[gallery, code])\n",
    "    \n",
    "demo.launch()"
   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
  }
 ],
 "metadata": {
--- a/docs/src/notebooks/youtube_transcript_search.ipynb
+++ b/docs/src/notebooks/youtube_transcript_search.ipynb
--- a/docs/src/python/python.md
+++ b/docs/src/python/python.md
--- a/docs/src/search.md
+++ b/docs/src/search.md
@@ -0,0 +1,85 @@
+# Vector Search
+
+`Vector Search` finds the nearest vectors from the database.
+In a recommendation system or search engine, you can find similar products from
+the one you searched.
+In LLM and other AI applications,
+each data point can be [presented by the embeddings generated from some models](embedding.md),
+it returns the most relevant features.
+
+A search in high-dimensional vector space, is to find `K-Nearest-Neighbors (KNN)` of the query vector.
+
+## Metric
+
+In LanceDB, a `Metric` is the way to describe the distance between a pair of vectors.
+Currently, we support the following metrics:
+
+| Metric      | Description                          |
+| ----------- | ------------------------------------ |
+| `L2`        | [Euclidean / L2 distance](https://en.wikipedia.org/wiki/Euclidean_distance) |
+| `Cosine`    | [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity)|
+
+
+## Search
+
+### Flat Search
+
+
+If there is no [vector index is created](ann_indexes.md), LanceDB will just brute-force scan
+the vector column and compute the distance.
+
+=== "Python"
+
+    ```python
+    import lancedb
+    db = lancedb.connect("data/sample-lancedb")
+
+    tbl = db.open_table("my_vectors")
+
+    df = tbl.search(np.random.random((768)))
+        .limit(10)
+        .to_df()
+    ```
+
+=== "JavaScript"
+
+    ```javascript
+    const vectordb = require('vectordb')
+    const db = await vectordb.connect('data/sample-lancedb')
+
+    tbl = db.open_table("my_vectors")
+
+    const results = await tbl.search(Array(768))
+        .limit(20)
+        .execute()
+    ```
+
+By default, `l2` will be used as `Metric` type. You can customize the metric type
+as well.
+
+=== "Python"
+
+    ```python
+    df = tbl.search(np.random.random((768)))
+        .metric("cosine")
+        .limit(10)
+        .to_df()
+    ```
+
+=== "JavaScript"
+
+    ```javascript
+    const vectordb = require('vectordb')
+    const db = await vectordb.connect('data/sample-lancedb')
+
+    tbl = db.open_table("my_vectors")
+
+    const results = await tbl.search(Array(768))
+        .metric("cosine")
+        .limit(20)
+        .execute()
+    ```
+
+### Search with Vector Index.
+
+See [ANN Index](ann_indexes.md) for more details.
--- a/docs/src/styles/global.css
+++ b/docs/src/styles/global.css
@@ -0,0 +1,6 @@
+:root {
+    --md-primary-fg-color:        #625eff;
+    --md-primary-fg-color--dark:  #4338ca;
+    --md-text-font: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
+    --md-code-font: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
+}
--- a/node/CHANGELOG.md
+++ b/node/CHANGELOG.md
@@ -0,0 +1,58 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.4] - 2023-06-03
+
+### Added
+
+- Select / Project query API
+
+### Changed
+
+-  Deprecated created_index in favor of createIndex
+
+## [0.1.3] - 2023-06-01
+
+### Added
+
+- Support S3 and Google Cloud Storage
+- Embedding functions support
+- OpenAI embedding function
+
+## [0.1.2] - 2023-05-27
+
+### Added
+
+- Append records API
+- Extra query params to to nodejs client
+- Create_index API
+ 
+### Fixed
+
+- bugfix: string columns should be converted to Utf8Array (#94)
+
+## [0.1.1] - 2023-05-16
+
+### Added
+
+- create_table API
+- limit parameter for queries
+- Typescript / JavaScript examples
+- Linux support
+
+## [0.1.0] - 2023-05-16
+
+### Added
+
+- Initial  JavaScript / Node.js library for LanceDB
+- Read-only api to query LanceDB datasets
+- Supports macOS arm only
+
+## [pre-0.1.0]
+
+- Various prototypes / test builds
+
--- a/node/README.md
+++ b/node/README.md
@@ -41,3 +41,9 @@ To run the linter and have it automatically fix all errors
 ```bash
 npm run lint -- --fix
 ```
+
+To build documentation
+
+```bash
+npx typedoc --plugin typedoc-plugin-markdown --out ../docs/src/javascript src/index.ts
+```
--- a/node/gen_test_data.py
+++ b/node/gen_test_data.py
@@ -1,8 +0,0 @@
-import lancedb
-
-uri = "sample-lancedb"
-db = lancedb.connect(uri)
-table = db.create_table("my_table",
-                         data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
-                               {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
-
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.1.3",
+  "version": "0.1.4",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.1.3",
+      "version": "0.1.4",
      "license": "Apache-2.0",
      "dependencies": {
        "@apache-arrow/ts": "^12.0.0",
@@ -32,6 +32,8 @@
        "temp": "^0.9.4",
        "ts-node": "^10.9.1",
        "ts-node-dev": "^2.0.0",
+        "typedoc": "^0.24.7",
+        "typedoc-plugin-markdown": "^3.15.3",
        "typescript": "*"
      }
    },
@@ -642,6 +644,12 @@
        "node": ">=8"
      }
    },
+    "node_modules/ansi-sequence-parser": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/ansi-sequence-parser/-/ansi-sequence-parser-1.1.0.tgz",
+      "integrity": "sha512-lEm8mt52to2fT8GhciPCGeCXACSz2UwIN4X2e2LJSnZ5uAbn2/dsYdOmUXq0AtWS5cpAupysIneExOgH0Vd2TQ==",
+      "dev": true
+    },
    "node_modules/ansi-styles": {
      "version": "4.3.0",
      "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
@@ -2284,6 +2292,27 @@
      "integrity": "sha512-bzh50DW9kTPM00T8y4o8vQg89Di9oLJVLW/KaOGIXJWP/iqCN6WKYkbNOF04vFLJhwcpYUh9ydh/+5vpOqV4YQ==",
      "dev": true
    },
+    "node_modules/handlebars": {
+      "version": "4.7.7",
+      "resolved": "https://registry.npmjs.org/handlebars/-/handlebars-4.7.7.tgz",
+      "integrity": "sha512-aAcXm5OAfE/8IXkcZvCepKU3VzW1/39Fb5ZuqMtgI/hT8X2YgoMvBY5dLhq/cpOvw7Lk1nK/UF71aLG/ZnVYRA==",
+      "dev": true,
+      "dependencies": {
+        "minimist": "^1.2.5",
+        "neo-async": "^2.6.0",
+        "source-map": "^0.6.1",
+        "wordwrap": "^1.0.0"
+      },
+      "bin": {
+        "handlebars": "bin/handlebars"
+      },
+      "engines": {
+        "node": ">=0.4.7"
+      },
+      "optionalDependencies": {
+        "uglify-js": "^3.1.4"
+      }
+    },
    "node_modules/has": {
      "version": "1.0.3",
      "resolved": "https://registry.npmjs.org/has/-/has-1.0.3.tgz",
@@ -2782,6 +2811,12 @@
        "json5": "lib/cli.js"
      }
    },
+    "node_modules/jsonc-parser": {
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/jsonc-parser/-/jsonc-parser-3.2.0.tgz",
+      "integrity": "sha512-gfFQZrcTc8CnKXp6Y4/CBT3fTc0OVuDofpre4aEeEpSBPV5X5v4+Vmx+8snU7RLPrNHPKSgLxGo9YuQzz20o+w==",
+      "dev": true
+    },
    "node_modules/just-extend": {
      "version": "4.2.1",
      "resolved": "https://registry.npmjs.org/just-extend/-/just-extend-4.2.1.tgz",
@@ -2870,12 +2905,30 @@
        "node": ">=10"
      }
    },
+    "node_modules/lunr": {
+      "version": "2.3.9",
+      "resolved": "https://registry.npmjs.org/lunr/-/lunr-2.3.9.tgz",
+      "integrity": "sha512-zTU3DaZaF3Rt9rhN3uBMGQD3dD2/vFQqnvZCDv4dl5iOzq2IZQqTxu90r4E5J+nP70J3ilqVCrbho2eWaeW8Ow==",
+      "dev": true
+    },
    "node_modules/make-error": {
      "version": "1.3.6",
      "resolved": "https://registry.npmjs.org/make-error/-/make-error-1.3.6.tgz",
      "integrity": "sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==",
      "dev": true
    },
+    "node_modules/marked": {
+      "version": "4.3.0",
+      "resolved": "https://registry.npmjs.org/marked/-/marked-4.3.0.tgz",
+      "integrity": "sha512-PRsaiG84bK+AMvxziE/lCFss8juXjNaWzVbN5tXAm4XjeaS9NAHhop+PjQxz2A9h8Q4M/xGmzP8vqNwy6JeK0A==",
+      "dev": true,
+      "bin": {
+        "marked": "bin/marked.js"
+      },
+      "engines": {
+        "node": ">= 12"
+      }
+    },
    "node_modules/merge2": {
      "version": "1.4.1",
      "resolved": "https://registry.npmjs.org/merge2/-/merge2-1.4.1.tgz",
@@ -3096,6 +3149,12 @@
      "integrity": "sha512-Tj+HTDSJJKaZnfiuw+iaF9skdPpTo2GtEly5JHnWV/hfv2Qj/9RKsGISQtLh2ox3l5EAGw487hnBee0sIJ6v2g==",
      "dev": true
    },
+    "node_modules/neo-async": {
+      "version": "2.6.2",
+      "resolved": "https://registry.npmjs.org/neo-async/-/neo-async-2.6.2.tgz",
+      "integrity": "sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw==",
+      "dev": true
+    },
    "node_modules/nise": {
      "version": "5.1.4",
      "resolved": "https://registry.npmjs.org/nise/-/nise-5.1.4.tgz",
@@ -3604,6 +3663,18 @@
        "node": ">=8"
      }
    },
+    "node_modules/shiki": {
+      "version": "0.14.2",
+      "resolved": "https://registry.npmjs.org/shiki/-/shiki-0.14.2.tgz",
+      "integrity": "sha512-ltSZlSLOuSY0M0Y75KA+ieRaZ0Trf5Wl3gutE7jzLuIcWxLp5i/uEnLoQWNvgKXQ5OMpGkJnVMRLAuzjc0LJ2A==",
+      "dev": true,
+      "dependencies": {
+        "ansi-sequence-parser": "^1.1.0",
+        "jsonc-parser": "^3.2.0",
+        "vscode-oniguruma": "^1.7.0",
+        "vscode-textmate": "^8.0.0"
+      }
+    },
    "node_modules/side-channel": {
      "version": "1.0.4",
      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.0.4.tgz",
@@ -4064,6 +4135,63 @@
        "url": "https://github.com/sponsors/ljharb"
      }
    },
+    "node_modules/typedoc": {
+      "version": "0.24.7",
+      "resolved": "https://registry.npmjs.org/typedoc/-/typedoc-0.24.7.tgz",
+      "integrity": "sha512-zzfKDFIZADA+XRIp2rMzLe9xZ6pt12yQOhCr7cD7/PBTjhPmMyMvGrkZ2lPNJitg3Hj1SeiYFNzCsSDrlpxpKw==",
+      "dev": true,
+      "dependencies": {
+        "lunr": "^2.3.9",
+        "marked": "^4.3.0",
+        "minimatch": "^9.0.0",
+        "shiki": "^0.14.1"
+      },
+      "bin": {
+        "typedoc": "bin/typedoc"
+      },
+      "engines": {
+        "node": ">= 14.14"
+      },
+      "peerDependencies": {
+        "typescript": "4.6.x || 4.7.x || 4.8.x || 4.9.x || 5.0.x"
+      }
+    },
+    "node_modules/typedoc-plugin-markdown": {
+      "version": "3.15.3",
+      "resolved": "https://registry.npmjs.org/typedoc-plugin-markdown/-/typedoc-plugin-markdown-3.15.3.tgz",
+      "integrity": "sha512-idntFYu3vfaY3eaD+w9DeRd0PmNGqGuNLKihPU9poxFGnATJYGn9dPtEhn2QrTdishFMg7jPXAhos+2T6YCWRQ==",
+      "dev": true,
+      "dependencies": {
+        "handlebars": "^4.7.7"
+      },
+      "peerDependencies": {
+        "typedoc": ">=0.24.0"
+      }
+    },
+    "node_modules/typedoc/node_modules/brace-expansion": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz",
+      "integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==",
+      "dev": true,
+      "dependencies": {
+        "balanced-match": "^1.0.0"
+      }
+    },
+    "node_modules/typedoc/node_modules/minimatch": {
+      "version": "9.0.1",
+      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.1.tgz",
+      "integrity": "sha512-0jWhJpD/MdhPXwPuiRkCbfYfSKp2qnn2eOc279qI7f+osl/l+prKSrvhg157zSYvx/1nmgn2NqdT6k2Z7zSH9w==",
+      "dev": true,
+      "dependencies": {
+        "brace-expansion": "^2.0.1"
+      },
+      "engines": {
+        "node": ">=16 || 14 >=14.17"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/isaacs"
+      }
+    },
    "node_modules/typescript": {
      "version": "5.0.4",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.0.4.tgz",
@@ -4085,6 +4213,19 @@
        "node": ">=8"
      }
    },
+    "node_modules/uglify-js": {
+      "version": "3.17.4",
+      "resolved": "https://registry.npmjs.org/uglify-js/-/uglify-js-3.17.4.tgz",
+      "integrity": "sha512-T9q82TJI9e/C1TAxYvfb16xO120tMVFZrGA3f9/P4424DNu6ypK103y0GPFVa17yotwSyZW5iYXgjYHkGrJW/g==",
+      "dev": true,
+      "optional": true,
+      "bin": {
+        "uglifyjs": "bin/uglifyjs"
+      },
+      "engines": {
+        "node": ">=0.8.0"
+      }
+    },
    "node_modules/unbox-primitive": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/unbox-primitive/-/unbox-primitive-1.0.2.tgz",
@@ -4115,6 +4256,18 @@
      "integrity": "sha512-wa7YjyUGfNZngI/vtK0UHAN+lgDCxBPCylVXGp0zu59Fz5aiGtNXaq3DhIov063MorB+VfufLh3JlF2KdTK3xg==",
      "dev": true
    },
+    "node_modules/vscode-oniguruma": {
+      "version": "1.7.0",
+      "resolved": "https://registry.npmjs.org/vscode-oniguruma/-/vscode-oniguruma-1.7.0.tgz",
+      "integrity": "sha512-L9WMGRfrjOhgHSdOYgCt/yRMsXzLDJSL7BPrOZt73gU0iWO4mpqzqQzOz5srxqTvMBaR0XZTSrVWo4j55Rc6cA==",
+      "dev": true
+    },
+    "node_modules/vscode-textmate": {
+      "version": "8.0.0",
+      "resolved": "https://registry.npmjs.org/vscode-textmate/-/vscode-textmate-8.0.0.tgz",
+      "integrity": "sha512-AFbieoL7a5LMqcnOF04ji+rpXadgOXnZsxQr//r83kLPr7biP7am3g9zbaZIaBGwBRWeSvoMD4mgPdX3e4NWBg==",
+      "dev": true
+    },
    "node_modules/which": {
      "version": "2.0.2",
      "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
@@ -4175,6 +4328,12 @@
        "node": ">=0.10.0"
      }
    },
+    "node_modules/wordwrap": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/wordwrap/-/wordwrap-1.0.0.tgz",
+      "integrity": "sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q==",
+      "dev": true
+    },
    "node_modules/wordwrapjs": {
      "version": "4.0.1",
      "resolved": "https://registry.npmjs.org/wordwrapjs/-/wordwrapjs-4.0.1.tgz",
@@ -4767,6 +4926,12 @@
      "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
      "dev": true
    },
+    "ansi-sequence-parser": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/ansi-sequence-parser/-/ansi-sequence-parser-1.1.0.tgz",
+      "integrity": "sha512-lEm8mt52to2fT8GhciPCGeCXACSz2UwIN4X2e2LJSnZ5uAbn2/dsYdOmUXq0AtWS5cpAupysIneExOgH0Vd2TQ==",
+      "dev": true
+    },
    "ansi-styles": {
      "version": "4.3.0",
      "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
@@ -5983,6 +6148,19 @@
      "integrity": "sha512-bzh50DW9kTPM00T8y4o8vQg89Di9oLJVLW/KaOGIXJWP/iqCN6WKYkbNOF04vFLJhwcpYUh9ydh/+5vpOqV4YQ==",
      "dev": true
    },
+    "handlebars": {
+      "version": "4.7.7",
+      "resolved": "https://registry.npmjs.org/handlebars/-/handlebars-4.7.7.tgz",
+      "integrity": "sha512-aAcXm5OAfE/8IXkcZvCepKU3VzW1/39Fb5ZuqMtgI/hT8X2YgoMvBY5dLhq/cpOvw7Lk1nK/UF71aLG/ZnVYRA==",
+      "dev": true,
+      "requires": {
+        "minimist": "^1.2.5",
+        "neo-async": "^2.6.0",
+        "source-map": "^0.6.1",
+        "uglify-js": "^3.1.4",
+        "wordwrap": "^1.0.0"
+      }
+    },
    "has": {
      "version": "1.0.3",
      "resolved": "https://registry.npmjs.org/has/-/has-1.0.3.tgz",
@@ -6324,6 +6502,12 @@
        "minimist": "^1.2.0"
      }
    },
+    "jsonc-parser": {
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/jsonc-parser/-/jsonc-parser-3.2.0.tgz",
+      "integrity": "sha512-gfFQZrcTc8CnKXp6Y4/CBT3fTc0OVuDofpre4aEeEpSBPV5X5v4+Vmx+8snU7RLPrNHPKSgLxGo9YuQzz20o+w==",
+      "dev": true
+    },
    "just-extend": {
      "version": "4.2.1",
      "resolved": "https://registry.npmjs.org/just-extend/-/just-extend-4.2.1.tgz",
@@ -6394,12 +6578,24 @@
        "yallist": "^4.0.0"
      }
    },
+    "lunr": {
+      "version": "2.3.9",
+      "resolved": "https://registry.npmjs.org/lunr/-/lunr-2.3.9.tgz",
+      "integrity": "sha512-zTU3DaZaF3Rt9rhN3uBMGQD3dD2/vFQqnvZCDv4dl5iOzq2IZQqTxu90r4E5J+nP70J3ilqVCrbho2eWaeW8Ow==",
+      "dev": true
+    },
    "make-error": {
      "version": "1.3.6",
      "resolved": "https://registry.npmjs.org/make-error/-/make-error-1.3.6.tgz",
      "integrity": "sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==",
      "dev": true
    },
+    "marked": {
+      "version": "4.3.0",
+      "resolved": "https://registry.npmjs.org/marked/-/marked-4.3.0.tgz",
+      "integrity": "sha512-PRsaiG84bK+AMvxziE/lCFss8juXjNaWzVbN5tXAm4XjeaS9NAHhop+PjQxz2A9h8Q4M/xGmzP8vqNwy6JeK0A==",
+      "dev": true
+    },
    "merge2": {
      "version": "1.4.1",
      "resolved": "https://registry.npmjs.org/merge2/-/merge2-1.4.1.tgz",
@@ -6564,6 +6760,12 @@
      "integrity": "sha512-Tj+HTDSJJKaZnfiuw+iaF9skdPpTo2GtEly5JHnWV/hfv2Qj/9RKsGISQtLh2ox3l5EAGw487hnBee0sIJ6v2g==",
      "dev": true
    },
+    "neo-async": {
+      "version": "2.6.2",
+      "resolved": "https://registry.npmjs.org/neo-async/-/neo-async-2.6.2.tgz",
+      "integrity": "sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw==",
+      "dev": true
+    },
    "nise": {
      "version": "5.1.4",
      "resolved": "https://registry.npmjs.org/nise/-/nise-5.1.4.tgz",
@@ -6908,6 +7110,18 @@
      "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
      "dev": true
    },
+    "shiki": {
+      "version": "0.14.2",
+      "resolved": "https://registry.npmjs.org/shiki/-/shiki-0.14.2.tgz",
+      "integrity": "sha512-ltSZlSLOuSY0M0Y75KA+ieRaZ0Trf5Wl3gutE7jzLuIcWxLp5i/uEnLoQWNvgKXQ5OMpGkJnVMRLAuzjc0LJ2A==",
+      "dev": true,
+      "requires": {
+        "ansi-sequence-parser": "^1.1.0",
+        "jsonc-parser": "^3.2.0",
+        "vscode-oniguruma": "^1.7.0",
+        "vscode-textmate": "^8.0.0"
+      }
+    },
    "side-channel": {
      "version": "1.0.4",
      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.0.4.tgz",
@@ -7236,6 +7450,47 @@
        "is-typed-array": "^1.1.9"
      }
    },
+    "typedoc": {
+      "version": "0.24.7",
+      "resolved": "https://registry.npmjs.org/typedoc/-/typedoc-0.24.7.tgz",
+      "integrity": "sha512-zzfKDFIZADA+XRIp2rMzLe9xZ6pt12yQOhCr7cD7/PBTjhPmMyMvGrkZ2lPNJitg3Hj1SeiYFNzCsSDrlpxpKw==",
+      "dev": true,
+      "requires": {
+        "lunr": "^2.3.9",
+        "marked": "^4.3.0",
+        "minimatch": "^9.0.0",
+        "shiki": "^0.14.1"
+      },
+      "dependencies": {
+        "brace-expansion": {
+          "version": "2.0.1",
+          "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz",
+          "integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==",
+          "dev": true,
+          "requires": {
+            "balanced-match": "^1.0.0"
+          }
+        },
+        "minimatch": {
+          "version": "9.0.1",
+          "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.1.tgz",
+          "integrity": "sha512-0jWhJpD/MdhPXwPuiRkCbfYfSKp2qnn2eOc279qI7f+osl/l+prKSrvhg157zSYvx/1nmgn2NqdT6k2Z7zSH9w==",
+          "dev": true,
+          "requires": {
+            "brace-expansion": "^2.0.1"
+          }
+        }
+      }
+    },
+    "typedoc-plugin-markdown": {
+      "version": "3.15.3",
+      "resolved": "https://registry.npmjs.org/typedoc-plugin-markdown/-/typedoc-plugin-markdown-3.15.3.tgz",
+      "integrity": "sha512-idntFYu3vfaY3eaD+w9DeRd0PmNGqGuNLKihPU9poxFGnATJYGn9dPtEhn2QrTdishFMg7jPXAhos+2T6YCWRQ==",
+      "dev": true,
+      "requires": {
+        "handlebars": "^4.7.7"
+      }
+    },
    "typescript": {
      "version": "5.0.4",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.0.4.tgz",
@@ -7247,6 +7502,13 @@
      "resolved": "https://registry.npmjs.org/typical/-/typical-4.0.0.tgz",
      "integrity": "sha512-VAH4IvQ7BDFYglMd7BPRDfLgxZZX4O4TFcRDA6EN5X7erNJJq+McIEp8np9aVtxrCJ6qx4GTYVfOWNjcqwZgRw=="
    },
+    "uglify-js": {
+      "version": "3.17.4",
+      "resolved": "https://registry.npmjs.org/uglify-js/-/uglify-js-3.17.4.tgz",
+      "integrity": "sha512-T9q82TJI9e/C1TAxYvfb16xO120tMVFZrGA3f9/P4424DNu6ypK103y0GPFVa17yotwSyZW5iYXgjYHkGrJW/g==",
+      "dev": true,
+      "optional": true
+    },
    "unbox-primitive": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/unbox-primitive/-/unbox-primitive-1.0.2.tgz",
@@ -7274,6 +7536,18 @@
      "integrity": "sha512-wa7YjyUGfNZngI/vtK0UHAN+lgDCxBPCylVXGp0zu59Fz5aiGtNXaq3DhIov063MorB+VfufLh3JlF2KdTK3xg==",
      "dev": true
    },
+    "vscode-oniguruma": {
+      "version": "1.7.0",
+      "resolved": "https://registry.npmjs.org/vscode-oniguruma/-/vscode-oniguruma-1.7.0.tgz",
+      "integrity": "sha512-L9WMGRfrjOhgHSdOYgCt/yRMsXzLDJSL7BPrOZt73gU0iWO4mpqzqQzOz5srxqTvMBaR0XZTSrVWo4j55Rc6cA==",
+      "dev": true
+    },
+    "vscode-textmate": {
+      "version": "8.0.0",
+      "resolved": "https://registry.npmjs.org/vscode-textmate/-/vscode-textmate-8.0.0.tgz",
+      "integrity": "sha512-AFbieoL7a5LMqcnOF04ji+rpXadgOXnZsxQr//r83kLPr7biP7am3g9zbaZIaBGwBRWeSvoMD4mgPdX3e4NWBg==",
+      "dev": true
+    },
    "which": {
      "version": "2.0.2",
      "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
@@ -7316,6 +7590,12 @@
      "integrity": "sha512-Hz/mrNwitNRh/HUAtM/VT/5VH+ygD6DV7mYKZAtHOrbs8U7lvPS6xf7EJKMF0uW1KJCl0H701g3ZGus+muE5vQ==",
      "dev": true
    },
+    "wordwrap": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/wordwrap/-/wordwrap-1.0.0.tgz",
+      "integrity": "sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q==",
+      "dev": true
+    },
    "wordwrapjs": {
      "version": "4.0.1",
      "resolved": "https://registry.npmjs.org/wordwrapjs/-/wordwrapjs-4.0.1.tgz",
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.1.3",
+  "version": "0.1.4",
  "description": " Serverless, low-latency vector database for AI applications",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
@@ -9,7 +9,8 @@
    "build": "cargo-cp-artifact --artifact cdylib vectordb-node index.node -- cargo build --message-format=json-render-diagnostics",
    "build-release": "npm run build -- --release",
    "test": "mocha -recursive dist/test",
-    "lint": "eslint src --ext .js,.ts"
+    "lint": "eslint src --ext .js,.ts",
+    "clean": "rm -rf node_modules *.node dist/"
  },
  "repository": {
    "type": "git",
@@ -38,11 +39,13 @@
    "eslint-plugin-n": "^15.7.0",
    "eslint-plugin-promise": "^6.1.1",
    "mocha": "^10.2.0",
-    "sinon": "^15.1.0",
    "openai": "^3.2.1",
+    "sinon": "^15.1.0",
    "temp": "^0.9.4",
    "ts-node": "^10.9.1",
    "ts-node-dev": "^2.0.0",
+    "typedoc": "^0.24.7",
+    "typedoc-plugin-markdown": "^3.15.3",
    "typescript": "*"
  },
  "dependencies": {
--- a/node/src/index.ts
+++ b/node/src/index.ts
@@ -168,9 +168,16 @@ export class Table<T = number[]> {
   *
   * @param indexParams The parameters of this Index, @see VectorIndexParams.
   */
-  async create_index (indexParams: VectorIndexParams): Promise<any> {
+  async createIndex (indexParams: VectorIndexParams): Promise<any> {
    return tableCreateVectorIndex.call(this._tbl, indexParams)
  }
+
+  /**
+   * @deprecated Use [Table.createIndex]
+   */
+  async create_index (indexParams: VectorIndexParams): Promise<any> {
+    return await this.createIndex(indexParams)
+  }
 }

 interface IvfPQIndexConfig {
@@ -233,7 +240,7 @@ export class Query<T = number[]> {
  private _limit: number
  private _refineFactor?: number
  private _nprobes: number
-  private readonly _columns?: string[]
+  private _select?: string[]
  private _filter?: string
  private _metricType?: MetricType
  private readonly _embeddings?: EmbeddingFunction<T>
@@ -244,7 +251,7 @@ export class Query<T = number[]> {
    this._limit = 10
    this._nprobes = 20
    this._refineFactor = undefined
-    this._columns = undefined
+    this._select = undefined
    this._filter = undefined
    this._metricType = undefined
    this._embeddings = embeddings
@@ -286,6 +293,15 @@ export class Query<T = number[]> {
    return this
  }

+  /** Return only the specified columns.
+   *
+   * @param value Only select the specified columns. If not specified, all columns will be returned.
+   */
+  select (value: string[]): Query<T> {
+    this._select = value
+    return this
+  }
+
  /**
   * The MetricType used for this Query.
   * @param value The metric to the. @see MetricType for the different options
--- a/node/src/test/test.ts
+++ b/node/src/test/test.ts
@@ -72,6 +72,22 @@ describe('LanceDB client', function () {
      assert.equal(results.length, 1)
      assert.equal(results[0].id, 2)
    })
+
+    it('select only a subset of columns', async function () {
+      const uri = await createTestDB()
+      const con = await lancedb.connect(uri)
+      const table = await con.openTable('vectors')
+      const results = await table.search([0.1, 0.1]).select(['is_active']).execute()
+      assert.equal(results.length, 2)
+      // vector and score are always returned
+      assert.isDefined(results[0].vector)
+      assert.isDefined(results[0].score)
+      assert.isDefined(results[0].is_active)
+
+      assert.isUndefined(results[0].id)
+      assert.isUndefined(results[0].name)
+      assert.isUndefined(results[0].price)
+    })
  })

  describe('when creating a new dataset', function () {
@@ -137,7 +153,7 @@ describe('LanceDB client', function () {
      const uri = await createTestDB(32, 300)
      const con = await lancedb.connect(uri)
      const table = await con.openTable('vectors')
-      await table.create_index({ type: 'ivf_pq', column: 'vector', num_partitions: 2, max_iters: 2 })
+      await table.createIndex({ type: 'ivf_pq', column: 'vector', num_partitions: 2, max_iters: 2 })
    }).timeout(10_000) // Timeout is high partially because GH macos runner is pretty slow
  })

@@ -181,11 +197,13 @@ describe('Query object', function () {
      .limit(1)
      .metricType(MetricType.Cosine)
      .refineFactor(100)
+      .select(['a', 'b'])
      .nprobes(20) as Record<string, any>
    assert.equal(query._limit, 1)
    assert.equal(query._metricType, MetricType.Cosine)
    assert.equal(query._refineFactor, 100)
    assert.equal(query._nprobes, 20)
+    assert.deepEqual(query._select, ['a', 'b'])
  })
 })

--- a/python/lancedb/fts.py
+++ b/python/lancedb/fts.py
@@ -118,6 +118,8 @@ def search_index(
    query = index.parse_query(query)
    # get top results
    results = searcher.search(query, limit)
+    if results.count == 0:
+        return tuple(), tuple()
    return tuple(
        zip(
            *[
--- a/python/lancedb/query.py
+++ b/python/lancedb/query.py
@@ -164,6 +164,8 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
        index = tantivy.Index.open(index_path)
        # get the scores and doc ids
        row_ids, scores = search_index(index, self._query, self._limit)
+        if len(row_ids) == 0:
+            return pd.DataFrame()
        scores = pa.array(scores)
        output_tbl = self._table.to_lance().take(row_ids, columns=self._columns)
        output_tbl = output_tbl.append_column("score", scores)
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -1,6 +1,6 @@
 [project]
 name = "lancedb"
-version = "0.1.5"
+version = "0.1.6"
 dependencies = ["pylance>=0.4.17", "ratelimiter", "retry", "tqdm"]
 description = "lancedb"
 authors = [
@@ -33,11 +33,11 @@ classifiers = [
 ]

 [project.urls]
-repository = "https://github.com/eto-ai/lancedb"
+repository = "https://github.com/lancedb/lancedb"

 [project.optional-dependencies]
 tests = [
-    "pytest"
+    "pytest", "pytest-mock"
 ]
 dev = [
    "ruff", "pre-commit", "black"
--- a/python/tests/test_fts.py
+++ b/python/tests/test_fts.py
@@ -82,3 +82,10 @@ def test_create_index_multiple_columns(tmp_path, table):
    assert len(df) == 10
    assert "text" in df.columns
    assert "text2" in df.columns
+
+
+def test_empty_rs(tmp_path, table, mocker):
+    table.create_fts_index(["text", "text2"])
+    mocker.patch("lancedb.fts.search_index", return_value=([], []))
+    df = table.search("puppy").limit(10).to_df()
+    assert len(df) == 0
--- a/rust/ffi/node/src/lib.rs
+++ b/rust/ffi/node/src/lib.rs
@@ -129,6 +129,17 @@ fn table_search(mut cx: FunctionContext) -> JsResult<JsPromise> {
    let limit = query_obj
        .get::<JsNumber, _, _>(&mut cx, "_limit")?
        .value(&mut cx);
+    let select = query_obj
+        .get_opt::<JsArray, _, _>(&mut cx, "_select")?
+        .map(|arr| {
+            let js_array = arr.deref();
+            let mut projection_vec: Vec<String> = Vec::new();
+            for i in 0..js_array.len(&mut cx) {
+                let entry: Handle<JsString> = js_array.get(&mut cx, i).unwrap();
+                projection_vec.push(entry.value(&mut cx));
+            }
+            projection_vec
+        });
    let filter = query_obj
        .get_opt::<JsString, _, _>(&mut cx, "_filter")?
        .map(|s| s.value(&mut cx));
@@ -161,7 +172,8 @@ fn table_search(mut cx: FunctionContext) -> JsResult<JsPromise> {
            .refine_factor(refine_factor)
            .nprobes(nprobes)
            .filter(filter)
-            .metric_type(metric_type);
+            .metric_type(metric_type)
+            .select(select);
        let record_batch_stream = builder.execute();
        let results = record_batch_stream
            .and_then(|stream| stream.try_collect::<Vec<_>>().map_err(Error::from))
--- a/rust/vectordb/src/query.rs
+++ b/rust/vectordb/src/query.rs
@@ -27,6 +27,7 @@ pub struct Query {
    pub query_vector: Float32Array,
    pub limit: usize,
    pub filter: Option<String>,
+    pub select: Option<Vec<String>>,
    pub nprobes: usize,
    pub refine_factor: Option<u32>,
    pub metric_type: Option<MetricType>,
@@ -54,6 +55,7 @@ impl Query {
            metric_type: None,
            use_index: false,
            filter: None,
+            select: None,
        }
    }

@@ -72,6 +74,9 @@ impl Query {
        )?;
        scanner.nprobs(self.nprobes);
        scanner.use_index(self.use_index);
+        self.select
+            .as_ref()
+            .map(|p| scanner.project(p.as_slice()));
        self.filter.as_ref().map(|f| scanner.filter(f));
        self.refine_factor.map(|rf| scanner.refine(rf));
        self.metric_type.map(|mt| scanner.distance_metric(mt));
@@ -138,10 +143,23 @@ impl Query {
        self
    }

+    ///  A filter statement to be applied to this query.
+    ///
+    /// # Arguments
+    ///
+    /// * `filter` -  value A filter in the same format used by a sql WHERE clause.
    pub fn filter(mut self, filter: Option<String>) -> Query {
        self.filter = filter;
        self
    }
+
+    /// Return only the specified columns.
+    ///
+    /// Only select the specified columns. If not specified, all columns will be returned.
+    pub fn select(mut self, columns: Option<Vec<String>>) -> Query {
+        self.select = columns;
+        self
+    }
 }

 #[cfg(test)]
Author	SHA1	Message	Date
Chang She	3798f56a9b	bump version for v0.1.6-python	2023-06-05 18:20:15 -07:00
Chang She	50cdb16b45	Better handle empty results from tantivy (#155 ) Closes #154 --------- Co-authored-by: Chang She <chang@lancedb.com>	2023-06-05 18:18:14 -07:00
gsilvestrin	d803482588	[nodejs] bumping version to 0.1.4 (#147 )	2023-06-03 13:59:58 -07:00
gsilvestrin	f37994b72a	[nodejs] deprecated created_index in favor of createIndex. (#145 )	2023-06-03 11:05:35 -07:00
gsilvestrin	2418de0a3c	[nodejs] add npm clean task (#146 )	2023-06-03 11:05:02 -07:00
gsilvestrin	d0c47e3838	added projection api for nodejs (#140 )	2023-06-03 10:34:08 -07:00
Jai	41cca31f48	Modal example using LangChain (#143 )	2023-06-03 06:08:31 -07:00
Jai	b621009d39	add multimodal gif, add copy about fts, sql (#144 )	2023-06-02 22:25:33 -07:00
Jai	6a9cde22de	Update broken doc links to refer to new directory and include gallery app for multimodal search (#142 ) closes #121 adds new multimodal example to gallery app	2023-06-02 21:27:26 -07:00
Chang She	bfa90b35ee	add code snippet for each example (#141 ) <img width="1937" alt="image" src="https://github.com/lancedb/lancedb/assets/759245/4ee52e4a-5955-47c2-9ffe-84d1bc0062ff"> --------- Co-authored-by: Chang She <chang@lancedb.com>	2023-06-02 21:27:02 -07:00
gsilvestrin	12ec29f55b	Adding nodejs CHANGELOG.md (#132 )	2023-06-02 18:27:53 -07:00
Lei Xu	cdd08ef35c	[Doc] Metrics types. (#135 ) Closes #129	2023-06-02 17:18:01 -07:00
Jai	adcb2a1387	Update mkdocs.yml (#138 )	2023-06-02 17:13:32 -07:00
Jai	9d52a32668	Minor patch to docs (#136 )	2023-06-02 16:26:03 -07:00
Jai	11b2e63eea	fix index docs (#134 )	2023-06-02 16:16:34 -07:00
Jai	daedf1396b	update references to end to end examples, use s3 for langchain exampl… (#133 )	2023-06-02 16:08:56 -07:00
Jai	8af5f19cc1	js docs, modal example, doc notebook integration, update doc styles (#131 )	2023-06-02 15:24:16 -07:00
				`@@ -0,0 +1 @@`
				TypeDoc added this file to prevent GitHub Pages from using Jekyll. You can turn off this behavior by setting the `githubPages` option to false.