[python] Bump version: 0.2.0 → 0.2.1

bump lance version to 0.6.5 for lancedb release (#453 )
Clean up docs testing - exclude by glob instead of by file (#450 )
2025-12-23 21:39:57 +00:00 · 2023-08-24 04:00:11 +00:00 · 2023-08-23 20:59:03 -07:00 · 2023-08-24 07:30:37 +05:30 · 2023-08-22 16:21:02 -07:00 · 2023-08-22 21:11:28 +00:00
42 changed files with 1590 additions and 292 deletions
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.1.19
+current_version = 0.2.3
 commit = True
 message = Bump version: {current_version} → {new_version}
 tag = True
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -6,7 +6,7 @@ members = [
 resolver = "2"

 [workspace.dependencies]
-lance = "=0.6.1"
+lance = "=0.6.5"
 arrow-array = "43.0"
 arrow-data = "43.0"
 arrow-schema = "43.0"
@@ -14,4 +14,3 @@ arrow-ipc = "43.0"
 half = { "version" = "=2.2.1", default-features = false }
 object_store = "0.6.1"
 snafu = "0.7.4"
-
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -12,6 +12,15 @@ theme:
    - content.code.copy
    - content.tabs.link
    - content.action.edit
+    - toc.follow
+    - toc.integrate
+    - navigation.top
+    - navigation.tabs
+    - navigation.tabs.sticky
+    - navigation.footer
+    - navigation.tracking
+    - navigation.instant
+    - navigation.indexes
  icon:
    repo: fontawesome/brands/github
  custom_dir: overrides
@@ -55,11 +64,43 @@ markdown_extensions:
 - md_in_html

 nav:
- Home: index.md
+- Home:
+  - 🏢 Home: index.md
+  - 💡 Basics: basic.md
+  - 🧬 Embeddings: embedding.md
+  - 🔍 Python full-text search: fts.md
+  - 🔌 Integrations:
+    - integrations/index.md
+    - Pandas and PyArrow: python/arrow.md
+    - DuckDB: python/duckdb.md
+    - LangChain 🔗: https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lancedb.html
+    - LangChain JS/TS 🔗: https://js.langchain.com/docs/modules/data_connection/vectorstores/integrations/lancedb
+    - LlamaIndex 🦙: https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html
+    - Pydantic: python/pydantic.md
+    - Voxel51: integrations/voxel51.md
+    - PromptTools: integrations/prompttools.md
+  - 🐍 Python examples:
+    - examples/index.md
+    - YouTube Transcript Search: notebooks/youtube_transcript_search.ipynb
+    - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
+    - Multimodal search using CLIP: notebooks/multimodal_search.ipynb
+    - Serverless QA Bot with S3 and Lambda: examples/serverless_lancedb_with_s3_and_lambda.md
+    - Serverless QA Bot with Modal: examples/serverless_qa_bot_with_modal_and_langchain.md
+  - 🌐 Javascript examples:
+    - Examples: examples/index_js.md
+    - Serverless Website Chatbot: examples/serverless_website_chatbot.md
+    - YouTube Transcript Search: examples/youtube_transcript_bot_with_nodejs.md
+    - TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md
+  - 📚 Guides:
+    - Tables: guides/tables.md
+    - Vector Search: search.md
+    - SQL filters: sql.md
+    - Indexing: ann_indexes.md
 - Basics: basic.md
 - Embeddings: embedding.md
 - Python full-text search: fts.md
 - Integrations:
+  - integrations/index.md
  - Pandas and PyArrow: python/arrow.md
  - DuckDB: python/duckdb.md
  - LangChain 🦜️🔗: https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lancedb.html
@@ -67,14 +108,18 @@ nav:
  - LlamaIndex 🦙: https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html
  - Pydantic: python/pydantic.md
  - Voxel51: integrations/voxel51.md
+  - PromptTools: integrations/prompttools.md
 - Python examples:
+  - examples/index.md
  - YouTube Transcript Search: notebooks/youtube_transcript_search.ipynb
  - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
  - Multimodal search using CLIP: notebooks/multimodal_search.ipynb
  - Serverless QA Bot with S3 and Lambda: examples/serverless_lancedb_with_s3_and_lambda.md
  - Serverless QA Bot with Modal: examples/serverless_qa_bot_with_modal_and_langchain.md
 - Javascript examples:
+  - examples/index_js.md
  - YouTube Transcript Search: examples/youtube_transcript_bot_with_nodejs.md
+  - Serverless Chatbot from any website: examples/serverless_website_chatbot.md
  - TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md

 - Guides:
@@ -85,6 +130,7 @@ nav:
 - API references:
  - Python API: python/python.md
  - Javascript API: javascript/modules.md
+- LanceDB Cloud↗: https://noteforms.com/forms/lancedb-mailing-list-cloud-kty1o5?notionforms=1&utm_source=notionforms

 extra_css:
  - styles/global.css
--- a/docs/src/assets/ecosystem-illustration.png
+++ b/docs/src/assets/ecosystem-illustration.png
--- a/docs/src/assets/langchain.png
+++ b/docs/src/assets/langchain.png
--- a/docs/src/assets/llama-index.jpg
+++ b/docs/src/assets/llama-index.jpg
--- a/docs/src/assets/prompttools.jpeg
+++ b/docs/src/assets/prompttools.jpeg
--- a/docs/src/assets/vercel-template.gif
+++ b/docs/src/assets/vercel-template.gif
--- a/docs/src/basic.md
+++ b/docs/src/basic.md
@@ -198,3 +198,15 @@ you can pass in `ignore_missing=True`.
 This section covered the very basics of the LanceDB API.
 LanceDB supports many additional features when creating indices to speed up search and options for search.
 These are contained in the next section of the documentation.
+
+## Note: Bundling vectorDB apps with webpack
+Since LanceDB contains a prebuilt Node binary, you must configure `next.config.js` to exclude it from webpack. This is required for both using Next.js and deploying on Vercel.
+```javascript
+/** @type {import('next').NextConfig} */
+module.exports = ({
+  webpack(config) {
+    config.externals.push({ vectordb: 'vectordb' })
+    return config;
+  }
+})
+```
--- a/docs/src/embedding.md
+++ b/docs/src/embedding.md
@@ -66,7 +66,7 @@ You can also use an external API like OpenAI to generate embeddings
    to generate embeddings for each row.

    Say if you have a pandas DataFrame with a `text` column that you want to be embedded,
-    you can use the [with_embeddings](https://lancedb.github.io/lancedb/python/#lancedb.embeddings.with_embeddings)
+    you can use the [with_embeddings](https://lancedb.github.io/lancedb/python/python/#lancedb.embeddings.with_embeddings)
    function to generate embeddings and add create a combined pyarrow table:


--- a/docs/src/examples/index.md
+++ b/docs/src/examples/index.md
@@ -0,0 +1,23 @@
+# Examples
+
+Here are some of the examples, projects and applications using LanceDB python library. Some examples are covered in detail in the next sections. You can find more on [VectorDB Recipes](https://github.com/lancedb/vectordb-recipes)
+
+| Example | Interactive Envs | Scripts  |
+|-------- | ---------------- | ------   |
+| | | |
+| [Youtube transcript search bot](https://github.com/lancedb/vectordb-recipes/tree/main/examples/youtube_bot/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/youtube_bot/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>| [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/youtube_bot/main.py)|
+| [Langchain: Code Docs QA bot](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Code-Documentation-QA-Bot/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>| [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Code-Documentation-QA-Bot/main.py) |
+| [AI Agents: Reducing Hallucination](https://github.com/lancedb/vectordb-recipes/tree/main/examples/reducing_hallucinations_ai_agents/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/reducing_hallucinations_ai_agents/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>| [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/reducing_hallucinations_ai_agents/main.py)|
+| [Multimodal CLIP: DiffusionDB](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_clip/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multimodal_clip/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>| [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_clip/main.py)  |
+| [Multimodal CLIP: Youtube videos](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_video_search/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multimodal_video_search/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>| [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_video_search/main.py)  |
+| [Movie Recommender](https://github.com/lancedb/vectordb-recipes/tree/main/examples/movie-recommender/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/movie-recommender/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> | [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/movie-recommender/main.py)  |
+| [Audio Search](https://github.com/lancedb/vectordb-recipes/tree/main/examples/audio_search/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/audio_search/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> | [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/audio_search/main.py)  |
+| [Multimodal Image + Text Search](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_search/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/multimodal_search/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> | [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/multimodal_search/main.py)  |
+| [Evaluating Prompts with Prompttools](https://github.com/lancedb/vectordb-recipes/tree/main/examples/prompttools-eval-prompts/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/prompttools-eval-prompts/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> |  |
+
+## Projects & Applications powered by LanceDB
+
+| Project Name                                        | Description                                                                                                          | Screenshot                                |
+|-----------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|-------------------------------------------|
+| [YOLOExplorer](https://github.com/lancedb/yoloexplorer) | Iterate on your YOLO / CV datasets using SQL, Vector semantic search, and more within seconds                  | ![YOLOExplorer](https://github.com/lancedb/vectordb-recipes/assets/15766192/ae513a29-8f15-4e0b-99a1-ccd8272b6131) |
+| [Website Chatbot (Deployable Vercel Template)](https://github.com/lancedb/lancedb-vercel-chatbot) | Create a chatbot from the sitemap of any website/docs of your choice. Built using vectorDB serverless native javascript package. | ![Chatbot](../assets/vercel-template.gif)    |
--- a/docs/src/examples/index_js.md
+++ b/docs/src/examples/index_js.md
@@ -0,0 +1,19 @@
+# Examples
+
+Here are some of the examples, projects and applications using vectordb native javascript library.
+Some examples are covered in detail in the next sections. You can find more on [VectorDB Recipes](https://github.com/lancedb/vectordb-recipes)
+
+| Example | Scripts  |
+|-------- | ------   |
+| | |
+| [Youtube transcript search bot](https://github.com/lancedb/vectordb-recipes/tree/main/examples/youtube_bot/) | [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/youtube_bot/index.js)|
+| [Langchain: Code Docs QA bot](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Code-Documentation-QA-Bot/) | [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/Code-Documentation-QA-Bot/index.js)|
+| [AI Agents: Reducing Hallucination](https://github.com/lancedb/vectordb-recipes/tree/main/examples/reducing_hallucinations_ai_agents/) | [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/reducing_hallucinations_ai_agents/index.js)|
+| [TransformersJS Embedding example](https://github.com/lancedb/vectordb-recipes/tree/main/examples/js-transformers/) | [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](https://github.com/lancedb/vectordb-recipes/tree/main/examples/js-transformers/index.js)  |
+
+## Projects & Applications
+
+| Project Name                                        | Description                                                                                                          | Screenshot                                |
+|-----------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|-------------------------------------------|
+| [YOLOExplorer](https://github.com/lancedb/yoloexplorer) | Iterate on your YOLO / CV datasets using SQL, Vector semantic search, and more within seconds                  | ![YOLOExplorer](https://github.com/lancedb/vectordb-recipes/assets/15766192/ae513a29-8f15-4e0b-99a1-ccd8272b6131) |
+| [Website Chatbot (Deployable Vercel Template)](https://github.com/lancedb/lancedb-vercel-chatbot) | Create a chatbot from the sitemap of any website/docs of your choice. Built using vectorDB serverless native javascript package. | ![Chatbot](../assets/vercel-template.gif)    |
--- a/docs/src/examples/serverless_website_chatbot.md
+++ b/docs/src/examples/serverless_website_chatbot.md
@@ -0,0 +1,61 @@
+# LanceDB Chatbot - Vercel Next.js Template
+Use an AI chatbot with website context retrieved from a vector store like LanceDB. LanceDB is lightweight and can be embedded directly into Next.js, with data stored on-prem.
+
+## One click deploy on Vercel
+[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flancedb%2Flancedb-vercel-chatbot&env=OPENAI_API_KEY&envDescription=OpenAI%20API%20Key%20for%20chat%20completion.&project-name=lancedb-vercel-chatbot&repository-name=lancedb-vercel-chatbot&demo-title=LanceDB%20Chatbot%20Demo&demo-description=Demo%20website%20chatbot%20with%20LanceDB.&demo-url=https%3A%2F%2Flancedb.vercel.app&demo-image=https%3A%2F%2Fi.imgur.com%2FazVJtvr.png)
+
+![Demo website landing page](../assets/vercel-template.gif)
+
+## Development
+
+First, rename `.env.example` to `.env.local`, and fill out `OPENAI_API_KEY` with your OpenAI API key. You can get one [here](https://openai.com/blog/openai-api).
+
+Run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
+
+## Learn More
+
+To learn more about LanceDB or Next.js, take a look at the following resources:
+
+- [LanceDB Documentation](https://lancedb.github.io/lancedb/) - learn about LanceDB, the developer-friendly serverless vector database.
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+## LanceDB on Next.js and Vercel
+
+FYI: these configurations have been pre-implemented in this template.
+
+Since LanceDB contains a prebuilt Node binary, you must configure `next.config.js` to exclude it from webpack. This is required for both using Next.js and deploying on Vercel.
+```js
+/** @type {import('next').NextConfig} */
+module.exports = ({
+  webpack(config) {
+    config.externals.push({ vectordb: 'vectordb' })
+    return config;
+  }
+})
+```
+
+To deploy on Vercel, we need to make sure that the NodeJS runtime static file analysis for Vercel can find the binary, since LanceDB uses dynamic imports by default. We can do this by modifying `package.json` in the `scripts` section.
+```json
+{
+  ...
+  "scripts": {
+    ...
+    "vercel-build": "sed -i 's/nativeLib = require(`@lancedb\\/vectordb-\\${currentTarget()}`);/nativeLib = require(`@lancedb\\/vectordb-linux-x64-gnu`);/' node_modules/vectordb/native.js && next build",
+    ...
+  },
+  ...
+}
+```
--- a/docs/src/examples/youtube_transcript_bot.md
+++ b/docs/src/examples/youtube_transcript_bot.md
@@ -7,7 +7,7 @@

 <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/youtube_bot/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab">

-Scripts - [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](./examples/youtube_bot/main.py)  [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](./examples/youtube_bot/index.js)
+Scripts - [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://github.com/lancedb/vectordb-recipesexamples/youtube_bot/main.py)  [![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](https://github.com/lancedb/vectordb-recipes/examples/youtube_bot/index.js)


 This example is in a [notebook](https://github.com/lancedb/lancedb/blob/main/docs/src/notebooks/youtube_transcript_search.ipynb)
--- a/docs/src/guides/tables.md
+++ b/docs/src/guides/tables.md
@@ -1,4 +1,5 @@
-A Table is a collection of Records in a LanceDB Database.
+ <a href="https://colab.research.google.com/github/lancedb/lancedb/blob/main/docs/src/notebooks/tables_guide.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a><br/>
+A Table is a collection of Records in a LanceDB Database. You can follow along on colab!

 ## Creating a LanceDB Table

@@ -63,6 +64,25 @@ A Table is a collection of Records in a LanceDB Database.
    table = db.create_table("table3", data, schema=custom_schema)
    ```

+    ### From PyArrow Tables
+    You can also create LanceDB tables directly from pyarrow tables
+    
+    ```python
+    table = pa.Table.from_arrays(
+            [
+                pa.array([[3.1, 4.1], [5.9, 26.5]],
+                        pa.list_(pa.float32(), 2)),
+                pa.array(["foo", "bar"]),
+                pa.array([10.0, 20.0]),
+            ],
+            ["vector", "item", "price"],
+        )
+
+    db = lancedb.connect("db")
+
+    tbl = db.create_table("test1", table)
+    ```
+
    ### From Pydantic Models
    LanceDB supports to create Apache Arrow Schema from a Pydantic BaseModel via pydantic_to_schema() method.

@@ -86,10 +106,14 @@ A Table is a collection of Records in a LanceDB Database.
    table = db.create_table(table_name, schema=Content.to_arrow_schema())
    ```

-    ### Using RecordBatch Iterator / Writing Large Datasets
+    ### Using Iterators / Writing Large Datasets

-    It is recommended to use RecordBatch itertator to add large datasets in batches when creating your table in one go. This does not create multiple versions of your dataset unlike manually adding batches using `table.add()`
+    It is recommended to use itertators to add large datasets in batches when creating your table in one go. This does not create multiple versions of your dataset unlike manually adding batches using `table.add()`

+    LanceDB additionally supports pyarrow's `RecordBatch` Iterators or other generators producing supported data types.
+
+    Here's an example using using `RecordBatch` iterator for creating tables.
+    
    ```python
    import pyarrow as pa

@@ -97,7 +121,8 @@ A Table is a collection of Records in a LanceDB Database.
        for i in range(5):
            yield pa.RecordBatch.from_arrays(
                [
-                    pa.array([[3.1, 4.1], [5.9, 26.5]]),
+                    pa.array([[3.1, 4.1], [5.9, 26.5]],
+                            pa.list_(pa.float32(), 2)),
                    pa.array(["foo", "bar"]),
                    pa.array([10.0, 20.0]),
                ],
@@ -105,7 +130,7 @@ A Table is a collection of Records in a LanceDB Database.
            )

    schema = pa.schema([
-        pa.field("vector", pa.list_(pa.float32())),
+        pa.field("vector", pa.list_(pa.float32(), 2)),
        pa.field("item", pa.utf8()),
        pa.field("price", pa.float32()),
    ])
@@ -113,20 +138,7 @@ A Table is a collection of Records in a LanceDB Database.
    db.create_table("table4", make_batches(), schema=schema)
    ```

-    You can also use Pandas dataframe directly in the above example by converting it to `RecordBatch` object
-
-    ```python
-    import pandas as pd
-    import pyarrow as pa
-
-    df = pd.DataFrame({'vector': [[0,1], [2,3], [4,5],[6,7]],
-                        'month': [3, 5, 7, 9],
-                        'day': [1, 5, 9, 13],
-                        'n_legs': [2, 4, 5, 100],
-                        'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
-
-    batch = pa.RecordBatch.from_pandas(df)
-    ```
+    You can also use iterators of other types like Pandas dataframe or Pylists directly in the above example.

    ## Creating Empty Table
    You can also create empty tables in python. Initialize it with schema and later ingest data into it.
@@ -160,7 +172,6 @@ A Table is a collection of Records in a LanceDB Database.
    
    tbl = db.create_table("table5", schema=Model.to_arrow_schema())
    ```
-    

 === "Javascript/Typescript"

@@ -235,23 +246,21 @@ After a table has been created, you can always add more data to it using
    tbl.add(df)
    ```

-    You can also add a large dataset batch in one go using pyArrow RecordBatch Iterator.
+    You can also add a large dataset batch in one go using Iterator of any supported data types.

-    ### Adding RecordBatch Iterator
+    ### Adding to table using Iterator
    
    ```python
-    import pyarrow as pa
+    import pandas as pd

    def make_batches():
        for i in range(5):
-            yield pa.RecordBatch.from_arrays(
-                [
-                    pa.array([[3.1, 4.1], [5.9, 26.5]]),
-                    pa.array(["foo", "bar"]),
-                    pa.array([10.0, 20.0]),
-                ],
-                ["vector", "item", "price"],
-            )
+            yield pd.DataFrame(
+                {
+                    "vector": [[3.1, 4.1], [1, 1]],
+                    "item": ["foo", "bar"],
+                    "price": [10.0, 20.0],
+                })
    
    tbl.add(make_batches())
    ```
@@ -283,8 +292,6 @@ Use the `delete()` method on tables to delete rows from a table. To choose which
    tbl.delete('item = "fizz"')
    ```

-    ## Examples
-
    ### Deleting row with specific column value

    ```python
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -1,20 +1,23 @@
-# Welcome to LanceDB's Documentation
+# LanceDB

 LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrevial, filtering and management of embeddings.

-The key features of LanceDB include:
+![Illustration](/lancedb/assets/ecosystem-illustration.png)

-* Production-scale vector search with no servers to manage.
+The key features of LanceDB include:

 * Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).

-* Support for vector similarity search, full-text search and SQL.
+* Support for production-scale vector similarity search, full-text search and SQL, with no servers to manage.

 * Native Python and Javascript/Typescript support.

 * Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.

-* Ecosystem integrations with [LangChain 🦜️🔗](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lancedb.html), [LlamaIndex 🦙](https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html), Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
+* Persisted on HDD, allowing scalability without breaking the bank.
+
+* Ingest your favorite data formats directly, like pandas DataFrames, Pydantic objects and more.
+

 LanceDB's core is written in Rust 🦀 and is built using <a href="https://github.com/lancedb/lance">Lance</a>, an open-source columnar format designed for performant ML workloads.

--- a/docs/src/integrations/index.md
+++ b/docs/src/integrations/index.md
@@ -0,0 +1,21 @@
+# Integrations
+
+## Data Formats
+
+LanceDB supports ingesting from your favorite data tools.
+
+![Illustration](/lancedb/assets/ecosystem-illustration.png)
+
+
+## Tools
+
+LanceDB is integrated with most of the popular AI tools, with more coming soon.
+Get started using these examples and quick links.
+
+| Integrations | |
+|---|---:|
+| <h3> LlamaIndex </h3>LlamaIndex is a simple, flexible data framework for connecting custom data sources to large language models. Llama index integrates with LanceDB as the serverless VectorDB. <h3>[Lean More](https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/LanceDBIndexDemo.html) </h3> |<img src="../assets/llama-index.jpg" alt="image" width="150" height="auto">|
+| <h3>Langchain</h3>Langchain allows building applications with LLMs through composability <h3>[Lean More](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/lancedb.html) | <img src="../assets/langchain.png" alt="image" width="150" height="auto">|
+| <h3>Langchain TS</h3> Javascript bindings for Langchain. It integrates with LanceDB's serverless vectordb allowing you to build powerful AI applications through composibility using only serverless functions. <h3>[Learn More]( https://js.langchain.com/docs/modules/data_connection/vectorstores/integrations/lancedb) | <img src="../assets/langchain.png" alt="image" width="150" height="auto">|
+| <h3>Voxel51</h3>  It is an open source toolkit that enables you to build better computer vision workflows by improving the quality of your datasets and delivering insights about your models.<h3>[Learn More](./voxel51.md) | <img src="../assets/voxel.gif" alt="image" width="150" height="auto">|
+| <h3>PromptTools</h3>  Offers a set of free, open-source tools for testing and experimenting with models, prompts, and configurations. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. You can use it to experiment with different configurations of LanceDB, and test how LanceDB integrates with the LLM of your choice.<h3>[Learn More](./prompttools.md) | <img src="../assets/prompttools.jpeg" alt="image" width="150" height="auto">|
--- a/docs/src/integrations/prompttools.md
+++ b/docs/src/integrations/prompttools.md
@@ -0,0 +1,7 @@
+
+[PromptTools](https://github.com/hegelai/prompttools) offers a set of free, open-source tools for testing and experimenting with models, prompts, and configurations. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. You can use it to experiment with different configurations of LanceDB, and test how LanceDB integrates with the LLM of your choice.
+
+[Evaluating Prompts with PromptTools](./examples/prompttools-eval-prompts/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/prompttools-eval-prompts/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
+
+![Alt text](https://prompttools.readthedocs.io/en/latest/_images/demo.gif "a title")
+
--- a/docs/src/notebooks/tables_guide.ipynb
+++ b/docs/src/notebooks/tables_guide.ipynb
@@ -0,0 +1,831 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d24eb4c6-e246-44ca-ba7c-6eae7923bd4c",
+   "metadata": {},
+   "source": [
+    "## LanceDB Tables\n",
+    "A Table is a collection of Records in a LanceDB Database.\n",
+    "\n",
+    "![illustration](../assets/ecosystem-illustration.png)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "id": "c1b4e34b-a49c-471d-a343-a5940bb5138a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install lancedb -qq"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4e5a8d07-d9a1-48c1-913a-8e0629289579",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lancedb\n",
+    "db = lancedb.connect(\"./.lancedb\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "66fb93d5-3551-406b-99b2-488442d61d06",
+   "metadata": {},
+   "source": [
+    "LanceDB allows ingesting data from various sources - `dict`, `list[dict]`, `pd.DataFrame`, `pa.Table` or a `Iterator[pa.RecordBatch]`. Let's take a look at some of the these.\n",
+    "\n",
+    " ### From list of tuples or dictionaries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5df12f66-8d99-43ad-8d0b-22189ec0a6b9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "pyarrow.Table\n",
+       "vector: fixed_size_list<item: float>[2]\n",
+       "  child 0, item: float\n",
+       "lat: double\n",
+       "long: double\n",
+       "----\n",
+       "vector: [[[1.1,1.2],[0.2,1.8]]]\n",
+       "lat: [[45.5,40.1]]\n",
+       "long: [[-122.7,-74.1]]"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import lancedb\n",
+    "\n",
+    "db = lancedb.connect(\"./.lancedb\")\n",
+    "\n",
+    "data = [{\"vector\": [1.1, 1.2], \"lat\": 45.5, \"long\": -122.7},\n",
+    "        {\"vector\": [0.2, 1.8], \"lat\": 40.1, \"long\": -74.1}]\n",
+    "\n",
+    "db.create_table(\"my_table\", data)\n",
+    "\n",
+    "db[\"my_table\"].head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10ce802f-1a10-49ee-8ee3-a9bfb302d86c",
+   "metadata": {},
+   "source": [
+    "## From pandas DataFrame\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "f4d87ae9-0ccb-48eb-b31d-bb8f2370e47e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "pyarrow.Table\n",
+       "vector: fixed_size_list<item: float>[2]\n",
+       "  child 0, item: float\n",
+       "lat: double\n",
+       "long: double\n",
+       "----\n",
+       "vector: [[[1.1,1.2],[0.2,1.8]]]\n",
+       "lat: [[45.5,40.1]]\n",
+       "long: [[-122.7,-74.1]]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "data = pd.DataFrame({\n",
+    "    \"vector\": [[1.1, 1.2], [0.2, 1.8]],\n",
+    "    \"lat\": [45.5, 40.1],\n",
+    "    \"long\": [-122.7, -74.1]\n",
+    "})\n",
+    "\n",
+    "db.create_table(\"table2\", data)\n",
+    "\n",
+    "db[\"table2\"].head() "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4be81469-5b57-4f78-9c72-3938c0378d9d",
+   "metadata": {},
+   "source": [
+    "Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.\n",
+    "  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "25f34bcf-fca0-4431-8601-eac95d1bd347",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "vector: fixed_size_list<item: float>[2]\n",
+       "  child 0, item: float\n",
+       "lat: float\n",
+       "long: float"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import pyarrow as pa\n",
+    "\n",
+    "custom_schema = pa.schema([\n",
+    "pa.field(\"vector\", pa.list_(pa.float32(), 2)),\n",
+    "pa.field(\"lat\", pa.float32()),\n",
+    "pa.field(\"long\", pa.float32())\n",
+    "])\n",
+    "\n",
+    "table = db.create_table(\"table3\", data, schema=custom_schema, mode=\"overwrite\")\n",
+    "table.schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4df51925-7ca2-4005-9c72-38b3d26240c6",
+   "metadata": {},
+   "source": [
+    "### From PyArrow Tables\n",
+    "\n",
+    "You can also create LanceDB tables directly from pyarrow tables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "90a880f6-be43-4c9d-ba65-0b05197c0f6f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "vector: fixed_size_list<item: float>[2]\n",
+       "  child 0, item: float\n",
+       "item: string\n",
+       "price: double"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "table = pa.Table.from_arrays(\n",
+    "        [\n",
+    "            pa.array([[3.1, 4.1], [5.9, 26.5]],\n",
+    "                    pa.list_(pa.float32(), 2)),\n",
+    "            pa.array([\"foo\", \"bar\"]),\n",
+    "            pa.array([10.0, 20.0]),\n",
+    "        ],\n",
+    "        [\"vector\", \"item\", \"price\"],\n",
+    "    )\n",
+    "\n",
+    "db = lancedb.connect(\"db\")\n",
+    "\n",
+    "tbl = db.create_table(\"test1\", table, mode=\"overwrite\")\n",
+    "tbl.schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f36c51c-d902-449d-8292-700e53990c32",
+   "metadata": {},
+   "source": [
+    "### From Pydantic Models\n",
+    "\n",
+    "LanceDB supports to create Apache Arrow Schema from a Pydantic BaseModel."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "d81121d7-e4b7-447c-a48c-974b6ebb464a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "movie_id: int64 not null\n",
+       "vector: fixed_size_list<item: float>[128] not null\n",
+       "  child 0, item: float\n",
+       "genres: string not null\n",
+       "title: string not null\n",
+       "imdb_id: int64 not null"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from lancedb.pydantic import vector, LanceModel\n",
+    "\n",
+    "class Content(LanceModel):\n",
+    "    movie_id: int\n",
+    "    vector: vector(128)\n",
+    "    genres: str\n",
+    "    title: str\n",
+    "    imdb_id: int\n",
+    "        \n",
+    "    @property\n",
+    "    def imdb_url(self) -> str:\n",
+    "        return f\"https://www.imdb.com/title/tt{self.imdb_id}\"\n",
+    "\n",
+    "import pyarrow as pa\n",
+    "db = lancedb.connect(\"~/.lancedb\")\n",
+    "table_name = \"movielens_small\"\n",
+    "table = db.create_table(table_name, schema=Content)\n",
+    "table.schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "860e1f77-e860-46a9-98b7-b2979092ccd6",
+   "metadata": {},
+   "source": [
+    "### Using Iterators / Writing Large Datasets\n",
+    "\n",
+    "It is recommended to use itertators to add large datasets in batches when creating your table in one go. This does not create multiple versions of your dataset unlike manually adding batches using `table.add()`\n",
+    "\n",
+    "LanceDB additionally supports pyarrow's `RecordBatch` Iterators or other generators producing supported data types.\n",
+    "\n",
+    "## Here's an example using using `RecordBatch` iterator for creating tables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "bc247142-4e3c-41a2-b94c-8e00d2c2a508",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "LanceTable(table4)"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import pyarrow as pa\n",
+    "\n",
+    "def make_batches():\n",
+    "    for i in range(5):\n",
+    "        yield pa.RecordBatch.from_arrays(\n",
+    "            [\n",
+    "                pa.array([[3.1, 4.1], [5.9, 26.5]],\n",
+    "                        pa.list_(pa.float32(), 2)),\n",
+    "                pa.array([\"foo\", \"bar\"]),\n",
+    "                pa.array([10.0, 20.0]),\n",
+    "            ],\n",
+    "            [\"vector\", \"item\", \"price\"],\n",
+    "        )\n",
+    "\n",
+    "schema = pa.schema([\n",
+    "    pa.field(\"vector\", pa.list_(pa.float32(), 2)),\n",
+    "    pa.field(\"item\", pa.utf8()),\n",
+    "    pa.field(\"price\", pa.float32()),\n",
+    "])\n",
+    "\n",
+    "db.create_table(\"table4\", make_batches(), schema=schema)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "94f7dd2b-bae4-4bdf-8534-201437c31027",
+   "metadata": {},
+   "source": [
+    "### Using pandas `DataFrame` Iterator and Pydantic Schema\n",
+    "\n",
+    "You can set the schema via pyarrow schema object or using Pydantic object"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "25ad3523-e0c9-4c28-b3df-38189c4e0e5f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "vector: fixed_size_list<item: float>[2] not null\n",
+       "  child 0, item: float\n",
+       "item: string not null\n",
+       "price: double not null"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import pyarrow as pa\n",
+    "import pandas as pd\n",
+    "\n",
+    "class PydanticSchema(LanceModel):\n",
+    "    vector: vector(2)\n",
+    "    item: str\n",
+    "    price: float\n",
+    "\n",
+    "def make_batches():\n",
+    "    for i in range(5):\n",
+    "        yield pd.DataFrame(\n",
+    "                    {\n",
+    "                        \"vector\": [[3.1, 4.1], [1, 1]],\n",
+    "                        \"item\": [\"foo\", \"bar\"],\n",
+    "                        \"price\": [10.0, 20.0],\n",
+    "                    })\n",
+    "\n",
+    "tbl = db.create_table(\"table5\", make_batches(), schema=PydanticSchema)\n",
+    "tbl.schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4aa955e9-fcd0-4c99-b644-f218f3bb3f1a",
+   "metadata": {},
+   "source": [
+    "## Creating Empty Table\n",
+    "\n",
+    "You can create an empty table by just passing the schema and later add to it using `table.add()`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "2814173a-eacc-4dd8-a64d-6312b44582cc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lancedb\n",
+    "from lancedb.pydantic import LanceModel, vector\n",
+    "\n",
+    "class Model(LanceModel):\n",
+    "      vector: vector(2)\n",
+    "\n",
+    "tbl = db.create_table(\"table6\", schema=Model.to_arrow_schema())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d1b0f5c-a1d9-459f-8614-8376b6f577e1",
+   "metadata": {},
+   "source": [
+    "## Open Existing Tables\n",
+    "\n",
+    "If you forget the name of your table, you can always get a listing of all table names:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "df9e13c0-41f6-437f-9dfa-2fd71d3d9c45",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['table6', 'table4', 'table5', 'movielens_small']"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "db.table_names()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "9343f5ad-6024-42ee-ac2f-6c1471df8679",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>vector</th>\n",
+       "      <th>item</th>\n",
+       "      <th>price</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>[3.1, 4.1]</td>\n",
+       "      <td>foo</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>[5.9, 26.5]</td>\n",
+       "      <td>bar</td>\n",
+       "      <td>20.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>[3.1, 4.1]</td>\n",
+       "      <td>foo</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>[5.9, 26.5]</td>\n",
+       "      <td>bar</td>\n",
+       "      <td>20.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>[3.1, 4.1]</td>\n",
+       "      <td>foo</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>[5.9, 26.5]</td>\n",
+       "      <td>bar</td>\n",
+       "      <td>20.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>[3.1, 4.1]</td>\n",
+       "      <td>foo</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>[5.9, 26.5]</td>\n",
+       "      <td>bar</td>\n",
+       "      <td>20.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>[3.1, 4.1]</td>\n",
+       "      <td>foo</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>[5.9, 26.5]</td>\n",
+       "      <td>bar</td>\n",
+       "      <td>20.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "        vector item  price\n",
+       "0   [3.1, 4.1]  foo   10.0\n",
+       "1  [5.9, 26.5]  bar   20.0\n",
+       "2   [3.1, 4.1]  foo   10.0\n",
+       "3  [5.9, 26.5]  bar   20.0\n",
+       "4   [3.1, 4.1]  foo   10.0\n",
+       "5  [5.9, 26.5]  bar   20.0\n",
+       "6   [3.1, 4.1]  foo   10.0\n",
+       "7  [5.9, 26.5]  bar   20.0\n",
+       "8   [3.1, 4.1]  foo   10.0\n",
+       "9  [5.9, 26.5]  bar   20.0"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tbl = db.open_table(\"table4\")\n",
+    "tbl.to_pandas()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5019246f-12e3-4f78-88a8-9f4939802c76",
+   "metadata": {},
+   "source": [
+    "## Adding to table\n",
+    "After a table has been created, you can always add more data to it using\n",
+    "\n",
+    "You can add any of the valid data structures accepted by LanceDB table, i.e, `dict`, `list[dict]`, `pd.DataFrame`, or a `Iterator[pa.RecordBatch]`. Here are some examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "8a56250f-73a1-4c26-a6ad-5c7a0ce3a9ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.DataFrame([{\"vector\": [1.3, 1.4], \"item\": \"fizz\", \"price\": 100.0},\n",
+    "              {\"vector\": [9.5, 56.2], \"item\": \"buzz\", \"price\": 200.0}])\n",
+    "tbl.add(df)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9985f6ee-67e1-45a9-b233-94e3d121ecbf",
+   "metadata": {},
+   "source": [
+    "You can also add a large dataset batch in one go using Iterator of supported data types\n",
+    "\n",
+    "### Adding via Iterator\n",
+    "\n",
+    "here, we'll use pandas DataFrame Iterator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "030c7057-b98e-4e2f-be14-b8c1f927f83c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "def make_batches():\n",
+    "    for i in range(5):\n",
+    "        yield pd.DataFrame(\n",
+    "                {\n",
+    "                    \"vector\": [[3.1, 4.1], [1, 1]],\n",
+    "                    \"item\": [\"foo\", \"bar\"],\n",
+    "                    \"price\": [10.0, 20.0],\n",
+    "                })\n",
+    "tbl.add(make_batches())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b8316d5d-0a23-4675-b0ee-178711db873a",
+   "metadata": {},
+   "source": [
+    "## Deleting from a Table\n",
+    "\n",
+    "Use the `delete()` method on tables to delete rows from a table. To choose which rows to delete, provide a filter that matches on the metadata columns. This can delete any number of rows that match the filter, like:\n",
+    "\n",
+    "\n",
+    "```python\n",
+    "tbl.delete('item = \"fizz\"')\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "e7a17de2-08d2-41b7-bd05-f63d1045ab1f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "32\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "17"
+      ]
+     },
+     "execution_count": 24,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "print(len(tbl))\n",
+    "      \n",
+    "tbl.delete(\"price = 20.0\")\n",
+    "      \n",
+    "len(tbl)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "74ac180b-5432-4c14-b1a8-22c35ac83af8",
+   "metadata": {},
+   "source": [
+    "### Delete from a list of values"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "fe3310bd-08f4-4a22-a63b-b3127d22f9f7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "         vector  item  price\n",
+      "0    [3.1, 4.1]   foo   10.0\n",
+      "1    [3.1, 4.1]   foo   10.0\n",
+      "2    [3.1, 4.1]   foo   10.0\n",
+      "3    [3.1, 4.1]   foo   10.0\n",
+      "4    [3.1, 4.1]   foo   10.0\n",
+      "5    [1.3, 1.4]  fizz  100.0\n",
+      "6   [9.5, 56.2]  buzz  200.0\n",
+      "7    [3.1, 4.1]   foo   10.0\n",
+      "8    [3.1, 4.1]   foo   10.0\n",
+      "9    [3.1, 4.1]   foo   10.0\n",
+      "10   [3.1, 4.1]   foo   10.0\n",
+      "11   [3.1, 4.1]   foo   10.0\n",
+      "12   [3.1, 4.1]   foo   10.0\n",
+      "13   [3.1, 4.1]   foo   10.0\n",
+      "14   [3.1, 4.1]   foo   10.0\n",
+      "15   [3.1, 4.1]   foo   10.0\n",
+      "16   [3.1, 4.1]   foo   10.0\n"
+     ]
+    },
+    {
+     "ename": "OSError",
+     "evalue": "LanceError(IO): Error during planning: column foo does not exist",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[30], line 4\u001b[0m\n\u001b[1;32m      2\u001b[0m to_remove \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin(\u001b[38;5;28mstr\u001b[39m(v) \u001b[38;5;28;01mfor\u001b[39;00m v \u001b[38;5;129;01min\u001b[39;00m to_remove)\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28mprint\u001b[39m(tbl\u001b[38;5;241m.\u001b[39mto_pandas())\n\u001b[0;32m----> 4\u001b[0m \u001b[43mtbl\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdelete\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43mf\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mitem IN (\u001b[39;49m\u001b[38;5;132;43;01m{\u001b[39;49;00m\u001b[43mto_remove\u001b[49m\u001b[38;5;132;43;01m}\u001b[39;49;00m\u001b[38;5;124;43m)\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m tbl\u001b[38;5;241m.\u001b[39mto_pandas()\n",
+      "File \u001b[0;32m~/Documents/lancedb/lancedb/python/lancedb/table.py:610\u001b[0m, in \u001b[0;36mLanceTable.delete\u001b[0;34m(self, where)\u001b[0m\n\u001b[1;32m    609\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mdelete\u001b[39m(\u001b[38;5;28mself\u001b[39m, where: \u001b[38;5;28mstr\u001b[39m):\n\u001b[0;32m--> 610\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_dataset\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdelete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mwhere\u001b[49m\u001b[43m)\u001b[49m\n",
+      "File \u001b[0;32m~/Documents/lancedb/lancedb/env/lib/python3.11/site-packages/lance/dataset.py:489\u001b[0m, in \u001b[0;36mLanceDataset.delete\u001b[0;34m(self, predicate)\u001b[0m\n\u001b[1;32m    487\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(predicate, pa\u001b[38;5;241m.\u001b[39mcompute\u001b[38;5;241m.\u001b[39mExpression):\n\u001b[1;32m    488\u001b[0m     predicate \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mstr\u001b[39m(predicate)\n\u001b[0;32m--> 489\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_ds\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdelete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpredicate\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[0;31mOSError\u001b[0m: LanceError(IO): Error during planning: column foo does not exist"
+     ]
+    }
+   ],
+   "source": [
+    "to_remove = [\"foo\", \"buzz\"]\n",
+    "to_remove = \", \".join(str(v) for v in to_remove)\n",
+    "print(tbl.to_pandas())\n",
+    "tbl.delete(f\"item IN ({to_remove})\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "87d5bc21-847f-4c81-b56e-f6dbe5d05aac",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.DataFrame(\n",
+    "                    {\n",
+    "                        \"vector\": [[3.1, 4.1], [1, 1]],\n",
+    "                        \"item\": [\"foo\", \"bar\"],\n",
+    "                        \"price\": [10.0, 20.0],\n",
+    "                    })\n",
+    "\n",
+    "tbl = db.create_table(\"table7\", data=df, mode=\"overwrite\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "9cba4519-eb3a-4941-ab7e-873d762e750f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "to_remove = [10.0, 20.0]\n",
+    "to_remove = \", \".join(str(v) for v in to_remove)\n",
+    "\n",
+    "tbl.delete(f\"price IN ({to_remove})\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "5bdc9801-d5ed-4871-92d0-88b27108e788",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>vector</th>\n",
+       "      <th>item</th>\n",
+       "      <th>price</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "Empty DataFrame\n",
+       "Columns: [vector, item, price]\n",
+       "Index: []"
+      ]
+     },
+     "execution_count": 46,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tbl.to_pandas()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "752d33d4-ce1c-48e5-90d2-c85f0982182d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
--- a/docs/src/styles/global.css
+++ b/docs/src/styles/global.css
@@ -3,4 +3,13 @@
    --md-primary-fg-color--dark:  #4338ca;
    --md-text-font: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
    --md-code-font: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
-}
+}
+
+.md-nav__item, .md-tabs__item {
+    font-size: large;
+}
+
+/* Maximum space for text block */
+.md-grid {
+    max-width: 90%;
+  }
--- a/docs/test/md_testing.js
+++ b/docs/test/md_testing.js
@@ -2,20 +2,17 @@ const glob = require("glob");
 const fs = require("fs");
 const path = require("path");

-const excludedFiles = [
+const globString = "../src/**/*.md";
+const excludedGlobs = [
  "../src/fts.md",
  "../src/embedding.md",
-  "../src/ann_indexes.md",
-  "../src/examples/serverless_lancedb_with_s3_and_lambda.md",
-  "../src/examples/serverless_qa_bot_with_modal_and_langchain.md",
-  "../src/examples/transformerjs_embedding_search_nodejs.md",
-  "../src/examples/youtube_transcript_bot_with_nodejs.md",
+  "../src/examples/*.md",
  "../src/guides/tables.md",
 ];
+
 const nodePrefix = "javascript";
 const nodeFile = ".js";
 const nodeFolder = "node";
-const globString = "../src/**/*.md";
 const asyncPrefix = "(async () => {\n";
 const asyncSuffix = "})();";

@@ -34,6 +31,7 @@ function* yieldLines(lines, prefix, suffix) {
 }

 const files = glob.sync(globString, { recursive: true });
+const excludedFiles = glob.sync(excludedGlobs, { recursive: true });

 for (const file of files.filter((file) => !excludedFiles.includes(file))) {
  const lines = [];
@@ -51,4 +49,4 @@ for (const file of files.filter((file) => !excludedFiles.includes(file))) {
    fs.mkdirSync(path.dirname(outPath), { recursive: true });
    fs.writeFileSync(outPath, asyncPrefix + "\n" + lines.join("\n") + asyncSuffix);
  }
-}
+}
--- a/docs/test/md_testing.py
+++ b/docs/test/md_testing.py
@@ -2,12 +2,11 @@ import glob
 from typing import Iterator
 from pathlib import Path

-excluded_files = [
+glob_string = "../src/**/*.md"
+excluded_globs = [
    "../src/fts.md",
    "../src/embedding.md",
-    "../src/examples/serverless_lancedb_with_s3_and_lambda.md",
-    "../src/examples/serverless_qa_bot_with_modal_and_langchain.md",
-    "../src/examples/youtube_transcript_bot_with_nodejs.md",
+    "../src/examples/*.md",
    "../src/integrations/voxel51.md",
    "../src/guides/tables.md"
 ]
@@ -15,7 +14,9 @@ excluded_files = [
 python_prefix = "py"
 python_file = ".py"
 python_folder = "python"
-glob_string = "../src/**/*.md"
+
+files = glob.glob(glob_string, recursive=True)
+excluded_files = [f for excluded_glob in excluded_globs for f in glob.glob(excluded_glob, recursive=True)]

 def yield_lines(lines: Iterator[str], prefix: str, suffix: str):
    in_code_block = False
@@ -31,7 +32,7 @@ def yield_lines(lines: Iterator[str], prefix: str, suffix: str):
        elif in_code_block:
            yield line[strip_length:]

-for file in filter(lambda file: file not in excluded_files, glob.glob(glob_string, recursive=True)):
+for file in filter(lambda file: file not in excluded_files, files):
    with open(file, "r") as f:
        lines = list(yield_lines(iter(f), "```", "```"))

@@ -40,4 +41,4 @@ for file in filter(lambda file: file not in excluded_files, glob.glob(glob_strin
        print(out_path)
        out_path.parent.mkdir(exist_ok=True, parents=True)
        with open(out_path, "w") as out:
-            out.writelines(lines)
+            out.writelines(lines)
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.1.19",
+  "version": "0.2.3",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.1.19",
+      "version": "0.2.3",
      "cpu": [
        "x64",
        "arm64"
@@ -51,11 +51,11 @@
        "typescript": "*"
      },
      "optionalDependencies": {
-        "@lancedb/vectordb-darwin-arm64": "0.1.19",
-        "@lancedb/vectordb-darwin-x64": "0.1.19",
-        "@lancedb/vectordb-linux-arm64-gnu": "0.1.19",
-        "@lancedb/vectordb-linux-x64-gnu": "0.1.19",
-        "@lancedb/vectordb-win32-x64-msvc": "0.1.19"
+        "@lancedb/vectordb-darwin-arm64": "0.2.3",
+        "@lancedb/vectordb-darwin-x64": "0.2.3",
+        "@lancedb/vectordb-linux-arm64-gnu": "0.2.3",
+        "@lancedb/vectordb-linux-x64-gnu": "0.2.3",
+        "@lancedb/vectordb-win32-x64-msvc": "0.2.3"
      }
    },
    "node_modules/@apache-arrow/ts": {
@@ -315,9 +315,9 @@
      }
    },
    "node_modules/@lancedb/vectordb-darwin-arm64": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.1.19.tgz",
-      "integrity": "sha512-efQhJkBKvMNhjFq3Sw3/qHo9D9gb9UqiIr98n3STsbNxBQjMnWemXn91Ckl40siRG1O8qXcINW7Qs/EGmus+kg==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.2.3.tgz",
+      "integrity": "sha512-/9dRCXrV/UsZv3fqAC/Q+D2FPKXMRprcb+a77tt4I0Iy5iGT55UDRfpaXvmJeKquhTJkZ0AuyoK5BmOh7cY41w==",
      "cpu": [
        "arm64"
      ],
@@ -327,9 +327,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-darwin-x64": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.1.19.tgz",
-      "integrity": "sha512-r6OZNVyemAssABz2w7CRhe7dyREwBEfTytn+ux1zzTnzsgMgDovCQ0rQ3WZcxWvcy7SFCxiemA9IP1b/lsb4tQ==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.2.3.tgz",
+      "integrity": "sha512-p06WkjmdVwDxkH8ghIWh59SCgUhjXBpy1gQISgktouymqfoFbBHz7vmeI6VO1oBA5ji6vSgGZxqjmeLRKM6blA==",
      "cpu": [
        "x64"
      ],
@@ -339,9 +339,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-linux-arm64-gnu": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.1.19.tgz",
-      "integrity": "sha512-mL/hRmZp6Kw7hmGJBdOZfp/tTYiCdlOcs8DA/+nr2eiXERv0gIhyiKvr2P5DwbBmut3qXEkDalMHTo95BSdL2A==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.2.3.tgz",
+      "integrity": "sha512-cSDcJgfbnRmCXZ3AoRWpCAa07PMdB/k8m1LjmxnhpOnP1ohg1eUl99jwPCgd+5GK+iZmezRqbyO+YXlgsCp7GQ==",
      "cpu": [
        "arm64"
      ],
@@ -351,9 +351,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-linux-x64-gnu": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.1.19.tgz",
-      "integrity": "sha512-AG0FHksbbr+cHVKPi4B8cmBtqb6T9E0uaK4kyZkXrX52/xtv9RYVZcykaB/tSSm0XNFPWWRnx9R8UqNZV/hxMA==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.2.3.tgz",
+      "integrity": "sha512-AFA3J4hBYapGC37iXheiN6tGruitx5bmoWXkUcDv/qAaE4tizVZHB9cgx9ThTB0RDsvZEOZ5zCy7BOzPH+oCOg==",
      "cpu": [
        "x64"
      ],
@@ -363,9 +363,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-win32-x64-msvc": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.1.19.tgz",
-      "integrity": "sha512-PDWZ2hvLVXH4Z4WIO1rsWY8ev3NpNm7aXlaey32P+l1Iz9Hia9+F2GBpp2UiEQKfvbk82ucAvBLRmpSsHY8Tlw==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.2.3.tgz",
+      "integrity": "sha512-LI1mz1HdcpNXTM7HbcLdXz0qvUU4LxSqRC7/kMU918VlOeWy/PnryRrjHnCjcgciGzu1rVlvCqRPh7fVwaG6Kg==",
      "cpu": [
        "x64"
      ],
@@ -4852,33 +4852,33 @@
      }
    },
    "@lancedb/vectordb-darwin-arm64": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.1.19.tgz",
-      "integrity": "sha512-efQhJkBKvMNhjFq3Sw3/qHo9D9gb9UqiIr98n3STsbNxBQjMnWemXn91Ckl40siRG1O8qXcINW7Qs/EGmus+kg==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.2.3.tgz",
+      "integrity": "sha512-/9dRCXrV/UsZv3fqAC/Q+D2FPKXMRprcb+a77tt4I0Iy5iGT55UDRfpaXvmJeKquhTJkZ0AuyoK5BmOh7cY41w==",
      "optional": true
    },
    "@lancedb/vectordb-darwin-x64": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.1.19.tgz",
-      "integrity": "sha512-r6OZNVyemAssABz2w7CRhe7dyREwBEfTytn+ux1zzTnzsgMgDovCQ0rQ3WZcxWvcy7SFCxiemA9IP1b/lsb4tQ==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.2.3.tgz",
+      "integrity": "sha512-p06WkjmdVwDxkH8ghIWh59SCgUhjXBpy1gQISgktouymqfoFbBHz7vmeI6VO1oBA5ji6vSgGZxqjmeLRKM6blA==",
      "optional": true
    },
    "@lancedb/vectordb-linux-arm64-gnu": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.1.19.tgz",
-      "integrity": "sha512-mL/hRmZp6Kw7hmGJBdOZfp/tTYiCdlOcs8DA/+nr2eiXERv0gIhyiKvr2P5DwbBmut3qXEkDalMHTo95BSdL2A==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.2.3.tgz",
+      "integrity": "sha512-cSDcJgfbnRmCXZ3AoRWpCAa07PMdB/k8m1LjmxnhpOnP1ohg1eUl99jwPCgd+5GK+iZmezRqbyO+YXlgsCp7GQ==",
      "optional": true
    },
    "@lancedb/vectordb-linux-x64-gnu": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.1.19.tgz",
-      "integrity": "sha512-AG0FHksbbr+cHVKPi4B8cmBtqb6T9E0uaK4kyZkXrX52/xtv9RYVZcykaB/tSSm0XNFPWWRnx9R8UqNZV/hxMA==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.2.3.tgz",
+      "integrity": "sha512-AFA3J4hBYapGC37iXheiN6tGruitx5bmoWXkUcDv/qAaE4tizVZHB9cgx9ThTB0RDsvZEOZ5zCy7BOzPH+oCOg==",
      "optional": true
    },
    "@lancedb/vectordb-win32-x64-msvc": {
-      "version": "0.1.19",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.1.19.tgz",
-      "integrity": "sha512-PDWZ2hvLVXH4Z4WIO1rsWY8ev3NpNm7aXlaey32P+l1Iz9Hia9+F2GBpp2UiEQKfvbk82ucAvBLRmpSsHY8Tlw==",
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.2.3.tgz",
+      "integrity": "sha512-LI1mz1HdcpNXTM7HbcLdXz0qvUU4LxSqRC7/kMU918VlOeWy/PnryRrjHnCjcgciGzu1rVlvCqRPh7fVwaG6Kg==",
      "optional": true
    },
    "@neon-rs/cli": {
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.1.19",
+  "version": "0.2.3",
  "description": " Serverless, low-latency vector database for AI applications",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
@@ -78,10 +78,10 @@
    }
  },
  "optionalDependencies": {
-    "@lancedb/vectordb-darwin-arm64": "0.1.19",
-    "@lancedb/vectordb-darwin-x64": "0.1.19",
-    "@lancedb/vectordb-linux-arm64-gnu": "0.1.19",
-    "@lancedb/vectordb-linux-x64-gnu": "0.1.19",
-    "@lancedb/vectordb-win32-x64-msvc": "0.1.19"
+    "@lancedb/vectordb-darwin-arm64": "0.2.3",
+    "@lancedb/vectordb-darwin-x64": "0.2.3",
+    "@lancedb/vectordb-linux-arm64-gnu": "0.2.3",
+    "@lancedb/vectordb-linux-x64-gnu": "0.2.3",
+    "@lancedb/vectordb-win32-x64-msvc": "0.2.3"
  }
 }
--- a/node/src/arrow.ts
+++ b/node/src/arrow.ts
@@ -13,18 +13,19 @@
 // limitations under the License.

 import {
-  Field,
+  Field, type FixedSizeListBuilder,
  Float32,
-  List, type ListBuilder,
  makeBuilder,
  RecordBatchFileWriter,
-  Table, Utf8,
+  Utf8,
  type Vector,
-  vectorFromArray
+  FixedSizeList,
+  vectorFromArray, type Schema, Table as ArrowTable
 } from 'apache-arrow'
 import { type EmbeddingFunction } from './index'

-export async function convertToTable<T> (data: Array<Record<string, unknown>>, embeddings?: EmbeddingFunction<T>): Promise<Table> {
+// Converts an Array of records into an Arrow Table, optionally applying an embeddings function to it.
+export async function convertToTable<T> (data: Array<Record<string, unknown>>, embeddings?: EmbeddingFunction<T>): Promise<ArrowTable> {
  if (data.length === 0) {
    throw new Error('At least one record needs to be provided')
  }
@@ -34,8 +35,8 @@ export async function convertToTable<T> (data: Array<Record<string, unknown>>, e

  for (const columnsKey of columns) {
    if (columnsKey === 'vector') {
-      const listBuilder = newVectorListBuilder()
      const vectorSize = (data[0].vector as any[]).length
+      const listBuilder = newVectorBuilder(vectorSize)
      for (const datum of data) {
        if ((datum[columnsKey] as any[]).length !== vectorSize) {
          throw new Error(`Invalid vector size, expected ${vectorSize}`)
@@ -52,9 +53,7 @@ export async function convertToTable<T> (data: Array<Record<string, unknown>>, e

      if (columnsKey === embeddings?.sourceColumn) {
        const vectors = await embeddings.embed(values as T[])
-        const listBuilder = newVectorListBuilder()
-        vectors.map(v => listBuilder.append(v))
-        records.vector = listBuilder.finish().toVector()
+        records.vector = vectorFromArray(vectors, newVectorType(vectors[0].length))
      }

      if (typeof values[0] === 'string') {
@@ -66,20 +65,47 @@ export async function convertToTable<T> (data: Array<Record<string, unknown>>, e
    }
  }

-  return new Table(records)
+  return new ArrowTable(records)
 }

 // Creates a new Arrow ListBuilder that stores a Vector column
-function newVectorListBuilder (): ListBuilder<Float32, any> {
-  const children = new Field<Float32>('item', new Float32())
-  const list = new List(children)
+function newVectorBuilder (dim: number): FixedSizeListBuilder<Float32> {
  return makeBuilder({
-    type: list
+    type: newVectorType(dim)
  })
 }

+// Creates the Arrow Type for a Vector column with dimension `dim`
+function newVectorType (dim: number): FixedSizeList<Float32> {
+  const children = new Field<Float32>('item', new Float32())
+  return new FixedSizeList(dim, children)
+}
+
+// Converts an Array of records into Arrow IPC format
 export async function fromRecordsToBuffer<T> (data: Array<Record<string, unknown>>, embeddings?: EmbeddingFunction<T>): Promise<Buffer> {
  const table = await convertToTable(data, embeddings)
  const writer = RecordBatchFileWriter.writeAll(table)
  return Buffer.from(await writer.toUint8Array())
 }
+
+// Converts an Arrow Table into Arrow IPC format
+export async function fromTableToBuffer<T> (table: ArrowTable, embeddings?: EmbeddingFunction<T>): Promise<Buffer> {
+  if (embeddings !== undefined) {
+    const source = table.getChild(embeddings.sourceColumn)
+
+    if (source === null) {
+      throw new Error(`The embedding source column ${embeddings.sourceColumn} was not found in the Arrow Table`)
+    }
+
+    const vectors = await embeddings.embed(source.toArray() as T[])
+    const column = vectorFromArray(vectors, newVectorType(vectors[0].length))
+    table = table.assign(new ArrowTable({ vector: column }))
+  }
+  const writer = RecordBatchFileWriter.writeAll(table)
+  return Buffer.from(await writer.toUint8Array())
+}
+
+// Creates an empty Arrow Table
+export function createEmptyTable (schema: Schema): ArrowTable {
+  return new ArrowTable(schema)
+}
--- a/node/src/index.ts
+++ b/node/src/index.ts
@@ -13,10 +13,10 @@
 // limitations under the License.

 import {
-  RecordBatchFileWriter,
-  type Table as ArrowTable
+  type Schema,
+  Table as ArrowTable
 } from 'apache-arrow'
-import { fromRecordsToBuffer } from './arrow'
+import { createEmptyTable, fromRecordsToBuffer, fromTableToBuffer } from './arrow'
 import type { EmbeddingFunction } from './embedding/embedding_function'
 import { RemoteConnection } from './remote'
 import { Query } from './query'
@@ -42,6 +42,8 @@ export interface ConnectionOptions {

  awsCredentials?: AwsCredentials

+  awsRegion?: string
+
  // API key for the remote connections
  apiKey?: string
  // Region to connect
@@ -51,6 +53,40 @@ export interface ConnectionOptions {
  hostOverride?: string
 }

+function getAwsArgs (opts: ConnectionOptions): any[] {
+  const callArgs = []
+  const awsCredentials = opts.awsCredentials
+  if (awsCredentials !== undefined) {
+    callArgs.push(awsCredentials.accessKeyId)
+    callArgs.push(awsCredentials.secretKey)
+    callArgs.push(awsCredentials.sessionToken)
+  } else {
+    callArgs.push(undefined)
+    callArgs.push(undefined)
+    callArgs.push(undefined)
+  }
+
+  callArgs.push(opts.awsRegion)
+  return callArgs
+}
+
+export interface CreateTableOptions<T> {
+  // Name of Table
+  name: string
+
+  // Data to insert into the Table
+  data?: Array<Record<string, unknown>> | ArrowTable | undefined
+
+  // Optional Arrow Schema for this table
+  schema?: Schema | undefined
+
+  // Optional embedding function used to create embeddings
+  embeddingFunction?: EmbeddingFunction<T> | undefined
+
+  // WriteOptions for this operation
+  writeOptions?: WriteOptions | undefined
+}
+
 /**
 * Connect to a LanceDB instance at the given URI
 * @param uri The uri of the database.
@@ -97,6 +133,17 @@ export interface Connection {
   */
  openTable<T>(name: string, embeddings?: EmbeddingFunction<T>): Promise<Table<T>>

+  /**
+   * Creates a new Table, optionally initializing it with new data.
+   *
+   * @param {string} name - The name of the table.
+   * @param data - Array of Records to be inserted into the table
+   * @param schema - An Arrow Schema that describe this table columns
+   * @param {EmbeddingFunction} embeddings - An embedding function to use on this table
+   * @param {WriteOptions} writeOptions - The write options to use when creating the table.
+   */
+  createTable<T> ({ name, data, schema, embeddingFunction, writeOptions }: CreateTableOptions<T>): Promise<Table<T>>
+
  /**
   * Creates a new Table and initialize it with new data.
   *
@@ -132,8 +179,6 @@ export interface Connection {
   */
  createTable<T> (name: string, data: Array<Record<string, unknown>>, embeddings: EmbeddingFunction<T>, options: WriteOptions): Promise<Table<T>>

-  createTableArrow(name: string, table: ArrowTable): Promise<Table>
-
  /**
   * Drop an existing table.
   * @param name The name of the table to drop.
@@ -221,16 +266,16 @@ export interface Table<T = number[]> {
 * A connection to a LanceDB database.
 */
 export class LocalConnection implements Connection {
-  private readonly _options: ConnectionOptions
+  private readonly _options: () => ConnectionOptions
  private readonly _db: any

  constructor (db: any, options: ConnectionOptions) {
-    this._options = options
+    this._options = () => options
    this._db = db
  }

  get uri (): string {
-    return this._options.uri
+    return this._options().uri
  }

  /**
@@ -256,48 +301,66 @@ export class LocalConnection implements Connection {
  async openTable<T> (name: string, embeddings: EmbeddingFunction<T>): Promise<Table<T>>
  async openTable<T> (name: string, embeddings?: EmbeddingFunction<T>): Promise<Table<T>>
  async openTable<T> (name: string, embeddings?: EmbeddingFunction<T>): Promise<Table<T>> {
-    const tbl = await databaseOpenTable.call(this._db, name)
+    const tbl = await databaseOpenTable.call(this._db, name, ...getAwsArgs(this._options()))
    if (embeddings !== undefined) {
-      return new LocalTable(tbl, name, this._options, embeddings)
+      return new LocalTable(tbl, name, this._options(), embeddings)
    } else {
-      return new LocalTable(tbl, name, this._options)
+      return new LocalTable(tbl, name, this._options())
    }
  }

-  async createTable<T> (name: string, data: Array<Record<string, unknown>>, optsOrEmbedding?: WriteOptions | EmbeddingFunction<T>, opt?: WriteOptions): Promise<Table<T>> {
-    let writeOptions: WriteOptions = new DefaultWriteOptions()
-    if (opt !== undefined && isWriteOptions(opt)) {
-      writeOptions = opt
-    } else if (optsOrEmbedding !== undefined && isWriteOptions(optsOrEmbedding)) {
-      writeOptions = optsOrEmbedding
-    }
-
-    let embeddings: undefined | EmbeddingFunction<T>
-    if (optsOrEmbedding !== undefined && isEmbeddingFunction(optsOrEmbedding)) {
-      embeddings = optsOrEmbedding
-    }
-    const createArgs = [this._db, name, await fromRecordsToBuffer(data, embeddings), writeOptions.writeMode?.toString()]
-    if (this._options.awsCredentials !== undefined) {
-      createArgs.push(this._options.awsCredentials.accessKeyId)
-      createArgs.push(this._options.awsCredentials.secretKey)
-      if (this._options.awsCredentials.sessionToken !== undefined) {
-        createArgs.push(this._options.awsCredentials.sessionToken)
+  async createTable<T> (name: string | CreateTableOptions<T>, data?: Array<Record<string, unknown>>, optsOrEmbedding?: WriteOptions | EmbeddingFunction<T>, opt?: WriteOptions): Promise<Table<T>> {
+    if (typeof name === 'string') {
+      let writeOptions: WriteOptions = new DefaultWriteOptions()
+      if (opt !== undefined && isWriteOptions(opt)) {
+        writeOptions = opt
+      } else if (optsOrEmbedding !== undefined && isWriteOptions(optsOrEmbedding)) {
+        writeOptions = optsOrEmbedding
      }
-    }

-    const tbl = await tableCreate.call(...createArgs)
-
-    if (embeddings !== undefined) {
-      return new LocalTable(tbl, name, this._options, embeddings)
-    } else {
-      return new LocalTable(tbl, name, this._options)
+      let embeddings: undefined | EmbeddingFunction<T>
+      if (optsOrEmbedding !== undefined && isEmbeddingFunction(optsOrEmbedding)) {
+        embeddings = optsOrEmbedding
+      }
+      return await this.createTableImpl({ name, data, embeddingFunction: embeddings, writeOptions })
    }
+    return await this.createTableImpl(name)
  }

-  async createTableArrow (name: string, table: ArrowTable): Promise<Table> {
-    const writer = RecordBatchFileWriter.writeAll(table)
-    await tableCreate.call(this._db, name, Buffer.from(await writer.toUint8Array()))
-    return await this.openTable(name)
+  private async createTableImpl<T> ({ name, data, schema, embeddingFunction, writeOptions = new DefaultWriteOptions() }: {
+    name: string
+    data?: Array<Record<string, unknown>> | ArrowTable | undefined
+    schema?: Schema | undefined
+    embeddingFunction?: EmbeddingFunction<T> | undefined
+    writeOptions?: WriteOptions | undefined
+  }): Promise<Table<T>> {
+    let buffer: Buffer
+
+    function isEmpty (data: Array<Record<string, unknown>> | ArrowTable<any>): boolean {
+      if (data instanceof ArrowTable) {
+        return data.data.length === 0
+      }
+      return data.length === 0
+    }
+
+    if ((data === undefined) || isEmpty(data)) {
+      if (schema === undefined) {
+        throw new Error('Either data or schema needs to defined')
+      }
+      buffer = await fromTableToBuffer(createEmptyTable(schema))
+    } else if (data instanceof ArrowTable) {
+      buffer = await fromTableToBuffer(data, embeddingFunction)
+    } else {
+      // data is Array<Record<...>>
+      buffer = await fromRecordsToBuffer(data, embeddingFunction)
+    }
+
+    const tbl = await tableCreate.call(this._db, name, buffer, writeOptions?.writeMode?.toString(), ...getAwsArgs(this._options()))
+    if (embeddingFunction !== undefined) {
+      return new LocalTable(tbl, name, this._options(), embeddingFunction)
+    } else {
+      return new LocalTable(tbl, name, this._options())
+    }
  }

  /**
@@ -313,7 +376,7 @@ export class LocalTable<T = number[]> implements Table<T> {
  private _tbl: any
  private readonly _name: string
  private readonly _embeddings?: EmbeddingFunction<T>
-  private readonly _options: ConnectionOptions
+  private readonly _options: () => ConnectionOptions

  constructor (tbl: any, name: string, options: ConnectionOptions)
  /**
@@ -327,7 +390,7 @@ export class LocalTable<T = number[]> implements Table<T> {
    this._tbl = tbl
    this._name = name
    this._embeddings = embeddings
-    this._options = options
+    this._options = () => options
  }

  get name (): string {
@@ -349,15 +412,12 @@ export class LocalTable<T = number[]> implements Table<T> {
   * @return The number of rows added to the table
   */
  async add (data: Array<Record<string, unknown>>): Promise<number> {
-    const callArgs = [this._tbl, await fromRecordsToBuffer(data, this._embeddings), WriteMode.Append.toString()]
-    if (this._options.awsCredentials !== undefined) {
-      callArgs.push(this._options.awsCredentials.accessKeyId)
-      callArgs.push(this._options.awsCredentials.secretKey)
-      if (this._options.awsCredentials.sessionToken !== undefined) {
-        callArgs.push(this._options.awsCredentials.sessionToken)
-      }
-    }
-    return tableAdd.call(...callArgs).then((newTable: any) => { this._tbl = newTable })
+    return tableAdd.call(
+      this._tbl,
+      await fromRecordsToBuffer(data, this._embeddings),
+      WriteMode.Append.toString(),
+      ...getAwsArgs(this._options())
+    ).then((newTable: any) => { this._tbl = newTable })
  }

  /**
@@ -367,15 +427,12 @@ export class LocalTable<T = number[]> implements Table<T> {
   * @return The number of rows added to the table
   */
  async overwrite (data: Array<Record<string, unknown>>): Promise<number> {
-    const callArgs = [this._tbl, await fromRecordsToBuffer(data, this._embeddings), WriteMode.Overwrite.toString()]
-    if (this._options.awsCredentials !== undefined) {
-      callArgs.push(this._options.awsCredentials.accessKeyId)
-      callArgs.push(this._options.awsCredentials.secretKey)
-      if (this._options.awsCredentials.sessionToken !== undefined) {
-        callArgs.push(this._options.awsCredentials.sessionToken)
-      }
-    }
-    return tableAdd.call(...callArgs).then((newTable: any) => { this._tbl = newTable })
+    return tableAdd.call(
+      this._tbl,
+      await fromRecordsToBuffer(data, this._embeddings),
+      WriteMode.Overwrite.toString(),
+      ...getAwsArgs(this._options())
+    ).then((newTable: any) => { this._tbl = newTable })
  }

  /**
--- a/node/src/query.ts
+++ b/node/src/query.ts
@@ -112,7 +112,8 @@ export class Query<T = number[]> {
      this._queryVector = this._query as number[]
    }

-    const buffer = await tableSearch.call(this._tbl, this)
+    const isElectron = this.isElectron()
+    const buffer = await tableSearch.call(this._tbl, this, isElectron)
    const data = tableFromIPC(buffer)

    return data.toArray().map((entry: Record<string, unknown>) => {
@@ -127,4 +128,14 @@ export class Query<T = number[]> {
      return newObject as unknown as T
    })
  }
+
+  // See https://github.com/electron/electron/issues/2288
+  private isElectron (): boolean {
+    try {
+      // eslint-disable-next-line no-prototype-builtins
+      return (process?.versions?.hasOwnProperty('electron') || navigator?.userAgent?.toLowerCase()?.includes(' electron'))
+    } catch (e) {
+      return false
+    }
+  }
 }
--- a/node/src/remote/index.ts
+++ b/node/src/remote/index.ts
@@ -14,11 +14,11 @@

 import {
  type EmbeddingFunction, type Table, type VectorIndexParams, type Connection,
-  type ConnectionOptions
+  type ConnectionOptions, type CreateTableOptions, type WriteOptions
 } from '../index'
 import { Query } from '../query'

-import { type Table as ArrowTable, Vector } from 'apache-arrow'
+import { Vector } from 'apache-arrow'
 import { HttpLancedbClient } from './client'

 /**
@@ -66,13 +66,7 @@ export class RemoteConnection implements Connection {
    }
  }

-  async createTable (name: string, data: Array<Record<string, unknown>>): Promise<Table>
-  async createTable<T> (name: string, data: Array<Record<string, unknown>>, embeddings: EmbeddingFunction<T>): Promise<Table<T>>
-  async createTable<T> (name: string, data: Array<Record<string, unknown>>, embeddings?: EmbeddingFunction<T>): Promise<Table<T>> {
-    throw new Error('Not implemented')
-  }
-
-  async createTableArrow (name: string, table: ArrowTable): Promise<Table> {
+  async createTable<T> (name: string | CreateTableOptions<T>, data?: Array<Record<string, unknown>>, optsOrEmbedding?: WriteOptions | EmbeddingFunction<T>, opt?: WriteOptions): Promise<Table<T>> {
    throw new Error('Not implemented')
  }

--- a/node/src/test/io.ts
+++ b/node/src/test/io.ts
@@ -47,7 +47,9 @@ describe('LanceDB S3 client', function () {
        }
      }
      const table = await createTestDB(opts, 2, 20)
+      console.log(table)
      const con = await lancedb.connect(opts)
+      console.log(con)
      assert.equal(con.uri, opts.uri)

      const results = await table.search([0.1, 0.3]).limit(5).execute()
@@ -70,5 +72,5 @@ async function createTestDB (opts: ConnectionOptions, numDimensions: number = 2,
    data.push({ id: i + 1, name: `name_${i}`, price: i + 10, is_active: (i % 2 === 0), vector })
  }

-  return await con.createTable('vectors', data)
+  return await con.createTable('vectors_2', data)
 }
--- a/node/src/test/test.ts
+++ b/node/src/test/test.ts
@@ -19,6 +19,7 @@ import * as chaiAsPromised from 'chai-as-promised'

 import * as lancedb from '../index'
 import { type AwsCredentials, type EmbeddingFunction, MetricType, Query, WriteMode, DefaultWriteOptions, isWriteOptions } from '../index'
+import { Field, Int32, makeVector, Schema, Utf8, Table as ArrowTable, vectorFromArray } from 'apache-arrow'

 const expect = chai.expect
 const assert = chai.assert
@@ -119,6 +120,45 @@ describe('LanceDB client', function () {
  })

  describe('when creating a new dataset', function () {
+    it('create an empty table', async function () {
+      const dir = await track().mkdir('lancejs')
+      const con = await lancedb.connect(dir)
+
+      const schema = new Schema(
+        [new Field('id', new Int32()), new Field('name', new Utf8())]
+      )
+      const table = await con.createTable({ name: 'vectors', schema })
+      assert.equal(table.name, 'vectors')
+      assert.deepEqual(await con.tableNames(), ['vectors'])
+    })
+
+    it('create a table with a empty data array', async function () {
+      const dir = await track().mkdir('lancejs')
+      const con = await lancedb.connect(dir)
+
+      const schema = new Schema(
+        [new Field('id', new Int32()), new Field('name', new Utf8())]
+      )
+      const table = await con.createTable({ name: 'vectors', schema, data: [] })
+      assert.equal(table.name, 'vectors')
+      assert.deepEqual(await con.tableNames(), ['vectors'])
+    })
+
+    it('create a table from an Arrow Table', async function () {
+      const dir = await track().mkdir('lancejs')
+      const con = await lancedb.connect(dir)
+
+      const i32s = new Int32Array(new Array<number>(10))
+      const i32 = makeVector(i32s)
+
+      const data = new ArrowTable({ vector: i32 })
+
+      const table = await con.createTable({ name: 'vectors', data })
+      assert.equal(table.name, 'vectors')
+      assert.equal(await table.countRows(), 10)
+      assert.deepEqual(await con.tableNames(), ['vectors'])
+    })
+
    it('creates a new table from javascript objects', async function () {
      const dir = await track().mkdir('lancejs')
      const con = await lancedb.connect(dir)
@@ -291,6 +331,20 @@ describe('LanceDB client', function () {
      const results = await table.search('foo').execute()
      assert.equal(results.length, 2)
    })
+
+    it('should create embeddings for Arrow Table', async function () {
+      const dir = await track().mkdir('lancejs')
+      const con = await lancedb.connect(dir)
+      const embeddingFunction = new TextEmbedding('name')
+
+      const names = vectorFromArray(['foo', 'bar'], new Utf8())
+      const data = new ArrowTable({ name: names })
+
+      const table = await con.createTable({ name: 'vectors', data, embeddingFunction })
+      assert.equal(table.name, 'vectors')
+      const results = await table.search('foo').execute()
+      assert.equal(results.length, 2)
+    })
  })
 })

--- a/python/.bumpversion.cfg
+++ b/python/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.2.0
+current_version = 0.2.1
 commit = True
 message = [python] Bump version: {current_version} → {new_version}
 tag = True
--- a/python/lancedb/db.py
+++ b/python/lancedb/db.py
@@ -149,14 +149,14 @@ class DBConnection(ABC):
        ...     for i in range(5):
        ...         yield pa.RecordBatch.from_arrays(
        ...             [
-        ...                 pa.array([[3.1, 4.1], [5.9, 26.5]]),
+        ...                 pa.array([[3.1, 4.1], [5.9, 26.5]], pa.list_(pa.float32(), 2)),
        ...                 pa.array(["foo", "bar"]),
        ...                 pa.array([10.0, 20.0]),
        ...             ],
        ...             ["vector", "item", "price"],
        ...         )
        >>> schema=pa.schema([
-        ...     pa.field("vector", pa.list_(pa.float32())),
+        ...     pa.field("vector", pa.list_(pa.float32(), 2)),
        ...     pa.field("item", pa.utf8()),
        ...     pa.field("price", pa.float32()),
        ... ])
--- a/python/lancedb/table.py
+++ b/python/lancedb/table.py
@@ -56,11 +56,22 @@ def _sanitize_data(data, schema, on_bad_vectors, fill_value):
        metadata = {k: v for k, v in metadata.items() if k != b"pandas"}
        schema = data.schema.with_metadata(metadata)
        data = pa.Table.from_arrays(data.columns, schema=schema)
+    if isinstance(data, Iterable):
+        data = _to_record_batch_generator(data, schema, on_bad_vectors, fill_value)
    if not isinstance(data, (pa.Table, Iterable)):
        raise TypeError(f"Unsupported data type: {type(data)}")
    return data


+def _to_record_batch_generator(data: Iterable, schema, on_bad_vectors, fill_value):
+    for batch in data:
+        if not isinstance(batch, pa.RecordBatch):
+            table = _sanitize_data(batch, schema, on_bad_vectors, fill_value)
+            for batch in table.to_batches():
+                yield batch
+        yield batch
+
+
 class Table(ABC):
    """
    A [Table](Table) is a collection of Records in a LanceDB [Database](Database).
@@ -268,10 +279,11 @@ class LanceTable(Table):
        self.name = name
        self._version = version

-    def _reset_dataset(self):
+    def _reset_dataset(self, version=None):
        try:
            if "_dataset" in self.__dict__:
                del self.__dict__["_dataset"]
+            self._version = version
        except AttributeError:
            pass

@@ -297,7 +309,9 @@ class LanceTable(Table):
    def checkout(self, version: int):
        """Checkout a version of the table. This is an in-place operation.

-        This allows viewing previous versions of the table.
+        This allows viewing previous versions of the table. If you wish to
+        keep writing to the dataset starting from an old version, then use
+        the `restore` function instead.

        Parameters
        ----------
@@ -325,7 +339,49 @@ class LanceTable(Table):
        max_ver = max([v["version"] for v in self._dataset.versions()])
        if version < 1 or version > max_ver:
            raise ValueError(f"Invalid version {version}")
-        self._version = version
+        self._reset_dataset(version=version)
+
+    def restore(self, version: int):
+        """Restore a version of the table. This is an in-place operation.
+
+        This creates a new version where the data is equivalent to the
+        specified previous version. Note that this creates a new snapshot.
+
+        Parameters
+        ----------
+        version : int
+            The version to restore.
+
+        Examples
+        --------
+        >>> import lancedb
+        >>> db = lancedb.connect("./.lancedb")
+        >>> table = db.create_table("my_table", [{"vector": [1.1, 0.9], "type": "vector"}])
+        >>> table.version
+        1
+        >>> table.to_pandas()
+               vector    type
+        0  [1.1, 0.9]  vector
+        >>> table.add([{"vector": [0.5, 0.2], "type": "vector"}])
+        >>> table.version
+        2
+        >>> table.restore(1)
+        >>> table.to_pandas()
+               vector    type
+        0  [1.1, 0.9]  vector
+        >>> len(table.list_versions())
+        3
+        """
+        max_ver = max([v["version"] for v in self._dataset.versions()])
+        if version < 1 or version >= max_ver:
+            raise ValueError(f"Invalid version {version}")
+        if version == max_ver:
+            self._reset_dataset()
+            return
+        self.checkout(version)
+        data = self.to_arrow()
+        self.checkout(max_ver)
+        self.add(data, mode="overwrite")
        self._reset_dataset()

    def __len__(self):
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -1,8 +1,8 @@
 [project]
 name = "lancedb"
-version = "0.2.0"
+version = "0.2.1"
 dependencies = [
-    "pylance==0.6.1",
+    "pylance==0.6.5",
    "ratelimiter",
    "retry",
    "tqdm",
--- a/python/tests/test_db.py
+++ b/python/tests/test_db.py
@@ -17,7 +17,7 @@ import pyarrow as pa
 import pytest

 import lancedb
-from lancedb.pydantic import LanceModel
+from lancedb.pydantic import LanceModel, vector


 def test_basic(tmp_path):
@@ -77,35 +77,78 @@ def test_ingest_pd(tmp_path):
    assert db.open_table("test").name == db["test"].name


-def test_ingest_record_batch_iterator(tmp_path):
-    def batch_reader():
-        for i in range(5):
-            yield pa.RecordBatch.from_arrays(
-                [
-                    pa.array([[3.1, 4.1], [5.9, 26.5]]),
-                    pa.array(["foo", "bar"]),
-                    pa.array([10.0, 20.0]),
-                ],
-                ["vector", "item", "price"],
-            )
+def test_ingest_iterator(tmp_path):
+    class PydanticSchema(LanceModel):
+        vector: vector(2)
+        item: str
+        price: float

-    db = lancedb.connect(tmp_path)
-    tbl = db.create_table(
-        "test",
-        batch_reader(),
-        schema=pa.schema(
-            [
-                pa.field("vector", pa.list_(pa.float32())),
-                pa.field("item", pa.utf8()),
-                pa.field("price", pa.float32()),
-            ]
-        ),
+    arrow_schema = pa.schema(
+        [
+            pa.field("vector", pa.list_(pa.float32(), 2)),
+            pa.field("item", pa.utf8()),
+            pa.field("price", pa.float32()),
+        ]
    )

-    tbl_len = len(tbl)
-    tbl.add(batch_reader())
-    assert len(tbl) == tbl_len * 2
-    assert len(tbl.list_versions()) == 2
+    def make_batches():
+        for _ in range(5):
+            yield from [
+                # pandas
+                pd.DataFrame(
+                    {
+                        "vector": [[3.1, 4.1], [1, 1]],
+                        "item": ["foo", "bar"],
+                        "price": [10.0, 20.0],
+                    }
+                ),
+                # pylist
+                [
+                    {"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
+                    {"vector": [5.9, 26.5], "item": "bar", "price": 20.0},
+                ],
+                # recordbatch
+                pa.RecordBatch.from_arrays(
+                    [
+                        pa.array([[3.1, 4.1], [5.9, 26.5]], pa.list_(pa.float32(), 2)),
+                        pa.array(["foo", "bar"]),
+                        pa.array([10.0, 20.0]),
+                    ],
+                    ["vector", "item", "price"],
+                ),
+                # pa Table
+                pa.Table.from_arrays(
+                    [
+                        pa.array([[3.1, 4.1], [5.9, 26.5]], pa.list_(pa.float32(), 2)),
+                        pa.array(["foo", "bar"]),
+                        pa.array([10.0, 20.0]),
+                    ],
+                    ["vector", "item", "price"],
+                ),
+                # pydantic list
+                [
+                    PydanticSchema(vector=[3.1, 4.1], item="foo", price=10.0),
+                    PydanticSchema(vector=[5.9, 26.5], item="bar", price=20.0),
+                ]
+                # TODO: test pydict separately. it is unique column number and names contraint
+            ]
+
+    def run_tests(schema):
+        db = lancedb.connect(tmp_path)
+        tbl = db.create_table("table2", make_batches(), schema=schema, mode="overwrite")
+
+        tbl.to_pandas()
+        assert tbl.search([3.1, 4.1]).limit(1).to_df()["_distance"][0] == 0.0
+        assert tbl.search([5.9, 26.5]).limit(1).to_df()["_distance"][0] == 0.0
+
+        tbl_len = len(tbl)
+        tbl.add(make_batches())
+        assert len(tbl) == tbl_len * 2
+        assert len(tbl.list_versions()) == 2
+        db.drop_database()
+
+    run_tests(arrow_schema)
+    run_tests(PydanticSchema)


 def test_create_mode(tmp_path):
--- a/python/tests/test_table.py
+++ b/python/tests/test_table.py
@@ -268,3 +268,15 @@ def test_add_with_nans(db):
    arrow_tbl = table.to_lance().to_table(filter="item == 'bar'")
    v = arrow_tbl["vector"].to_pylist()[0]
    assert np.allclose(v, np.array([0.0, 0.0]))
+
+
+def test_restore(db):
+    table = LanceTable.create(
+        db,
+        "my_table",
+        data=[{"vector": [1.1, 0.9], "type": "vector"}],
+    )
+    table.add([{"vector": [0.5, 0.2], "type": "vector"}])
+    table.restore(1)
+    assert len(table.list_versions()) == 3
+    assert len(table) == 1
--- a/rust/ffi/node/Cargo.toml
+++ b/rust/ffi/node/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "vectordb-node"
-version = "0.1.19"
+version = "0.2.3"
 description = "Serverless, low-latency vector database for AI applications"
 license = "Apache-2.0"
 edition = "2018"
--- a/rust/ffi/node/src/arrow.rs
+++ b/rust/ffi/node/src/arrow.rs
@@ -14,60 +14,33 @@

 use std::io::Cursor;
 use std::ops::Deref;
-use std::sync::Arc;

-use arrow_array::cast::as_list_array;
-use arrow_array::{Array, ArrayRef, FixedSizeListArray, RecordBatch};
+use arrow_array::RecordBatch;
 use arrow_ipc::reader::FileReader;
 use arrow_ipc::writer::FileWriter;
-use arrow_schema::{DataType, Field, Schema};
-use lance::arrow::{FixedSizeListArrayExt, RecordBatchExt};
+use arrow_schema::SchemaRef;
 use vectordb::table::VECTOR_COLUMN_NAME;

 use crate::error::{MissingColumnSnafu, Result};
 use snafu::prelude::*;

-pub(crate) fn convert_record_batch(record_batch: RecordBatch) -> Result<RecordBatch> {
-    let column = get_column(VECTOR_COLUMN_NAME, &record_batch)?;
-
-    // TODO: we should just consume the underlying js buffer in the future instead of this arrow around a bunch of times
-    let arr = as_list_array(column.as_ref());
-    let list_size = arr.values().len() / record_batch.num_rows();
-    let r = FixedSizeListArray::try_new_from_values(arr.values().to_owned(), list_size as i32)?;
-
-    let schema = Arc::new(Schema::new(vec![Field::new(
-        VECTOR_COLUMN_NAME,
-        DataType::FixedSizeList(
-            Arc::new(Field::new("item", DataType::Float32, true)),
-            list_size as i32,
-        ),
-        true,
-    )]));
-
-    let mut new_batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(r)])?;
-
-    if record_batch.num_columns() > 1 {
-        let rb = record_batch.drop_column(VECTOR_COLUMN_NAME)?;
-        new_batch = new_batch.merge(&rb)?;
-    }
-    Ok(new_batch)
-}
-
-fn get_column(column_name: &str, record_batch: &RecordBatch) -> Result<ArrayRef> {
+fn validate_vector_column(record_batch: &RecordBatch) -> Result<()> {
    record_batch
-        .column_by_name(column_name)
-        .cloned()
-        .context(MissingColumnSnafu { name: column_name })
+        .column_by_name(VECTOR_COLUMN_NAME)
+        .map(|_| ())
+        .context(MissingColumnSnafu { name: VECTOR_COLUMN_NAME })
 }

-pub(crate) fn arrow_buffer_to_record_batch(slice: &[u8]) -> Result<Vec<RecordBatch>> {
+pub(crate) fn arrow_buffer_to_record_batch(slice: &[u8]) -> Result<(Vec<RecordBatch>, SchemaRef)> {
    let mut batches: Vec<RecordBatch> = Vec::new();
    let file_reader = FileReader::try_new(Cursor::new(slice), None)?;
+    let schema = file_reader.schema().clone();
    for b in file_reader {
-        let record_batch = convert_record_batch(b?)?;
+        let record_batch = b?;
+        validate_vector_column(&record_batch)?;
        batches.push(record_batch);
    }
-    Ok(batches)
+    Ok((batches, schema))
 }

 pub(crate) fn record_batch_to_buffer(batches: Vec<RecordBatch>) -> Result<Vec<u8>> {
--- a/rust/ffi/node/src/lib.rs
+++ b/rust/ffi/node/src/lib.rs
@@ -121,26 +121,28 @@ fn database_table_names(mut cx: FunctionContext) -> JsResult<JsPromise> {
    Ok(promise)
 }

-fn get_aws_creds<T>(
+/// Get AWS creds arguments from the context
+/// Consumes 3 arguments
+fn get_aws_creds(
    cx: &mut FunctionContext,
    arg_starting_location: i32,
-) -> Result<Option<AwsCredentialProvider>, NeonResult<T>> {
+) -> NeonResult<Option<AwsCredentialProvider>> {
    let secret_key_id = cx
        .argument_opt(arg_starting_location)
-        .map(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
-        .flatten()
+        .filter(|arg| arg.is_a::<JsString, _>(cx))
+        .and_then(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
        .map(|v| v.value(cx));

    let secret_key = cx
        .argument_opt(arg_starting_location + 1)
-        .map(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
-        .flatten()
+        .filter(|arg| arg.is_a::<JsString, _>(cx))
+        .and_then(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
        .map(|v| v.value(cx));

    let temp_token = cx
        .argument_opt(arg_starting_location + 2)
-        .map(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
-        .flatten()
+        .filter(|arg| arg.is_a::<JsString, _>(cx))
+        .and_then(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx).ok())
        .map(|v| v.value(cx));

    match (secret_key_id, secret_key, temp_token) {
@@ -152,7 +154,21 @@ fn get_aws_creds<T>(
            }),
        ))),
        (None, None, None) => Ok(None),
-        _ => Err(cx.throw_error("Invalid credentials configuration")),
+        _ => cx.throw_error("Invalid credentials configuration"),
+    }
+}
+
+/// Get AWS region arguments from the context
+fn get_aws_region(cx: &mut FunctionContext, arg_location: i32) -> NeonResult<Option<String>> {
+    let region = cx
+        .argument_opt(arg_location)
+        .filter(|arg| arg.is_a::<JsString, _>(cx))
+        .map(|arg| arg.downcast_or_throw::<JsString, FunctionContext>(cx));
+
+    match region {
+        Some(Ok(region)) => Ok(Some(region.value(cx))),
+        None => Ok(None),
+        Some(Err(e)) => Err(e),
    }
 }

@@ -162,14 +178,14 @@ fn database_open_table(mut cx: FunctionContext) -> JsResult<JsPromise> {
        .downcast_or_throw::<JsBox<JsDatabase>, _>(&mut cx)?;
    let table_name = cx.argument::<JsString>(0)?.value(&mut cx);

-    let aws_creds = match get_aws_creds(&mut cx, 1) {
-        Ok(creds) => creds,
-        Err(err) => return err,
-    };
+    let aws_creds = get_aws_creds(&mut cx, 1)?;
+
+    let aws_region = get_aws_region(&mut cx, 4)?;

    let params = ReadParams {
        store_options: Some(ObjectStoreParams {
            aws_credentials: aws_creds,
+            aws_region,
            ..ObjectStoreParams::default()
        }),
        ..ReadParams::default()
--- a/rust/ffi/node/src/query.rs
+++ b/rust/ffi/node/src/query.rs
@@ -7,6 +7,7 @@ use lance::index::vector::MetricType;
 use neon::context::FunctionContext;
 use neon::handle::Handle;
 use neon::prelude::*;
+use neon::types::buffer::TypedArray;

 use crate::arrow::record_batch_to_buffer;
 use crate::error::ResultExt;
@@ -47,6 +48,11 @@ impl JsQuery {
            .map(|s| s.value(&mut cx))
            .map(|s| MetricType::try_from(s.as_str()).unwrap());

+        let is_electron = cx
+            .argument::<JsBoolean>(1)
+            .or_throw(&mut cx)?
+            .value(&mut cx);
+
        let rt = runtime(&mut cx)?;

        let (deferred, promise) = cx.promise();
@@ -76,9 +82,26 @@ impl JsQuery {
            deferred.settle_with(&channel, move |mut cx| {
                let results = results.or_throw(&mut cx)?;
                let buffer = record_batch_to_buffer(results).or_throw(&mut cx)?;
-                Ok(JsBuffer::external(&mut cx, buffer))
+                Self::new_js_buffer(buffer, &mut cx, is_electron)
            });
        });
        Ok(promise)
    }
+
+    // Creates a new JsBuffer from a rust buffer with a special logic for electron
+    fn new_js_buffer<'a>(
+        buffer: Vec<u8>,
+        cx: &mut TaskContext<'a>,
+        is_electron: bool,
+    ) -> NeonResult<Handle<'a, JsBuffer>> {
+        if is_electron {
+            // Electron does not support `external`: https://github.com/neon-bindings/neon/pull/937
+            let mut js_buffer = JsBuffer::new(cx, buffer.len()).or_throw(cx)?;
+            let buffer_data = js_buffer.as_mut_slice(cx);
+            buffer_data.copy_from_slice(buffer.as_slice());
+            Ok(js_buffer)
+        } else {
+            Ok(JsBuffer::external(cx, buffer))
+        }
+    }
 }
--- a/rust/ffi/node/src/table.rs
+++ b/rust/ffi/node/src/table.rs
@@ -22,7 +22,7 @@ use neon::types::buffer::TypedArray;
 use vectordb::Table;

 use crate::error::ResultExt;
-use crate::{get_aws_creds, runtime, JsDatabase};
+use crate::{get_aws_creds, get_aws_region, runtime, JsDatabase};

 pub(crate) struct JsTable {
    pub table: Table,
@@ -43,8 +43,7 @@ impl JsTable {
            .downcast_or_throw::<JsBox<JsDatabase>, _>(&mut cx)?;
        let table_name = cx.argument::<JsString>(0)?.value(&mut cx);
        let buffer = cx.argument::<JsBuffer>(1)?;
-        let batches = arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;
-        let schema = batches[0].schema();
+        let (batches, schema) = arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;

        // Write mode
        let mode = match cx.argument::<JsString>(2)?.value(&mut cx).as_str() {
@@ -62,14 +61,13 @@ impl JsTable {
        let (deferred, promise) = cx.promise();
        let database = db.database.clone();

-        let aws_creds = match get_aws_creds(&mut cx, 3) {
-            Ok(creds) => creds,
-            Err(err) => return err,
-        };
+        let aws_creds = get_aws_creds(&mut cx, 3)?;
+        let aws_region = get_aws_region(&mut cx, 6)?;

        let params = WriteParams {
            store_params: Some(ObjectStoreParams {
                aws_credentials: aws_creds,
+                aws_region,
                ..ObjectStoreParams::default()
            }),
            mode: mode,
@@ -94,10 +92,7 @@ impl JsTable {
        let js_table = cx.this().downcast_or_throw::<JsBox<JsTable>, _>(&mut cx)?;
        let buffer = cx.argument::<JsBuffer>(0)?;
        let write_mode = cx.argument::<JsString>(1)?.value(&mut cx);
-
-        let batches = arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;
-        let schema = batches[0].schema();
-
+        let (batches, schema) = arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;
        let rt = runtime(&mut cx)?;
        let channel = cx.channel();
        let mut table = js_table.table.clone();
@@ -109,14 +104,13 @@ impl JsTable {
            "overwrite" => WriteMode::Overwrite,
            s => return cx.throw_error(format!("invalid write mode {}", s)),
        };
-        let aws_creds = match get_aws_creds(&mut cx, 2) {
-            Ok(creds) => creds,
-            Err(err) => return err,
-        };
+        let aws_creds = get_aws_creds(&mut cx, 2)?;
+        let aws_region = get_aws_region(&mut cx, 5)?;

        let params = WriteParams {
            store_params: Some(ObjectStoreParams {
                aws_credentials: aws_creds,
+                aws_region,
                ..ObjectStoreParams::default()
            }),
            mode: write_mode,
--- a/rust/vectordb/Cargo.toml
+++ b/rust/vectordb/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "vectordb"
-version = "0.1.19"
+version = "0.2.3"
 edition = "2021"
 description = "Serverless, low-latency vector database for AI applications"
 license = "Apache-2.0"
Author	SHA1	Message	Date
Lance Release	909b7e90cd	[python] Bump version: 0.2.0 → 0.2.1	2023-08-24 04:00:11 +00:00
QianZhu	ae8486cc8f	bump lance version to 0.6.5 for lancedb release (#453 )	2023-08-23 20:59:03 -07:00
Tevin Wang	b8f32d082f	Clean up docs testing - exclude by glob instead of by file (#450 )	2023-08-24 07:30:37 +05:30
Jai	ea7522baa5	fix url to image in docs (#444 )	2023-08-22 16:21:02 -07:00
Lance Release	8764741116	Updating package-lock.json	2023-08-22 21:11:28 +00:00
Ayush Chaurasia	cc916389a6	[DOCS] Major Docs Revamp (#435 )	2023-08-22 14:06:26 -07:00
Lance Release	3d7d903d88	Updating package-lock.json	2023-08-22 20:15:13 +00:00
Lance Release	cc5e2d3e10	Bump version: 0.2.2 → 0.2.3	2023-08-22 20:14:58 +00:00
Rob Meng	30f5bc5865	expose awsRegion to be configurable (#441 )	2023-08-22 16:00:14 -04:00
gsilvestrin	2737315cb2	feat(node): Create empty tables / Arrow Tables (#399 ) - Supports creating an empty table as long as an Arrow Schema is provided - Supports creating a table from an Arrow Table (can be passed as data) - Simplified some Arrow code in the TS/FFI side - removed createTableArrow method, it was never documented / tested.	2023-08-22 10:57:45 -07:00
Rob Meng	d52422603c	use a lambda function to hide the value of credentials when printing a connection/table (#438 ) Previously when logging the `LocalConnection` and `LocalTable` classes, we would expose the aws creds inside them. This PR changes the stored creds to a anonymous function to hide the creds	2023-08-21 23:06:44 -04:00
Ayush Chaurasia	f35f8e451f	[DOCS] Update integrations + small typos (#432 ) Depends on - https://github.com/lancedb/lancedb/pull/430 --------- Co-authored-by: Kevin Tse <NivekT@users.noreply.github.com>	2023-08-18 09:59:22 +05:30
Ayush Chaurasia	0b9924b432	Make creating (and adding to) tables via Iterators more flexible & intuitive (#430 ) It improves the UX as iterators can be of any type supported by the table (plus recordbatch) & there is no separate requirement. Also expands the test cases for pydantic & arrow schema. If this is looks good I'll update the docs. Example usage: ``` class Content(LanceModel): vector: vector(2) item: str price: float def make_batches(): for _ in range(5): yield from [ # pandas pd.DataFrame({ "vector": [[3.1, 4.1], [1, 1]], "item": ["foo", "bar"], "price": [10.0, 20.0], }), # pylist [ {"vector": [3.1, 4.1], "item": "foo", "price": 10.0}, {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}, ], # recordbatch pa.RecordBatch.from_arrays( [ pa.array([[3.1, 4.1], [5.9, 26.5]], pa.list_(pa.float32(), 2)), pa.array(["foo", "bar"]), pa.array([10.0, 20.0]), ], ["vector", "item", "price"], ), # pydantic list [ Content(vector=[3.1, 4.1], item="foo", price=10.0), Content(vector=[5.9, 26.5], item="bar", price=20.0), ]] db = lancedb.connect("db") tbl = db.create_table("tabley", make_batches(), schema=Content, mode="overwrite") tbl.add(make_batches()) ``` Same should with arrow schema. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>	2023-08-18 09:56:30 +05:30
Lance Release	ba416a571d	Updating package-lock.json	2023-08-17 23:48:01 +00:00
Lance Release	13317ffb46	Updating package-lock.json	2023-08-17 23:07:51 +00:00
Lance Release	ca961567fe	Bump version: 0.2.1 → 0.2.2	2023-08-17 23:07:36 +00:00
gsilvestrin	31a12a141d	fix(node) Electron crashes when creating external buffer (#424 )	2023-08-17 14:47:54 -07:00
Chang She	e3061d4cb4	[python] Temporary restore feature (#428 ) This adds LanceTable.restore as a temporary feature. It reads data from a previous version and creates a new snapshot version using that data. This makes the version writeable unlike checkout. This should be replaced once the feature is implemented in pylance. Co-authored-by: Chang She <chang@lancedb.com>	2023-08-14 20:10:29 -07:00
Lance Release	1fcc67fd2c	Updating package-lock.json	2023-08-14 23:02:39 +00:00
Rob Meng	ac18812af0	fix moka version (#427 )	2023-08-14 18:28:55 -04:00
Lance Release	8324e0f171	Bump version: 0.2.0 → 0.2.1	2023-08-14 22:22:24 +00:00
Rob Meng	f0bcb26f32	Upgrade lance and pass AWS creds when opening a table (#426 )	2023-08-14 18:22:02 -04:00
Lance Release	b281c5255c	Updating package-lock.json	2023-08-14 17:03:51 +00:00
Lance Release	d349d2a44a	Updating package-lock.json	2023-08-14 16:06:52 +00:00
Lance Release	0699a6fa7b	Bump version: 0.1.19 → 0.2.0	2023-08-14 16:06:36 +00:00