tests pass; still need catalog

almost
initial changes to enable an in-memory dataset
2025-12-23 05:19:58 +00:00 · 2023-12-21 20:13:08 -08:00 · 2023-12-21 19:45:10 -08:00 · 2023-12-21 08:52:11 -08:00 · 2023-12-21 11:37:19 -05:00 · 2023-12-20 20:05:22 -08:00
53 changed files with 2054 additions and 363 deletions
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.9
+current_version = 0.4.0
 commit = True
 message = Bump version: {current_version} → {new_version}
 tag = True
--- a/.github/ISSUE_TEMPLATE/bug-node.yml
+++ b/.github/ISSUE_TEMPLATE/bug-node.yml
@@ -0,0 +1,33 @@
+name: Bug Report - Node / Typescript
+description: File a bug report
+title: "bug(node): "
+labels: [bug, typescript]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: input
+    id: version
+    attributes:
+      label: LanceDB version
+      description: What version of LanceDB are you using? `npm list | grep vectordb`.
+      placeholder: v0.3.2
+    validations:
+      required: false
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened?
+      description: Also tell us, what did you expect to happen?
+    validations:
+      required: true
+  - type: textarea
+    id: reproduction
+    attributes:
+      label: Are there known steps to reproduce?
+      description: |
+        Let us know how to reproduce the bug and we may be able to fix it more
+        quickly. This is not required, but it is helpful.
+    validations:
+      required: false
--- a/.github/ISSUE_TEMPLATE/bug-python.yml
+++ b/.github/ISSUE_TEMPLATE/bug-python.yml
@@ -0,0 +1,33 @@
+name: Bug Report - Python
+description: File a bug report
+title: "bug(python): "
+labels: [bug, python]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: input
+    id: version
+    attributes:
+      label: LanceDB version
+      description: What version of LanceDB are you using? `python -c "import lancedb; print(lancedb.__version__)"`.
+      placeholder: v0.3.2
+    validations:
+      required: false
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened?
+      description: Also tell us, what did you expect to happen?
+    validations:
+      required: true
+  - type: textarea
+    id: reproduction
+    attributes:
+      label: Are there known steps to reproduce?
+      description: |
+        Let us know how to reproduce the bug and we may be able to fix it more
+        quickly. This is not required, but it is helpful.
+    validations:
+      required: false
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Discord Community Support
+    url: https://discord.com/invite/zMM32dvNtd
+    about: Please ask and answer questions here.
--- a/.github/ISSUE_TEMPLATE/documentation.yml
+++ b/.github/ISSUE_TEMPLATE/documentation.yml
@@ -0,0 +1,23 @@
+name: 'Documentation improvement'
+description: Report an issue with the documentation.
+labels: [documentation]
+
+body:
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: >
+        Describe the issue with the documentation and how it can be fixed or improved.
+    validations:
+      required: true
+
+  - type: input
+    id: link
+    attributes:
+      label: Link
+      description: >
+        Provide a link to the existing documentation, if applicable.
+      placeholder: ex. https://lancedb.github.io/lancedb/guides/tables/...
+    validations:
+      required: false
--- a/.github/ISSUE_TEMPLATE/feature.yml
+++ b/.github/ISSUE_TEMPLATE/feature.yml
@@ -0,0 +1,31 @@
+name: Feature suggestion
+description: Suggestion a new feature for LanceDB
+title: "Feature: "
+labels: [enhancement]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Share a new idea for a feature or improvement. Be sure to search existing
+        issues first to avoid duplicates.
+  - type: dropdown
+    id: sdk
+    attributes:
+      label: SDK
+      description: Which SDK are you using? This helps us prioritize.
+      options:
+        - Python
+        - Node
+        - Rust
+      default: 0
+    validations:
+      required: false
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Describe the feature and why it would be useful. If applicable, consider
+        providing a code example of what it might be like to use the feature.
+    validations:
+      required: true
--- a/.github/workflows/npm-publish.yml
+++ b/.github/workflows/npm-publish.yml
@@ -38,13 +38,17 @@ jobs:
            node/vectordb-*.tgz

  node-macos:
-    runs-on: macos-13
+    strategy:
+      matrix:
+        config:
+          - arch: x86_64-apple-darwin
+            runner: macos-13
+          - arch: aarch64-apple-darwin
+            # xlarge is implicitly arm64.
+            runner: macos-13-xlarge
+    runs-on: ${{ matrix.config.runner }}
    # Only runs on tags that matches the make-release action
    if: startsWith(github.ref, 'refs/tags/v')
-    strategy:
-      fail-fast: false
-      matrix:
-        target: [x86_64-apple-darwin, aarch64-apple-darwin]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
@@ -54,17 +58,15 @@ jobs:
        run: |
          cd node
          npm ci
-      - name: Install rustup target
-        if: ${{ matrix.target == 'aarch64-apple-darwin' }}
-        run: rustup target add aarch64-apple-darwin
      - name: Build MacOS native node modules
-        run: bash ci/build_macos_artifacts.sh ${{ matrix.target }}
+        run: bash ci/build_macos_artifacts.sh ${{ matrix.config.arch }}
      - name: Upload Darwin Artifacts
        uses: actions/upload-artifact@v3
        with:
          name: native-darwin
          path: |
            node/dist/lancedb-vectordb-darwin*.tgz
+         

  node-linux:
    name: node-linux (${{ matrix.config.arch}}-unknown-linux-gnu
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -44,12 +44,19 @@ jobs:
      run: pytest -m "not slow" -x -v --durations=30 tests
    - name: doctest
      run: pytest --doctest-modules lancedb
-  mac:
+  platform:
+    name: "Platform: ${{ matrix.config.name }}"
    timeout-minutes: 30
    strategy:
      matrix:
-        mac-runner: [ "macos-13", "macos-13-xlarge" ]
-    runs-on: "${{ matrix.mac-runner }}"
+        config: 
+          - name: x86 Mac
+            runner: macos-13
+          - name: Arm Mac
+            runner: macos-13-xlarge
+          - name: x86 Windows
+            runner: windows-latest
+    runs-on: "${{ matrix.config.runner }}"
    defaults:
      run:
        shell: bash
@@ -91,11 +98,7 @@ jobs:
        pip install "pydantic<2"
        pip install -e .[tests]
        pip install tantivy@git+https://github.com/quickwit-oss/tantivy-py#164adc87e1a033117001cf70e38c82a53014d985
-        pip install pytest pytest-mock black isort
-    - name: Black
-      run: black --check --diff --no-color --quiet .
-    - name: isort
-      run: isort --check --diff --quiet .
+        pip install pytest pytest-mock
    - name: Run tests
      run: pytest -m "not slow" -x -v --durations=30 tests
    - name: doctest
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -24,6 +24,29 @@ env:
  RUST_BACKTRACE: "1"

 jobs:
+  lint:
+    timeout-minutes: 30
+    runs-on: ubuntu-22.04
+    defaults:
+      run:
+        shell: bash
+        working-directory: rust
+    steps:
+    - uses: actions/checkout@v3
+      with:
+          fetch-depth: 0
+          lfs: true
+    - uses: Swatinem/rust-cache@v2
+      with:
+        workspaces: rust
+    - name: Install dependencies
+      run: |
+          sudo apt update
+          sudo apt install -y protobuf-compiler libssl-dev
+    - name: Run format
+      run: cargo fmt --all -- --check
+    - name: Run clippy
+      run: cargo clippy --all --all-features -- -D warnings
  linux:
    timeout-minutes: 30
    runs-on: ubuntu-22.04
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -5,24 +5,24 @@ exclude = ["python"]
 resolver = "2"

 [workspace.dependencies]
-lance = { "version" = "=0.8.17", "features" = ["dynamodb"] }
-lance-index = { "version" = "=0.8.17" }
-lance-linalg = { "version" = "=0.8.17" }
-lance-testing = { "version" = "=0.8.17" }
+lance = { "version" = "=0.9.1", "features" = ["dynamodb"] }
+lance-index = { "version" = "=0.9.1" }
+lance-linalg = { "version" = "=0.9.1" }
+lance-testing = { "version" = "=0.9.1" }
 # Note that this one does not include pyarrow
-arrow = { version = "47.0.0", optional = false }
-arrow-array = "47.0"
-arrow-data = "47.0"
-arrow-ipc = "47.0"
-arrow-ord = "47.0"
-arrow-schema = "47.0"
-arrow-arith = "47.0"
-arrow-cast = "47.0"
+arrow = { version = "49.0.0", optional = false }
+arrow-array = "49.0"
+arrow-data = "49.0"
+arrow-ipc = "49.0"
+arrow-ord = "49.0"
+arrow-schema = "49.0"
+arrow-arith = "49.0"
+arrow-cast = "49.0"
 chrono = "0.4.23"
 half = { "version" = "=2.3.1", default-features = false, features = [
    "num-traits",
 ] }
 log = "0.4"
-object_store = "0.7.1"
+object_store = "0.8.0"
 snafu = "0.7.4"
 url = "2"
--- a/README.md
+++ b/README.md
@@ -5,10 +5,11 @@

 **Developer-friendly, serverless vector database for AI applications**

-<a href="https://lancedb.github.io/lancedb/">Documentation</a> •
-<a href="https://blog.lancedb.com/">Blog</a> •
-<a href="https://discord.gg/zMM32dvNtd">Discord</a> •
-<a href="https://twitter.com/lancedb">Twitter</a>
+<a href='https://github.com/lancedb/vectordb-recipes/tree/main' target="_blank"><img alt='LanceDB' src='https://img.shields.io/badge/VectorDB_Recipes-100000?style=for-the-badge&logo=LanceDB&logoColor=white&labelColor=645cfb&color=645cfb'/></a>
+<a href='https://lancedb.github.io/lancedb/' target="_blank"><img alt='lancdb' src='https://img.shields.io/badge/DOCS-100000?style=for-the-badge&logo=lancdb&logoColor=white&labelColor=645cfb&color=645cfb'/></a>
+[![Medium](https://img.shields.io/badge/Medium-12100E?style=for-the-badge&logo=medium&logoColor=white)](https://blog.lancedb.com/) 
+[![Discord](https://img.shields.io/badge/Discord-%235865F2.svg?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/zMM32dvNtd) 
+[![Twitter](https://img.shields.io/badge/Twitter-%231DA1F2.svg?style=for-the-badge&logo=Twitter&logoColor=white)](https://twitter.com/lancedb)

 </p>

--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -80,7 +80,6 @@ nav:
    - Ingest Embedding Functions: embeddings/embedding_functions.md
    - Available Functions: embeddings/default_embedding_functions.md
    - Create Custom Embedding Functions: embeddings/api.md
-    - Example - Calculate CLIP Embeddings with Roboflow Inference: examples/image_embeddings_roboflow.md
    - Example - Multi-lingual semantic search: notebooks/multi_lingual_example.ipynb
    - Example - MultiModal CLIP Embeddings: notebooks/DisappearingEmbeddingFunction.ipynb
  - 🔍 Python full-text search: fts.md
@@ -99,6 +98,7 @@ nav:
    - YouTube Transcript Search: notebooks/youtube_transcript_search.ipynb
    - Documentation QA Bot using LangChain: notebooks/code_qa_bot.ipynb
    - Multimodal search using CLIP: notebooks/multimodal_search.ipynb
+    - Example - Calculate CLIP Embeddings with Roboflow Inference: examples/image_embeddings_roboflow.md
    - Serverless QA Bot with S3 and Lambda: examples/serverless_lancedb_with_s3_and_lambda.md
    - Serverless QA Bot with Modal: examples/serverless_qa_bot_with_modal_and_langchain.md
  - 🌐 Javascript examples:
@@ -146,7 +146,8 @@ nav:
  - Serverless Chatbot from any website: examples/serverless_website_chatbot.md
  - TransformersJS Embedding Search: examples/transformerjs_embedding_search_nodejs.md
 - API references:
-  - Python API: python/python.md
+  - OSS Python API: python/python.md
+  - SaaS Python API: python/saas-python.md
  - Javascript API: javascript/modules.md
 - LanceDB Cloud↗: https://noteforms.com/forms/lancedb-mailing-list-cloud-kty1o5?notionforms=1&utm_source=notionforms

--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -2,3 +2,4 @@ mkdocs==1.4.2
 mkdocs-jupyter==0.24.1
 mkdocs-material==9.1.3
 mkdocstrings[python]==0.20.0
+pydantic
--- a/docs/src/basic.md
+++ b/docs/src/basic.md
@@ -64,18 +64,26 @@ We'll cover the basics of using LanceDB on your local machine in this section.
      tbl = db.create_table("table_from_df", data=df)
      ```

+      !!! warning
+
+            If the table already exists, LanceDB will raise an error by default.
+            If you want to overwrite the table, you can pass in `mode="overwrite"`
+            to the `createTable` function.
+
 === "Javascript"
      ```javascript
-      const tb = await db.createTable("my_table",
-                        data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
-                              {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
+      const tb = await db.createTable(
+        "myTable",
+        [{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
+         {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
      ```
-      
-!!! warning

-      If the table already exists, LanceDB will raise an error by default.
-      If you want to overwrite the table, you can pass in `mode="overwrite"`
-      to the `createTable` function.
+      !!! warning
+
+            If the table already exists, LanceDB will raise an error by default.
+            If you want to overwrite the table, you can pass in `"overwrite"`
+            to the `createTable` function like this: `await con.createTable(tableName, data, { writeMode: WriteMode.Overwrite })`
+      

 ??? info "Under the hood, LanceDB is converting the input data into an Apache Arrow table and persisting it to disk in [Lance format](https://www.github.com/lancedb/lance)."

@@ -108,7 +116,7 @@ Once created, you can open a table using the following code:

 === "Javascript"
      ```javascript
-      const tbl = await db.openTable("my_table");
+      const tbl = await db.openTable("myTable");
      ```

      If you forget the name of your table, you can always get a listing of all table names:
@@ -194,10 +202,17 @@ Use the `drop_table()` method on the database to remove a table.
      db.drop_table("my_table")
      ```

-This permanently removes the table and is not recoverable, unlike deleting rows.
-By default, if the table does not exist an exception is raised. To suppress this,
-you can pass in `ignore_missing=True`.
+      This permanently removes the table and is not recoverable, unlike deleting rows.
+      By default, if the table does not exist an exception is raised. To suppress this,
+      you can pass in `ignore_missing=True`.

+=== "JavaScript"
+      ```javascript
+      await db.dropTable('myTable')
+      ```
+
+      This permanently removes the table and is not recoverable, unlike deleting rows.
+      If the table does not exist an exception is raised. 

 ## What's next

--- a/docs/src/guides/tables.md
+++ b/docs/src/guides/tables.md
@@ -201,8 +201,8 @@ A Table is a collection of Records in a LanceDB Database. You can follow along o
    ```javascript
    data
    const tb = await db.createTable("my_table",
-                  data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
-                        {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
+      [{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
+       {"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
    ```

    !!! info "Note"
--- a/docs/src/python/saas-python.md
+++ b/docs/src/python/saas-python.md
@@ -0,0 +1,18 @@
+# LanceDB Python API Reference
+
+## Installation
+
+```shell
+pip install lancedb
+```
+
+## Connection
+
+::: lancedb.connect
+
+::: lancedb.remote.db.RemoteDBConnection
+
+## Table
+
+::: lancedb.remote.table.RemoteTable
+
--- a/docs/src/search.md
+++ b/docs/src/search.md
@@ -118,4 +118,101 @@ However, fast vector search using indices often entails making a trade-off with
 This is why it is often called **Approximate Nearest Neighbors (ANN)** search, while the Flat Search (KNN)
 always returns 100% recall.

-See [ANN Index](ann_indexes.md) for more details.
+See [ANN Index](ann_indexes.md) for more details.
+
+
+### Output formats
+
+LanceDB returns results in many different formats commonly used in python.
+Let's create a LanceDB table with a nested schema:
+
+```python
+from datetime import datetime
+import lancedb
+from lancedb.pydantic import LanceModel, Vector
+import numpy as np
+from pydantic import BaseModel
+uri = "data/sample-lancedb-nested"
+
+class Metadata(BaseModel):
+    source: str
+    timestamp: datetime
+
+class Document(BaseModel):
+    content: str
+    meta: Metadata
+
+class LanceSchema(LanceModel):
+    id: str
+    vector: Vector(1536)
+    payload: Document
+
+# Let's add 100 sample rows to our dataset
+data = [LanceSchema(
+    id=f"id{i}",
+    vector=np.random.randn(1536),
+    payload=Document(
+        content=f"document{i}", meta=Metadata(source=f"source{i%10}", timestamp=datetime.now())
+    ),
+) for i in range(100)]
+
+tbl = db.create_table("documents", data=data)
+```
+
+#### As a pyarrow table
+
+Using `to_arrow()` we can get the results back as a pyarrow Table.
+This result table has the same columns as the LanceDB table, with 
+the addition of an `_distance` column for vector search or a `score`
+column for full text search.
+
+```python
+tbl.search(np.random.randn(1536)).to_arrow()
+```
+
+#### As a pandas dataframe
+
+You can also get the results as a pandas dataframe.
+
+```python
+tbl.search(np.random.randn(1536)).to_pandas()
+```
+
+While other formats like Arrow/Pydantic/Python dicts have a natural 
+way to handle nested schemas, pandas can only store nested data as a 
+python dict column, which makes it difficult to support nested references.
+So for convenience, you can also tell LanceDB to flatten a nested schema 
+when creating the pandas dataframe. 
+
+```python
+tbl.search(np.random.randn(1536)).to_pandas(flatten=True)
+```
+
+If your table has a deeply nested struct, you can control how many levels
+of nesting to flatten by passing in a positive integer.
+
+```python
+tbl.search(np.random.randn(1536)).to_pandas(flatten=1)
+```
+
+
+#### As a list of python dicts
+
+You can of course return results as a list of python dicts.
+
+```python
+tbl.search(np.random.randn(1536)).to_list()
+```
+
+#### As a list of pydantic models
+
+We can add data using pydantic models, and we can certainly
+retrieve results as pydantic models
+
+```python
+tbl.search(np.random.randn(1536)).to_pydantic(LanceSchema)
+```
+
+Note that in this case the extra `_distance` field is discarded since
+it's not part of the LanceSchema.
+
--- a/docs/src/sql.md
+++ b/docs/src/sql.md
@@ -22,7 +22,7 @@ import numpy as np
 uri = "data/sample-lancedb"
 db = lancedb.connect(uri)

-data = [{"vector": row, "item": f"item {i}"}
+data = [{"vector": row, "item": f"item {i}", "id": i}
     for i, row in enumerate(np.random.random((10_000, 2)).astype('int'))]

 tbl = db.create_table("my_vectors", data=data)
@@ -35,33 +35,25 @@ const db = await vectordb.connect('data/sample-lancedb')

 let data = []
 for (let i = 0; i < 10_000; i++) {
-     data.push({vector: Array(1536).fill(i), id: `${i}`, content: "", longId: `${i}`},)
+     data.push({vector: Array(1536).fill(i), id: i, item: `item ${i}`, strId: `${i}`})
 }
-const tbl = await db.createTable('my_vectors', data)
+const tbl = await db.createTable('myVectors', data)
 ```
 -->
 === "Python"

    ```python
    tbl.search([100, 102]) \
-       .where("""(
-        (label IN [10, 20])
-        AND
-        (note.email IS NOT NULL)
-        ) OR NOT note.created
-        """)
-
+       .where("(item IN ('item 0', 'item 2')) AND (id > 10)") \
+       .to_arrow()
    ```
+
 === "Javascript"

    ```javascript
-    tbl.search([100, 102])
-       .where(`(
-            (label IN [10, 20])
-            AND
-            (note.email IS NOT NULL)
-        ) OR NOT note.created
-       `)
+    await tbl.search(Array(1536).fill(0))
+       .where("(item IN ('item 0', 'item 2')) AND (id > 10)")
+       .execute()
    ```


@@ -118,3 +110,22 @@ The mapping from SQL types to Arrow types is:

 [^1]: See precision mapping in previous table.

+
+## Filtering without Vector Search
+
+You can also filter your data without search.
+
+=== "Python"
+      ```python
+      tbl.search().where("id=10").limit(10).to_arrow()
+      ```
+
+=== "JavaScript"
+      ```javascript
+      await tbl.where('id=10').limit(10).execute()
+      ```
+
+!!! warning
+    If your table is large, this could potentially return a very large
+    amount of data. Please be sure to use a `limit` clause unless
+    you're sure you want to return the whole result set.
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.3.9",
+  "version": "0.4.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.3.9",
+      "version": "0.4.0",
      "cpu": [
        "x64",
        "arm64"
@@ -53,11 +53,11 @@
        "uuid": "^9.0.0"
      },
      "optionalDependencies": {
-        "@lancedb/vectordb-darwin-arm64": "0.3.9",
-        "@lancedb/vectordb-darwin-x64": "0.3.9",
-        "@lancedb/vectordb-linux-arm64-gnu": "0.3.9",
-        "@lancedb/vectordb-linux-x64-gnu": "0.3.9",
-        "@lancedb/vectordb-win32-x64-msvc": "0.3.9"
+        "@lancedb/vectordb-darwin-arm64": "0.4.0",
+        "@lancedb/vectordb-darwin-x64": "0.4.0",
+        "@lancedb/vectordb-linux-arm64-gnu": "0.4.0",
+        "@lancedb/vectordb-linux-x64-gnu": "0.4.0",
+        "@lancedb/vectordb-win32-x64-msvc": "0.4.0"
      }
    },
    "node_modules/@apache-arrow/ts": {
@@ -316,10 +316,22 @@
        "@jridgewell/sourcemap-codec": "^1.4.10"
      }
    },
+    "node_modules/@lancedb/vectordb-darwin-arm64": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.4.0.tgz",
+      "integrity": "sha512-cP6zGtBWXEcJHCI4uLNIP5ILtRvexvwmL8Uri1dnHG8dT8g12Ykug3BHO6Wt6wp/xASd2jJRIF/VAJsN9IeP1A==",
+      "cpu": [
+        "arm64"
+      ],
+      "optional": true,
+      "os": [
+        "darwin"
+      ]
+    },
    "node_modules/@lancedb/vectordb-darwin-x64": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.3.9.tgz",
-      "integrity": "sha512-4xXQoPheyIl1P5kRoKmZtaAHFrYdL9pw5yq+r6ewIx0TCemN4LSvzSUTqM5nZl3QPU8FeL0CGD8Gt2gMU0HQ2A==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.4.0.tgz",
+      "integrity": "sha512-ig0gV5ol1sFe2lb1HOatK0rizyj9I91WbnH79i7OdUl3nAQIcWm70CnxrPLtx0DS2NTGh2kFJbYCWcaUlu6YfA==",
      "cpu": [
        "x64"
      ],
@@ -329,9 +341,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-linux-arm64-gnu": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.3.9.tgz",
-      "integrity": "sha512-WIxCZKnLeSlz0PGURtKSX6hJ4CYE2o5P+IFmmuWOWB1uNapQu6zOpea6rNxcRFHUA0IJdO02lVxVfn2hDX4SMg==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.4.0.tgz",
+      "integrity": "sha512-gMXIDT2kriAPDwWIRKXdaTCNdOeFGEok1S9Y30AOruHXddW1vCIo4JNJIYbBqHnwAeI4wI3ae6GRCFaf1UxO3g==",
      "cpu": [
        "arm64"
      ],
@@ -341,9 +353,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-linux-x64-gnu": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.3.9.tgz",
-      "integrity": "sha512-bQbcV9adKzYbJLNzDjk9OYsMnT2IjmieLfb4IQ1hj5IUoWfbg80Bd0+gZUnrmrhG6fe56TIriFZYQR9i7TSE9Q==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.4.0.tgz",
+      "integrity": "sha512-ZQ3lDrDSz1IKdx/mS9Lz08agFO+OD5oSFrrcFNCoT1+H93eS1mCLdmCoEARu3jKbx0tMs38l5J9yXZ2QmJye3w==",
      "cpu": [
        "x64"
      ],
@@ -353,9 +365,9 @@
      ]
    },
    "node_modules/@lancedb/vectordb-win32-x64-msvc": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.3.9.tgz",
-      "integrity": "sha512-7EXI7P1QvAfgJNPWWBMDOkoJ696gSBAClcyEJNYg0JV21jVFZRwJVI3bZXflesWduFi/mTuzPkFFA68us1u19A==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.4.0.tgz",
+      "integrity": "sha512-toNcNwBRE1sdsSf5hr7W8QiqZ33csc/knVEek4CyvYkZHJGh4Z6WI+DJUIASo5wzUez4TX7qUPpRPL9HuaPMCg==",
      "cpu": [
        "x64"
      ],
@@ -4856,28 +4868,34 @@
        "@jridgewell/sourcemap-codec": "^1.4.10"
      }
    },
+    "@lancedb/vectordb-darwin-arm64": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.4.0.tgz",
+      "integrity": "sha512-cP6zGtBWXEcJHCI4uLNIP5ILtRvexvwmL8Uri1dnHG8dT8g12Ykug3BHO6Wt6wp/xASd2jJRIF/VAJsN9IeP1A==",
+      "optional": true
+    },
    "@lancedb/vectordb-darwin-x64": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.3.9.tgz",
-      "integrity": "sha512-4xXQoPheyIl1P5kRoKmZtaAHFrYdL9pw5yq+r6ewIx0TCemN4LSvzSUTqM5nZl3QPU8FeL0CGD8Gt2gMU0HQ2A==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.4.0.tgz",
+      "integrity": "sha512-ig0gV5ol1sFe2lb1HOatK0rizyj9I91WbnH79i7OdUl3nAQIcWm70CnxrPLtx0DS2NTGh2kFJbYCWcaUlu6YfA==",
      "optional": true
    },
    "@lancedb/vectordb-linux-arm64-gnu": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.3.9.tgz",
-      "integrity": "sha512-WIxCZKnLeSlz0PGURtKSX6hJ4CYE2o5P+IFmmuWOWB1uNapQu6zOpea6rNxcRFHUA0IJdO02lVxVfn2hDX4SMg==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.4.0.tgz",
+      "integrity": "sha512-gMXIDT2kriAPDwWIRKXdaTCNdOeFGEok1S9Y30AOruHXddW1vCIo4JNJIYbBqHnwAeI4wI3ae6GRCFaf1UxO3g==",
      "optional": true
    },
    "@lancedb/vectordb-linux-x64-gnu": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.3.9.tgz",
-      "integrity": "sha512-bQbcV9adKzYbJLNzDjk9OYsMnT2IjmieLfb4IQ1hj5IUoWfbg80Bd0+gZUnrmrhG6fe56TIriFZYQR9i7TSE9Q==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.4.0.tgz",
+      "integrity": "sha512-ZQ3lDrDSz1IKdx/mS9Lz08agFO+OD5oSFrrcFNCoT1+H93eS1mCLdmCoEARu3jKbx0tMs38l5J9yXZ2QmJye3w==",
      "optional": true
    },
    "@lancedb/vectordb-win32-x64-msvc": {
-      "version": "0.3.9",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.3.9.tgz",
-      "integrity": "sha512-7EXI7P1QvAfgJNPWWBMDOkoJ696gSBAClcyEJNYg0JV21jVFZRwJVI3bZXflesWduFi/mTuzPkFFA68us1u19A==",
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.4.0.tgz",
+      "integrity": "sha512-toNcNwBRE1sdsSf5hr7W8QiqZ33csc/knVEek4CyvYkZHJGh4Z6WI+DJUIASo5wzUez4TX7qUPpRPL9HuaPMCg==",
      "optional": true
    },
    "@neon-rs/cli": {
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.3.9",
+  "version": "0.4.0",
  "description": " Serverless, low-latency vector database for AI applications",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
@@ -81,10 +81,10 @@
    }
  },
  "optionalDependencies": {
-    "@lancedb/vectordb-darwin-arm64": "0.3.9",
-    "@lancedb/vectordb-darwin-x64": "0.3.9",
-    "@lancedb/vectordb-linux-arm64-gnu": "0.3.9",
-    "@lancedb/vectordb-linux-x64-gnu": "0.3.9",
-    "@lancedb/vectordb-win32-x64-msvc": "0.3.9"
+    "@lancedb/vectordb-darwin-arm64": "0.4.0",
+    "@lancedb/vectordb-darwin-x64": "0.4.0",
+    "@lancedb/vectordb-linux-arm64-gnu": "0.4.0",
+    "@lancedb/vectordb-linux-x64-gnu": "0.4.0",
+    "@lancedb/vectordb-win32-x64-msvc": "0.4.0"
  }
 }
--- a/node/src/index.ts
+++ b/node/src/index.ts
@@ -21,9 +21,10 @@ import type { EmbeddingFunction } from './embedding/embedding_function'
 import { RemoteConnection } from './remote'
 import { Query } from './query'
 import { isEmbeddingFunction } from './embedding/embedding_function'
+import { type Literal, toSQL } from './util'

 // eslint-disable-next-line @typescript-eslint/no-var-requires
-const { databaseNew, databaseTableNames, databaseOpenTable, databaseDropTable, tableCreate, tableAdd, tableCreateVectorIndex, tableCountRows, tableDelete, tableCleanupOldVersions, tableCompactFiles, tableListIndices, tableIndexStats } = require('../native.js')
+const { databaseNew, databaseTableNames, databaseOpenTable, databaseDropTable, tableCreate, tableAdd, tableCreateVectorIndex, tableCountRows, tableDelete, tableUpdate, tableCleanupOldVersions, tableCompactFiles, tableListIndices, tableIndexStats } = require('../native.js')

 export { Query }
 export type { EmbeddingFunction }
@@ -261,6 +262,39 @@ export interface Table<T = number[]> {
   */
  delete: (filter: string) => Promise<void>

+  /**
+   * Update rows in this table.
+   *
+   * This can be used to update a single row, many rows, all rows, or
+   * sometimes no rows (if your predicate matches nothing).
+   *
+   * @param args see {@link UpdateArgs} and {@link UpdateSqlArgs} for more details
+   *
+   * @examples
+   *
+   * ```ts
+   * const con = await lancedb.connect("./.lancedb")
+   * const data = [
+   *    {id: 1, vector: [3, 3], name: 'Ye'},
+   *    {id: 2, vector: [4, 4], name: 'Mike'},
+   * ];
+   * const tbl = await con.createTable("my_table", data)
+   *
+   * await tbl.update({
+   *   filter: "id = 2",
+   *   updates: { vector: [2, 2], name: "Michael" },
+   * })
+   *
+   * let results = await tbl.search([1, 1]).execute();
+   * // Returns [
+   * //   {id: 2, vector: [2, 2], name: 'Michael'}
+   * //   {id: 1, vector: [3, 3], name: 'Ye'}
+   * // ]
+   * ```
+   *
+   */
+  update: (args: UpdateArgs | UpdateSqlArgs) => Promise<void>
+
  /**
   * List the indicies on this table.
   */
@@ -272,6 +306,34 @@ export interface Table<T = number[]> {
  indexStats: (indexUuid: string) => Promise<IndexStats>
 }

+export interface UpdateArgs {
+  /**
+   * A filter in the same format used by a sql WHERE clause. The filter may be empty,
+   * in which case all rows will be updated.
+   */
+  where?: string
+
+  /**
+   * A key-value map of updates. The keys are the column names, and the values are the
+   * new values to set
+   */
+  values: Record<string, Literal>
+}
+
+export interface UpdateSqlArgs {
+  /**
+   * A filter in the same format used by a sql WHERE clause. The filter may be empty,
+   * in which case all rows will be updated.
+   */
+  where?: string
+
+  /**
+   * A key-value map of updates. The keys are the column names, and the values are the
+   * new values to set as SQL expressions.
+   */
+  valuesSql: Record<string, string>
+}
+
 export interface VectorIndex {
  columns: string[]
  name: string
@@ -426,6 +488,16 @@ export class LocalTable<T = number[]> implements Table<T> {
    return new Query(query, this._tbl, this._embeddings)
  }

+  /**
+   * Creates a filter query to find all rows matching the specified criteria
+   * @param value The filter criteria (like SQL where clause syntax)
+   */
+  filter (value: string): Query<T> {
+    return new Query(undefined, this._tbl, this._embeddings).filter(value)
+  }
+
+  where = this.filter
+
  /**
   * Insert records into this Table.
   *
@@ -481,6 +553,31 @@ export class LocalTable<T = number[]> implements Table<T> {
    return tableDelete.call(this._tbl, filter).then((newTable: any) => { this._tbl = newTable })
  }

+  /**
+   * Update rows in this table.
+   *
+   * @param args see {@link UpdateArgs} and {@link UpdateSqlArgs} for more details
+   *
+   * @returns
+   */
+  async update (args: UpdateArgs | UpdateSqlArgs): Promise<void> {
+    let filter: string | null
+    let updates: Record<string, string>
+
+    if ('valuesSql' in args) {
+      filter = args.where ?? null
+      updates = args.valuesSql
+    } else {
+      filter = args.where ?? null
+      updates = {}
+      for (const [key, value] of Object.entries(args.values)) {
+        updates[key] = toSQL(value)
+      }
+    }
+
+    return tableUpdate.call(this._tbl, filter, updates).then((newTable: any) => { this._tbl = newTable })
+  }
+
  /**
   * Clean up old versions of the table, freeing disk space.
   *
@@ -647,6 +744,11 @@ export interface IvfPQIndexConfig {
   */
  replace?: boolean

+  /**
+   * Cache size of the index
+   */
+  index_cache_size?: number
+
  type: 'ivf_pq'
 }

--- a/node/src/query.ts
+++ b/node/src/query.ts
@@ -23,10 +23,10 @@ const { tableSearch } = require('../native.js')
 * A builder for nearest neighbor queries for LanceDB.
 */
 export class Query<T = number[]> {
-  private readonly _query: T
+  private readonly _query?: T
  private readonly _tbl?: any
  private _queryVector?: number[]
-  private _limit: number
+  private _limit?: number
  private _refineFactor?: number
  private _nprobes: number
  private _select?: string[]
@@ -35,10 +35,10 @@ export class Query<T = number[]> {
  private _prefilter: boolean
  protected readonly _embeddings?: EmbeddingFunction<T>

-  constructor (query: T, tbl?: any, embeddings?: EmbeddingFunction<T>) {
+  constructor (query?: T, tbl?: any, embeddings?: EmbeddingFunction<T>) {
    this._tbl = tbl
    this._query = query
-    this._limit = 10
+    this._limit = undefined
    this._nprobes = 20
    this._refineFactor = undefined
    this._select = undefined
@@ -113,10 +113,12 @@ export class Query<T = number[]> {
     * Execute the query and return the results as an Array of Objects
     */
  async execute<T = Record<string, unknown>> (): Promise<T[]> {
-    if (this._embeddings !== undefined) {
-      this._queryVector = (await this._embeddings.embed([this._query]))[0]
-    } else {
-      this._queryVector = this._query as number[]
+    if (this._query !== undefined) {
+      if (this._embeddings !== undefined) {
+        this._queryVector = (await this._embeddings.embed([this._query]))[0]
+      } else {
+        this._queryVector = this._query as number[]
+      }
    }

    const isElectron = this.isElectron()
--- a/node/src/remote/index.ts
+++ b/node/src/remote/index.ts
@@ -16,7 +16,8 @@ import {
  type EmbeddingFunction, type Table, type VectorIndexParams, type Connection,
  type ConnectionOptions, type CreateTableOptions, type VectorIndex,
  type WriteOptions,
-  type IndexStats
+  type IndexStats,
+  type UpdateArgs, type UpdateSqlArgs
 } from '../index'
 import { Query } from '../query'

@@ -24,6 +25,7 @@ import { Vector, Table as ArrowTable } from 'apache-arrow'
 import { HttpLancedbClient } from './client'
 import { isEmbeddingFunction } from '../embedding/embedding_function'
 import { createEmptyTable, fromRecordsToStreamBuffer, fromTableToStreamBuffer } from '../arrow'
+import { toSQL } from '../util'

 /**
 * Remote connection.
@@ -55,8 +57,8 @@ export class RemoteConnection implements Connection {
    return 'db://' + this._client.uri
  }

-  async tableNames (): Promise<string[]> {
-    const response = await this._client.get('/v1/table/')
+  async tableNames (pageToken: string = '', limit: number = 10): Promise<string[]> {
+    const response = await this._client.get('/v1/table/', { limit, page_token: pageToken })
    return response.data.tables
  }

@@ -193,6 +195,17 @@ export class RemoteTable<T = number[]> implements Table<T> {
    return this._name
  }

+  get schema (): Promise<any> {
+    return this._client.post(`/v1/table/${this._name}/describe/`).then(res => {
+      if (res.status !== 200) {
+        throw new Error(`Server Error, status: ${res.status}, ` +
+        // eslint-disable-next-line @typescript-eslint/restrict-template-expressions
+          `message: ${res.statusText}: ${res.data}`)
+      }
+      return res.data?.schema
+    })
+  }
+
  search (query: T): Query<T> {
    return new RemoteQuery(query, this._client, this._name)//, this._embeddings_new)
  }
@@ -233,8 +246,41 @@ export class RemoteTable<T = number[]> implements Table<T> {
    return data.length
  }

-  async createIndex (indexParams: VectorIndexParams): Promise<any> {
-    throw new Error('Not implemented')
+  async createIndex (indexParams: VectorIndexParams): Promise<void> {
+    const unsupportedParams = [
+      'index_name',
+      'num_partitions',
+      'max_iters',
+      'use_opq',
+      'num_sub_vectors',
+      'num_bits',
+      'max_opq_iters',
+      'replace'
+    ]
+    for (const param of unsupportedParams) {
+      // eslint-disable-next-line @typescript-eslint/strict-boolean-expressions
+      if (indexParams[param as keyof VectorIndexParams]) {
+        throw new Error(`${param} is not supported for remote connections`)
+      }
+    }
+
+    const column = indexParams.column ?? 'vector'
+    const indexType = 'vector' // only vector index is supported for remote connections
+    const metricType = indexParams.metric_type ?? 'L2'
+    const indexCacheSize = indexParams ?? null
+
+    const data = {
+      column,
+      index_type: indexType,
+      metric_type: metricType,
+      index_cache_size: indexCacheSize
+    }
+    const res = await this._client.post(`/v1/table/${this._name}/create_index/`, data)
+    if (res.status !== 200) {
+      throw new Error(`Server Error, status: ${res.status}, ` +
+      // eslint-disable-next-line @typescript-eslint/restrict-template-expressions
+        `message: ${res.statusText}: ${res.data}`)
+    }
  }

  async countRows (): Promise<number> {
@@ -246,6 +292,26 @@ export class RemoteTable<T = number[]> implements Table<T> {
    await this._client.post(`/v1/table/${this._name}/delete/`, { predicate: filter })
  }

+  async update (args: UpdateArgs | UpdateSqlArgs): Promise<void> {
+    let filter: string | null
+    let updates: Record<string, string>
+
+    if ('valuesSql' in args) {
+      filter = args.where ?? null
+      updates = args.valuesSql
+    } else {
+      filter = args.where ?? null
+      updates = {}
+      for (const [key, value] of Object.entries(args.values)) {
+        updates[key] = toSQL(value)
+      }
+    }
+    await this._client.post(`/v1/table/${this._name}/update/`, {
+      predicate: filter,
+      updates: Object.entries(updates).map(([key, value]) => [key, value])
+    })
+  }
+
  async listIndices (): Promise<VectorIndex[]> {
    const results = await this._client.post(`/v1/table/${this._name}/index/list/`)
    return results.data.indexes?.map((index: any) => ({
--- a/node/src/test/test.ts
+++ b/node/src/test/test.ts
@@ -78,12 +78,31 @@ describe('LanceDB client', function () {
    })

    it('limits # of results', async function () {
-      const uri = await createTestDB()
+      const uri = await createTestDB(2, 100)
      const con = await lancedb.connect(uri)
      const table = await con.openTable('vectors')
-      const results = await table.search([0.1, 0.3]).limit(1).execute()
+      let results = await table.search([0.1, 0.3]).limit(1).execute()
      assert.equal(results.length, 1)
      assert.equal(results[0].id, 1)
+
+      // there is a default limit if unspecified
+      results = await table.search([0.1, 0.3]).execute()
+      assert.equal(results.length, 10)
+    })
+
+    it('uses a filter / where clause without vector search', async function () {
+      // eslint-disable-next-line @typescript-eslint/explicit-function-return-type
+      const assertResults = (results: Array<Record<string, unknown>>) => {
+        assert.equal(results.length, 50)
+      }
+
+      const uri = await createTestDB(2, 100)
+      const con = await lancedb.connect(uri)
+      const table = (await con.openTable('vectors')) as LocalTable
+      let results = await table.filter('id % 2 = 0').execute()
+      assertResults(results)
+      results = await table.where('id % 2 = 0').execute()
+      assertResults(results)
    })

    it('uses a filter / where clause', async function () {
@@ -260,6 +279,46 @@ describe('LanceDB client', function () {
      assert.equal(await table.countRows(), 2)
    })

+    it('can update records in the table', async function () {
+      const uri = await createTestDB()
+      const con = await lancedb.connect(uri)
+
+      const table = await con.openTable('vectors')
+      assert.equal(await table.countRows(), 2)
+
+      await table.update({ where: 'price = 10', valuesSql: { price: '100' } })
+      const results = await table.search([0.1, 0.2]).execute()
+      assert.equal(results[0].price, 100)
+      assert.equal(results[1].price, 11)
+    })
+
+    it('can update the records using a literal value', async function () {
+      const uri = await createTestDB()
+      const con = await lancedb.connect(uri)
+
+      const table = await con.openTable('vectors')
+      assert.equal(await table.countRows(), 2)
+
+      await table.update({ where: 'price = 10', values: { price: 100 } })
+      const results = await table.search([0.1, 0.2]).execute()
+      assert.equal(results[0].price, 100)
+      assert.equal(results[1].price, 11)
+    })
+
+    it('can update every record in the table', async function () {
+      const uri = await createTestDB()
+      const con = await lancedb.connect(uri)
+
+      const table = await con.openTable('vectors')
+      assert.equal(await table.countRows(), 2)
+
+      await table.update({ valuesSql: { price: '100' } })
+      const results = await table.search([0.1, 0.2]).execute()
+
+      assert.equal(results[0].price, 100)
+      assert.equal(results[1].price, 100)
+    })
+
    it('can delete records from a table', async function () {
      const uri = await createTestDB()
      const con = await lancedb.connect(uri)
@@ -542,7 +601,7 @@ describe('Compact and cleanup', function () {

    // should have no effect, but this validates the arguments are parsed.
    await table.compactFiles({
-      targetRowsPerFragment: 1024 * 10,
+      targetRowsPerFragment: 102410,
      maxRowsPerGroup: 1024,
      materializeDeletions: true,
      materializeDeletionsThreshold: 0.5,
--- a/node/src/test/util.ts
+++ b/node/src/test/util.ts
@@ -0,0 +1,45 @@
+// Copyright 2023 LanceDB Developers.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+import { toSQL } from '../util'
+import * as chai from 'chai'
+
+const expect = chai.expect
+
+describe('toSQL', function () {
+  it('should turn string to SQL expression', function () {
+    expect(toSQL('foo')).to.equal("'foo'")
+  })
+
+  it('should turn number to SQL expression', function () {
+    expect(toSQL(123)).to.equal('123')
+  })
+
+  it('should turn boolean to SQL expression', function () {
+    expect(toSQL(true)).to.equal('TRUE')
+  })
+
+  it('should turn null to SQL expression', function () {
+    expect(toSQL(null)).to.equal('NULL')
+  })
+
+  it('should turn Date to SQL expression', function () {
+    const date = new Date('05 October 2011 14:48 UTC')
+    expect(toSQL(date)).to.equal("'2011-10-05T14:48:00.000Z'")
+  })
+
+  it('should turn array to SQL expression', function () {
+    expect(toSQL(['foo', 'bar', true, 1])).to.equal("['foo', 'bar', TRUE, 1]")
+  })
+})
--- a/node/src/util.ts
+++ b/node/src/util.ts
@@ -0,0 +1,44 @@
+// Copyright 2023 LanceDB Developers.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+export type Literal = string | number | boolean | null | Date | Literal[]
+
+export function toSQL (value: Literal): string {
+  if (typeof value === 'string') {
+    return `'${value}'`
+  }
+
+  if (typeof value === 'number') {
+    return value.toString()
+  }
+
+  if (typeof value === 'boolean') {
+    return value ? 'TRUE' : 'FALSE'
+  }
+
+  if (value === null) {
+    return 'NULL'
+  }
+
+  if (value instanceof Date) {
+    return `'${value.toISOString()}'`
+  }
+
+  if (Array.isArray(value)) {
+    return `[${value.map(toSQL).join(', ')}]`
+  }
+
+  // eslint-disable-next-line @typescript-eslint/restrict-template-expressions
+  throw new Error(`Unsupported value type: ${typeof value} value: (${value})`)
+}
--- a/python/.bumpversion.cfg
+++ b/python/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.4
+current_version = 0.4.0
 commit = True
 message = [python] Bump version: {current_version} → {new_version}
 tag = True
--- a/python/lancedb/init.py
+++ b/python/lancedb/init.py
@@ -27,7 +27,7 @@ def connect(
    uri: URI,
    *,
    api_key: Optional[str] = None,
-    region: str = "us-west-2",
+    region: str = "us-east-1",
    host_override: Optional[str] = None,
 ) -> DBConnection:
    """Connect to a LanceDB database.
@@ -39,7 +39,7 @@ def connect(
    api_key: str, optional
        If presented, connect to LanceDB cloud.
        Otherwise, connect to a database on file system or cloud storage.
-    region: str, default "us-west-2"
+    region: str, default "us-east-1"
        The region to use for LanceDB Cloud.
    host_override: str, optional
        The override url for LanceDB Cloud.
--- a/python/lancedb/db.py
+++ b/python/lancedb/db.py
@@ -23,7 +23,7 @@ from overrides import EnforceOverrides, override
 from pyarrow import fs

 from .table import LanceTable, Table
-from .util import fs_from_uri, get_uri_location, get_uri_scheme
+from .util import fs_from_uri, get_uri_location, get_uri_scheme, join_uri

 if TYPE_CHECKING:
    from .common import DATA, URI
@@ -288,14 +288,13 @@ class LanceDBConnection(DBConnection):
            A list of table names.
        """
        try:
-            filesystem, path = fs_from_uri(self.uri)
+            filesystem = fs_from_uri(self.uri)[0]
        except pa.ArrowInvalid:
            raise NotImplementedError("Unsupported scheme: " + self.uri)

        try:
-            paths = filesystem.get_file_info(
-                fs.FileSelector(get_uri_location(self.uri))
-            )
+            loc = get_uri_location(self.uri)
+            paths = filesystem.get_file_info(fs.FileSelector(loc))
        except FileNotFoundError:
            # It is ok if the file does not exist since it will be created
            paths = []
@@ -373,7 +372,7 @@ class LanceDBConnection(DBConnection):
        """
        try:
            filesystem, path = fs_from_uri(self.uri)
-            table_path = os.path.join(path, name + ".lance")
+            table_path = join_uri(path, name + ".lance")
            filesystem.delete_dir(table_path)
        except FileNotFoundError:
            if not ignore_missing:
--- a/python/lancedb/fts.py
+++ b/python/lancedb/fts.py
@@ -75,8 +75,14 @@ def populate_index(index: tantivy.Index, table: LanceTable, fields: List[str]) -
        The number of rows indexed
    """
    # first check the fields exist and are string or large string type
+    nested = []
    for name in fields:
-        f = table.schema.field(name)  # raises KeyError if not found
+        try:
+            f = table.schema.field(name)  # raises KeyError if not found
+        except KeyError:
+            f = resolve_path(table.schema, name)
+            nested.append(name)
+
        if not pa.types.is_string(f.type) and not pa.types.is_large_string(f.type):
            raise TypeError(f"Field {name} is not a string type")

@@ -85,7 +91,16 @@ def populate_index(index: tantivy.Index, table: LanceTable, fields: List[str]) -
    # write data into index
    dataset = table.to_lance()
    row_id = 0
+
+    max_nested_level = 0
+    if len(nested) > 0:
+        max_nested_level = max([len(name.split(".")) for name in nested])
+
    for b in dataset.to_batches(columns=fields):
+        if max_nested_level > 0:
+            b = pa.Table.from_batches([b])
+            for _ in range(max_nested_level - 1):
+                b = b.flatten()
        for i in range(b.num_rows):
            doc = tantivy.Document()
            doc.add_integer("doc_id", row_id)
@@ -98,6 +113,30 @@ def populate_index(index: tantivy.Index, table: LanceTable, fields: List[str]) -
    return row_id


+def resolve_path(schema, field_name: str) -> pa.Field:
+    """
+    Resolve a nested field path to a list of field names
+
+    Parameters
+    ----------
+    field_name : str
+        The field name to resolve
+
+    Returns
+    -------
+    List[str]
+        The resolved path
+    """
+    path = field_name.split(".")
+    field = schema.field(path.pop(0))
+    for segment in path:
+        if pa.types.is_struct(field.type):
+            field = field.type.field(segment)
+        else:
+            raise KeyError(f"field {field_name} not found in schema {schema}")
+    return field
+
+
 def search_index(
    index: tantivy.Index, query: str, limit: int = 10
 ) -> Tuple[Tuple[int], Tuple[float]]:
--- a/python/lancedb/pydantic.py
+++ b/python/lancedb/pydantic.py
@@ -348,3 +348,20 @@ def get_extras(field_info: pydantic.fields.FieldInfo, key: str) -> Any:
    if PYDANTIC_VERSION.major >= 2:
        return (field_info.json_schema_extra or {}).get(key)
    return (field_info.field_info.extra or {}).get("json_schema_extra", {}).get(key)
+
+
+if PYDANTIC_VERSION.major < 2:
+
+    def model_to_dict(model: pydantic.BaseModel) -> Dict[str, Any]:
+        """
+        Convert a Pydantic model to a dictionary.
+        """
+        return model.dict()
+
+else:
+
+    def model_to_dict(model: pydantic.BaseModel) -> Dict[str, Any]:
+        """
+        Convert a Pydantic model to a dictionary.
+        """
+        return model.model_dump()
--- a/python/lancedb/query.py
+++ b/python/lancedb/query.py
@@ -185,14 +185,40 @@ class LanceQueryBuilder(ABC):
        """
        return self.to_pandas()

-    def to_pandas(self) -> "pd.DataFrame":
+    def to_pandas(self, flatten: Optional[Union[int, bool]] = None) -> "pd.DataFrame":
        """
        Execute the query and return the results as a pandas DataFrame.
        In addition to the selected columns, LanceDB also returns a vector
        and also the "_distance" column which is the distance between the query
        vector and the returned vector.
+
+        Parameters
+        ----------
+        flatten: Optional[Union[int, bool]]
+            If flatten is True, flatten all nested columns.
+            If flatten is an integer, flatten the nested columns up to the
+            specified depth.
+            If unspecified, do not flatten the nested columns.
        """
-        return self.to_arrow().to_pandas()
+        tbl = self.to_arrow()
+        if flatten is True:
+            while True:
+                tbl = tbl.flatten()
+                has_struct = False
+                # loop through all columns to check if there is any struct column
+                if any(pa.types.is_struct(col.type) for col in tbl.schema):
+                    continue
+                else:
+                    break
+        elif isinstance(flatten, int):
+            if flatten <= 0:
+                raise ValueError(
+                    "Please specify a positive integer for flatten or the boolean value `True`"
+                )
+            while flatten > 0:
+                tbl = tbl.flatten()
+                flatten -= 1
+        return tbl.to_pandas()

    @abstractmethod
    def to_arrow(self) -> pa.Table:
--- a/python/lancedb/remote/init.py
+++ b/python/lancedb/remote/init.py
@@ -18,6 +18,8 @@ import attrs
 import pyarrow as pa
 from pydantic import BaseModel

+from lancedb.common import VECTOR_COLUMN_NAME
+
 __all__ = ["LanceDBClient", "VectorQuery", "VectorQueryResult"]


@@ -43,6 +45,8 @@ class VectorQuery(BaseModel):

    refine_factor: Optional[int] = None

+    vector_column: str = VECTOR_COLUMN_NAME
+

@attrs.define
 class VectorQueryResult:
--- a/python/lancedb/remote/db.py
+++ b/python/lancedb/remote/db.py
@@ -56,16 +56,20 @@ class RemoteDBConnection(DBConnection):
            self._loop = asyncio.get_event_loop()

    def __repr__(self) -> str:
-        return f"RemoveConnect(name={self.db_name})"
+        return f"RemoteConnect(name={self.db_name})"

    @override
-    def table_names(self, page_token: Optional[str] = None, limit=10) -> Iterable[str]:
+    def table_names(
+        self, page_token: Optional[str] = None, limit: int = 10
+    ) -> Iterable[str]:
        """List the names of all tables in the database.

        Parameters
        ----------
        page_token: str
            The last token to start the new page.
+        limit: int, default 10
+            The maximum number of tables to return for each page.

        Returns
        -------
@@ -120,6 +124,97 @@ class RemoteDBConnection(DBConnection):
        fill_value: float = 0.0,
        embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
    ) -> Table:
+        """Create a [Table][lancedb.table.Table] in the database.
+
+        Parameters
+        ----------
+        name: str
+            The name of the table.
+        data: The data to initialize the table, *optional*
+            User must provide at least one of `data` or `schema`.
+            Acceptable types are:
+
+            - dict or list-of-dict
+
+            - pandas.DataFrame
+
+            - pyarrow.Table or pyarrow.RecordBatch
+        schema: The schema of the table, *optional*
+            Acceptable types are:
+
+            - pyarrow.Schema
+
+            - [LanceModel][lancedb.pydantic.LanceModel]
+        on_bad_vectors: str, default "error"
+            What to do if any of the vectors are not the same size or contains NaNs.
+            One of "error", "drop", "fill".
+        fill_value: float
+            The value to use when filling vectors. Only used if on_bad_vectors="fill".
+
+        Returns
+        -------
+        LanceTable
+            A reference to the newly created table.
+
+        !!! note
+
+            The vector index won't be created by default.
+            To create the index, call the `create_index` method on the table.
+
+        Examples
+        --------
+
+        Can create with list of tuples or dictionaries:
+
+        >>> import lancedb
+        >>> db = lancedb.connect("db://...", api_key="...", region="...") # doctest: +SKIP
+        >>> data = [{"vector": [1.1, 1.2], "lat": 45.5, "long": -122.7},
+        ...         {"vector": [0.2, 1.8], "lat": 40.1, "long":  -74.1}]
+        >>> db.create_table("my_table", data) # doctest: +SKIP
+        LanceTable(my_table)
+
+        You can also pass a pandas DataFrame:
+
+        >>> import pandas as pd
+        >>> data = pd.DataFrame({
+        ...    "vector": [[1.1, 1.2], [0.2, 1.8]],
+        ...    "lat": [45.5, 40.1],
+        ...    "long": [-122.7, -74.1]
+        ... })
+        >>> db.create_table("table2", data) # doctest: +SKIP
+        LanceTable(table2)
+
+        >>> custom_schema = pa.schema([
+        ...   pa.field("vector", pa.list_(pa.float32(), 2)),
+        ...   pa.field("lat", pa.float32()),
+        ...   pa.field("long", pa.float32())
+        ... ])
+        >>> db.create_table("table3", data, schema = custom_schema) # doctest: +SKIP
+        LanceTable(table3)
+
+        It is also possible to create an table from `[Iterable[pa.RecordBatch]]`:
+
+        >>> import pyarrow as pa
+        >>> def make_batches():
+        ...     for i in range(5):
+        ...         yield pa.RecordBatch.from_arrays(
+        ...             [
+        ...                 pa.array([[3.1, 4.1], [5.9, 26.5]],
+        ...                     pa.list_(pa.float32(), 2)),
+        ...                 pa.array(["foo", "bar"]),
+        ...                 pa.array([10.0, 20.0]),
+        ...             ],
+        ...             ["vector", "item", "price"],
+        ...         )
+        >>> schema=pa.schema([
+        ...     pa.field("vector", pa.list_(pa.float32(), 2)),
+        ...     pa.field("item", pa.utf8()),
+        ...     pa.field("price", pa.float32()),
+        ... ])
+        >>> db.create_table("table4", make_batches(), schema=schema) # doctest: +SKIP
+        LanceTable(table4)
+
+        """
        if data is None and schema is None:
            raise ValueError("Either data or schema must be provided.")
        if embedding_functions is not None:
--- a/python/lancedb/remote/table.py
+++ b/python/lancedb/remote/table.py
@@ -13,7 +13,7 @@

 import uuid
 from functools import cached_property
-from typing import Optional, Union
+from typing import Dict, Optional, Union

 import pyarrow as pa
 from lance import json_to_schema
@@ -22,6 +22,7 @@ from lancedb.common import DATA, VEC, VECTOR_COLUMN_NAME

 from ..query import LanceVectorQueryBuilder
 from ..table import Query, Table, _sanitize_data
+from ..util import value_to_sql
 from .arrow import to_ipc_binary
 from .client import ARROW_STREAM_CONTENT_TYPE
 from .db import RemoteDBConnection
@@ -37,7 +38,10 @@ class RemoteTable(Table):

    @cached_property
    def schema(self) -> pa.Schema:
-        """Return the schema of the table."""
+        """The [Arrow Schema](https://arrow.apache.org/docs/python/api/datatypes.html#)
+        of this Table
+
+        """
        resp = self._conn._loop.run_until_complete(
            self._conn._client.post(f"/v1/table/{self._name}/describe/")
        )
@@ -53,24 +57,17 @@ class RemoteTable(Table):
        return resp["version"]

    def to_arrow(self) -> pa.Table:
-        """Return the table as an Arrow table."""
+        """to_arrow() is not supported on the LanceDB cloud"""
        raise NotImplementedError("to_arrow() is not supported on the LanceDB cloud")

    def to_pandas(self):
-        """Return the table as a Pandas DataFrame.
-
-        Intercept `to_arrow()` for better error message.
-        """
+        """to_pandas() is not supported on the LanceDB cloud"""
        return NotImplementedError("to_pandas() is not supported on the LanceDB cloud")

    def create_index(
        self,
        metric="L2",
-        num_partitions=256,
-        num_sub_vectors=96,
        vector_column_name: str = VECTOR_COLUMN_NAME,
-        replace: bool = True,
-        accelerator: Optional[str] = None,
        index_cache_size: Optional[int] = None,
    ):
        """Create an index on the table.
@@ -81,39 +78,28 @@ class RemoteTable(Table):
        ----------
        metric : str
            The metric to use for the index. Default is "L2".
-        num_partitions : int
-            The number of partitions to use for the index. Default is 256.
-        num_sub_vectors : int
-            The number of sub-vectors to use for the index. Default is 96.
        vector_column_name : str
            The name of the vector column. Default is "vector".
-        replace : bool
-            Whether to replace the existing index. Default is True.
-        accelerator : str, optional
-            If set, use the given accelerator to create the index.
-            Default is None. Currently not supported.
-        index_cache_size : int, optional
-            The size of the index cache in number of entries. Default value is 256.

        Examples
        --------
-        import lancedb
-        import uuid
-        from lancedb.schema import vector
-        conn = lancedb.connect("db://...", api_key="...", region="...")
-        table_name = uuid.uuid4().hex
-        schema = pa.schema(
-            [
-                    pa.field("id", pa.uint32(), False),
-                    pa.field("vector", vector(128), False),
-                    pa.field("s", pa.string(), False),
-            ]
-        )
-        table = conn.create_table(
-            table_name,
-            schema=schema,
-        )
-        table.create_index()
+        >>> import lancedb
+        >>> import uuid
+        >>> from lancedb.schema import vector
+        >>> db = lancedb.connect("db://...", api_key="...", region="...") # doctest: +SKIP
+        >>> table_name = uuid.uuid4().hex
+        >>> schema = pa.schema(
+        ...     [
+        ...             pa.field("id", pa.uint32(), False),
+        ...            pa.field("vector", vector(128), False),
+        ...             pa.field("s", pa.string(), False),
+        ...     ]
+        ... )
+        >>> table = db.create_table( # doctest: +SKIP
+        ...     table_name, # doctest: +SKIP
+        ...     schema=schema, # doctest: +SKIP
+        ... )
+        >>> table.create_index("L2", "vector") # doctest: +SKIP
        """
        index_type = "vector"

@@ -135,6 +121,28 @@ class RemoteTable(Table):
        on_bad_vectors: str = "error",
        fill_value: float = 0.0,
    ) -> int:
+        """Add more data to the [Table](Table). It has the same API signature as the OSS version.
+
+        Parameters
+        ----------
+        data: DATA
+            The data to insert into the table. Acceptable types are:
+
+            - dict or list-of-dict
+
+            - pandas.DataFrame
+
+            - pyarrow.Table or pyarrow.RecordBatch
+        mode: str
+            The mode to use when writing the data. Valid values are
+            "append" and "overwrite".
+        on_bad_vectors: str, default "error"
+            What to do if any of the vectors are not the same size or contains NaNs.
+            One of "error", "drop", "fill".
+        fill_value: float, default 0.
+            The value to use when filling vectors. Only used if on_bad_vectors="fill".
+
+        """
        data = _sanitize_data(
            data,
            self.schema,
@@ -158,6 +166,58 @@ class RemoteTable(Table):
    def search(
        self, query: Union[VEC, str], vector_column_name: str = VECTOR_COLUMN_NAME
    ) -> LanceVectorQueryBuilder:
+        """Create a search query to find the nearest neighbors
+        of the given query vector. We currently support [vector search][search]
+
+        All query options are defined in [Query][lancedb.query.Query].
+
+        Examples
+        --------
+        >>> import lancedb
+        >>> db = lancedb.connect("db://...", api_key="...", region="...") # doctest: +SKIP
+        >>> data = [
+        ...    {"original_width": 100, "caption": "bar", "vector": [0.1, 2.3, 4.5]},
+        ...    {"original_width": 2000, "caption": "foo",  "vector": [0.5, 3.4, 1.3]},
+        ...    {"original_width": 3000, "caption": "test", "vector": [0.3, 6.2, 2.6]}
+        ... ]
+        >>> table = db.create_table("my_table", data) # doctest: +SKIP
+        >>> query = [0.4, 1.4, 2.4]
+        >>> (table.search(query, vector_column_name="vector") # doctest: +SKIP
+        ...     .where("original_width > 1000", prefilter=True) # doctest: +SKIP
+        ...     .select(["caption", "original_width"]) # doctest: +SKIP
+        ...     .limit(2) # doctest: +SKIP
+        ...     .to_pandas()) # doctest: +SKIP
+          caption  original_width           vector  _distance # doctest: +SKIP
+        0     foo            2000  [0.5, 3.4, 1.3]   5.220000 # doctest: +SKIP
+        1    test            3000  [0.3, 6.2, 2.6]  23.089996 # doctest: +SKIP
+
+        Parameters
+        ----------
+        query: list/np.ndarray/str/PIL.Image.Image, default None
+            The targetted vector to search for.
+
+            - *default None*.
+            Acceptable types are: list, np.ndarray, PIL.Image.Image
+
+            - If None then the select/where/limit clauses are applied to filter
+            the table
+        vector_column_name: str
+            The name of the vector column to search.
+            *default "vector"*
+
+        Returns
+        -------
+        LanceQueryBuilder
+            A query builder object representing the query.
+            Once executed, the query returns
+
+            - selected columns
+
+            - the vector
+
+            - and also the "_distance" column which is the distance between the query
+            vector and the returned vector.
+        """
        return LanceVectorQueryBuilder(self, query, vector_column_name)

    def _execute_query(self, query: Query) -> pa.Table:
@@ -165,8 +225,114 @@ class RemoteTable(Table):
        return self._conn._loop.run_until_complete(result).to_arrow()

    def delete(self, predicate: str):
-        """Delete rows from the table."""
+        """Delete rows from the table.
+
+        This can be used to delete a single row, many rows, all rows, or
+        sometimes no rows (if your predicate matches nothing).
+
+        Parameters
+        ----------
+        predicate: str
+            The SQL where clause to use when deleting rows.
+
+            - For example, 'x = 2' or 'x IN (1, 2, 3)'.
+
+            The filter must not be empty, or it will error.
+
+        Examples
+        --------
+        >>> import lancedb
+        >>> data = [
+        ...    {"x": 1, "vector": [1, 2]},
+        ...    {"x": 2, "vector": [3, 4]},
+        ...    {"x": 3, "vector": [5, 6]}
+        ... ]
+        >>> db = lancedb.connect("db://...", api_key="...", region="...") # doctest: +SKIP
+        >>> table = db.create_table("my_table", data) # doctest: +SKIP
+        >>> table.search([10,10]).to_pandas() # doctest: +SKIP
+           x      vector  _distance # doctest: +SKIP
+        0  3  [5.0, 6.0]       41.0 # doctest: +SKIP
+        1  2  [3.0, 4.0]       85.0 # doctest: +SKIP
+        2  1  [1.0, 2.0]      145.0 # doctest: +SKIP
+        >>> table.delete("x = 2") # doctest: +SKIP
+        >>> table.search([10,10]).to_pandas() # doctest: +SKIP
+           x      vector  _distance # doctest: +SKIP
+        0  3  [5.0, 6.0]       41.0 # doctest: +SKIP
+        1  1  [1.0, 2.0]      145.0 # doctest: +SKIP
+
+        If you have a list of values to delete, you can combine them into a
+        stringified list and use the `IN` operator:
+
+        >>> to_remove = [1, 3] # doctest: +SKIP
+        >>> to_remove = ", ".join([str(v) for v in to_remove]) # doctest: +SKIP
+        >>> table.delete(f"x IN ({to_remove})") # doctest: +SKIP
+        >>> table.search([10,10]).to_pandas() # doctest: +SKIP
+           x      vector  _distance # doctest: +SKIP
+        0  2  [3.0, 4.0]       85.0 # doctest: +SKIP
+        """
        payload = {"predicate": predicate}
        self._conn._loop.run_until_complete(
            self._conn._client.post(f"/v1/table/{self._name}/delete/", data=payload)
        )
+
+    def update(
+        self,
+        where: Optional[str] = None,
+        values: Optional[dict] = None,
+        *,
+        values_sql: Optional[Dict[str, str]] = None,
+    ):
+        """
+        This can be used to update zero to all rows depending on how many
+        rows match the where clause.
+
+        Parameters
+        ----------
+        where: str, optional
+            The SQL where clause to use when updating rows. For example, 'x = 2'
+            or 'x IN (1, 2, 3)'. The filter must not be empty, or it will error.
+        values: dict, optional
+            The values to update. The keys are the column names and the values
+            are the values to set.
+        values_sql: dict, optional
+            The values to update, expressed as SQL expression strings. These can
+            reference existing columns. For example, {"x": "x + 1"} will increment
+            the x column by 1.
+
+        Examples
+        --------
+        >>> import lancedb
+        >>> data = [
+        ...    {"x": 1, "vector": [1, 2]},
+        ...    {"x": 2, "vector": [3, 4]},
+        ...    {"x": 3, "vector": [5, 6]}
+        ... ]
+        >>> db = lancedb.connect("db://...", api_key="...", region="...") # doctest: +SKIP
+        >>> table = db.create_table("my_table", data) # doctest: +SKIP
+        >>> table.to_pandas() # doctest: +SKIP
+           x      vector # doctest: +SKIP
+        0  1  [1.0, 2.0] # doctest: +SKIP
+        1  2  [3.0, 4.0] # doctest: +SKIP
+        2  3  [5.0, 6.0] # doctest: +SKIP
+        >>> table.update(where="x = 2", values={"vector": [10, 10]}) # doctest: +SKIP
+        >>> table.to_pandas() # doctest: +SKIP
+           x        vector # doctest: +SKIP
+        0  1    [1.0, 2.0] # doctest: +SKIP
+        1  3    [5.0, 6.0] # doctest: +SKIP
+        2  2  [10.0, 10.0] # doctest: +SKIP
+
+        """
+        if values is not None and values_sql is not None:
+            raise ValueError("Only one of values or values_sql can be provided")
+        if values is None and values_sql is None:
+            raise ValueError("Either values or values_sql must be provided")
+
+        if values is not None:
+            updates = [[k, value_to_sql(v)] for k, v in values.items()]
+        else:
+            updates = [[k, v] for k, v in values_sql.items()]
+
+        payload = {"predicate": where, "updates": updates}
+        self._conn._loop.run_until_complete(
+            self._conn._client.post(f"/v1/table/{self._name}/update/", data=payload)
+        )
--- a/python/lancedb/table.py
+++ b/python/lancedb/table.py
@@ -17,20 +17,21 @@ import inspect
 import os
 from abc import ABC, abstractmethod
 from functools import cached_property
-from typing import TYPE_CHECKING, Any, Iterable, List, Optional, Union
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Union

 import lance
 import numpy as np
 import pyarrow as pa
 import pyarrow.compute as pc
+import pyarrow.fs as pa_fs
 from lance import LanceDataset
 from lance.vector import vec_to_table

 from .common import DATA, VEC, VECTOR_COLUMN_NAME
 from .embeddings import EmbeddingFunctionConfig, EmbeddingFunctionRegistry
-from .pydantic import LanceModel
+from .pydantic import LanceModel, model_to_dict
 from .query import LanceQueryBuilder, Query
-from .util import fs_from_uri, safe_import_pandas
+from .util import fs_from_uri, safe_import_pandas, value_to_sql, join_uri
 from .utils.events import register_event

 if TYPE_CHECKING:
@@ -53,8 +54,10 @@ def _sanitize_data(
        # convert to list of dict if data is a bunch of LanceModels
        if isinstance(data[0], LanceModel):
            schema = data[0].__class__.to_arrow_schema()
-            data = [dict(d) for d in data]
-        data = pa.Table.from_pylist(data)
+            data = [model_to_dict(d) for d in data]
+            data = pa.Table.from_pylist(data, schema=schema)
+        else:
+            data = pa.Table.from_pylist(data)
    elif isinstance(data, dict):
        data = vec_to_table(data)
    elif pd is not None and isinstance(data, pd.DataFrame):
@@ -394,14 +397,6 @@ class LanceTable(Table):
        self.name = name
        self._version = version

-    def _reset_dataset(self, version=None):
-        try:
-            if "_dataset" in self.__dict__:
-                del self.__dict__["_dataset"]
-            self._version = version
-        except AttributeError:
-            pass
-
    @property
    def schema(self) -> pa.Schema:
        """Return the schema of the table.
@@ -410,16 +405,16 @@ class LanceTable(Table):
        -------
        pa.Schema
            A PyArrow schema object."""
-        return self._dataset.schema
+        return self.to_lance().schema

    def list_versions(self):
        """List all versions of the table"""
-        return self._dataset.versions()
+        return self.to_lance().versions()

    @property
    def version(self) -> int:
        """Get the current version of the table"""
-        return self._dataset.version
+        return self.to_lance().version

    def checkout(self, version: int):
        """Checkout a version of the table. This is an in-place operation.
@@ -452,14 +447,12 @@ class LanceTable(Table):
               vector    type
        0  [1.1, 0.9]  vector
        """
-        max_ver = max([v["version"] for v in self._dataset.versions()])
+        max_ver = max([v["version"] for v in self.to_lance().versions()])
        if version < 1 or version > max_ver:
            raise ValueError(f"Invalid version {version}")
-        self._reset_dataset(version=version)

        try:
-            # Accessing the property updates the cached value
-            _ = self._dataset
+            self.to_lance().checkout(version)
        except Exception as e:
            if "not found" in str(e):
                raise ValueError(
@@ -502,7 +495,7 @@ class LanceTable(Table):
        >>> len(table.list_versions())
        4
        """
-        max_ver = max([v["version"] for v in self._dataset.versions()])
+        max_ver = max([v["version"] for v in self.to_lance().versions()])
        if version is None:
            version = self.version
        elif version < 1 or version > max_ver:
@@ -514,11 +507,10 @@ class LanceTable(Table):
            # no-op if restoring the latest version
            return

-        self._dataset.restore()
-        self._reset_dataset()
+        self.to_lance().restore()

    def __len__(self):
-        return self._dataset.count_rows()
+        return self.to_lance().count_rows()

    def __repr__(self) -> str:
        return f"LanceTable({self.name})"
@@ -528,7 +520,7 @@ class LanceTable(Table):

    def head(self, n=5) -> pa.Table:
        """Return the first n rows of the table."""
-        return self._dataset.head(n)
+        return self.to_lance().head(n)

    def to_pandas(self) -> "pd.DataFrame":
        """Return the table as a pandas DataFrame.
@@ -545,11 +537,11 @@ class LanceTable(Table):
        Returns
        -------
        pa.Table"""
-        return self._dataset.to_table()
+        return self.to_lance().to_table()

    @property
    def _dataset_uri(self) -> str:
-        return os.path.join(self._conn.uri, f"{self.name}.lance")
+        return join_uri(self._conn.uri, f"{self.name}.lance")

    def create_index(
        self,
@@ -572,10 +564,11 @@ class LanceTable(Table):
            accelerator=accelerator,
            index_cache_size=index_cache_size,
        )
-        self._reset_dataset()
        register_event("create_index")

-    def create_fts_index(self, field_names: Union[str, List[str]]):
+    def create_fts_index(
+        self, field_names: Union[str, List[str]], *, replace: bool = False
+    ):
        """Create a full-text search index on the table.

        Warning - this API is highly experimental and is highly likely to change
@@ -585,17 +578,35 @@ class LanceTable(Table):
        ----------
        field_names: str or list of str
            The name(s) of the field to index.
+        replace: bool, default False
+            If True, replace the existing index if it exists. Note that this is
+            not yet an atomic operation; the index will be temporarily
+            unavailable while the new index is being created.
        """
        from .fts import create_index, populate_index

        if isinstance(field_names, str):
            field_names = [field_names]
+
+        fs, path = fs_from_uri(self._get_fts_index_path())
+        index_exists = fs.get_file_info(path).type != pa_fs.FileType.NotFound
+        if index_exists:
+            if not replace:
+                raise ValueError(
+                    f"Index already exists. Use replace=True to overwrite."
+                )
+            try:
+                fs.delete_dir(path)
+            except FileNotFoundError as e:
+                if "Cannot get information for path" in str(e):
+                    pass
+
        index = create_index(self._get_fts_index_path(), field_names)
        populate_index(index, self, field_names)
        register_event("create_fts_index")

    def _get_fts_index_path(self):
-        return os.path.join(self._dataset_uri, "_indices", "tantivy")
+        return join_uri(self._dataset_uri, "_indices", "tantivy")

    @cached_property
    def _dataset(self) -> LanceDataset:
@@ -643,8 +654,7 @@ class LanceTable(Table):
            on_bad_vectors=on_bad_vectors,
            fill_value=fill_value,
        )
-        lance.write_dataset(data, self._dataset_uri, schema=self.schema, mode=mode)
-        self._reset_dataset()
+        self.to_lance().write(data, mode=mode)
        register_event("add")

    def merge(
@@ -705,10 +715,9 @@ class LanceTable(Table):
            other_table = other_table.to_lance()
        if isinstance(other_table, LanceDataset):
            other_table = other_table.to_table()
-        self._dataset.merge(
+        self.to_lance().merge(
            other_table, left_on=left_on, right_on=right_on, schema=schema
        )
-        self._reset_dataset()
        register_event("merge")

    @cached_property
@@ -785,7 +794,7 @@ class LanceTable(Table):
            and also the "_distance" column which is the distance between the query
            vector and the returned vector.
        """
-        register_event("search")
+        register_event("search_table")
        return LanceQueryBuilder.create(
            self, query, query_type, vector_column_name=vector_column_name
        )
@@ -906,35 +915,42 @@ class LanceTable(Table):
                f"Table {name} does not exist."
                f"Please first call db.create_table({name}, data)"
            )
+        register_event("open_table")
+
        return tbl

    def delete(self, where: str):
-        self._dataset.delete(where)
+        self.to_lance().delete(where)

-    def update(self, where: str, values: dict):
+    def update(
+        self,
+        where: Optional[str] = None,
+        values: Optional[dict] = None,
+        *,
+        values_sql: Optional[Dict[str, str]] = None,
+    ):
        """
-        EXPERIMENTAL: Update rows in the table (not threadsafe).
-
        This can be used to update zero to all rows depending on how many
        rows match the where clause.

        Parameters
        ----------
-        where: str
+        where: str, optional
            The SQL where clause to use when updating rows. For example, 'x = 2'
            or 'x IN (1, 2, 3)'. The filter must not be empty, or it will error.
-        values: dict
+        values: dict, optional
            The values to update. The keys are the column names and the values
            are the values to set.
+        values_sql: dict, optional
+            The values to update, expressed as SQL expression strings. These can
+            reference existing columns. For example, {"x": "x + 1"} will increment
+            the x column by 1.

        Examples
        --------
        >>> import lancedb
-        >>> data = [
-        ...    {"x": 1, "vector": [1, 2]},
-        ...    {"x": 2, "vector": [3, 4]},
-        ...    {"x": 3, "vector": [5, 6]}
-        ... ]
+        >>> import pandas as pd
+        >>> data = pd.DataFrame({"x": [1, 2, 3], "vector": [[1, 2], [3, 4], [5, 6]]})
        >>> db = lancedb.connect("./.lancedb")
        >>> table = db.create_table("my_table", data)
        >>> table.to_pandas()
@@ -950,19 +966,15 @@ class LanceTable(Table):
        2  2  [10.0, 10.0]

        """
-        orig_data = self._dataset.to_table(filter=where).combine_chunks()
-        if len(orig_data) == 0:
-            return
-        for col, val in values.items():
-            i = orig_data.column_names.index(col)
-            if i < 0:
-                raise ValueError(f"Column {col} does not exist")
-            orig_data = orig_data.set_column(
-                i, col, pa.array([val] * len(orig_data), type=orig_data[col].type)
-            )
-        self.delete(where)
-        self.add(orig_data, mode="append")
-        self._reset_dataset()
+        if values is not None and values_sql is not None:
+            raise ValueError("Only one of values or values_sql can be provided")
+        if values is None and values_sql is None:
+            raise ValueError("Either values or values_sql must be provided")
+
+        if values is not None:
+            values_sql = {k: value_to_sql(v) for k, v in values.items()}
+
+        self.to_lance().update(values_sql, where)
        register_event("update")

    def _execute_query(self, query: Query) -> pa.Table:
--- a/python/lancedb/util.py
+++ b/python/lancedb/util.py
@@ -12,9 +12,13 @@
 #  limitations under the License.

 import os
-from typing import Tuple
+from datetime import date, datetime
+from functools import singledispatch
+import pathlib
+from typing import Tuple, Union
 from urllib.parse import urlparse

+import numpy as np
 import pyarrow.fs as pa_fs


@@ -59,6 +63,12 @@ def get_uri_location(uri: str) -> str:
    str: Location part of the URL, without scheme
    """
    parsed = urlparse(uri)
+    if len(parsed.scheme) == 1:
+        # Windows drive names are parsed as the scheme
+        # e.g. "c:\path" -> ParseResult(scheme="c", netloc="", path="/path", ...)
+        # So we add special handling here for schemes that are a single character
+        return uri
+
    if not parsed.netloc:
        return parsed.path
    else:
@@ -81,6 +91,29 @@ def fs_from_uri(uri: str) -> Tuple[pa_fs.FileSystem, str]:
    return pa_fs.FileSystem.from_uri(uri)


+def join_uri(base: Union[str, pathlib.Path], *parts: str) -> str:
+    """
+    Join a URI with multiple parts, handles both local and remote paths
+
+    Parameters
+    ----------
+    base : str
+        The base URI
+    parts : str
+        The parts to join to the base URI, each separated by the
+        appropriate path separator for the URI scheme and OS
+    """
+    if isinstance(base, pathlib.Path):
+        return base.joinpath(*parts)
+    base = str(base)
+    if get_uri_scheme(base) == "file":
+        # using pathlib for local paths make this windows compatible
+        # `get_uri_scheme` returns `file` for windows drive names (e.g. `c:\path`)
+        return str(pathlib.Path(base, *parts))
+    # for remote paths, just use os.path.join
+    return "/".join([p.rstrip("/") for p in [base, *parts]])
+
+
 def safe_import_pandas():
    try:
        import pandas as pd
@@ -88,3 +121,53 @@ def safe_import_pandas():
        return pd
    except ImportError:
        return None
+
+
+@singledispatch
+def value_to_sql(value):
+    raise NotImplementedError("SQL conversion is not implemented for this type")
+
+
+@value_to_sql.register(str)
+def _(value: str):
+    return f"'{value}'"
+
+
+@value_to_sql.register(int)
+def _(value: int):
+    return str(value)
+
+
+@value_to_sql.register(float)
+def _(value: float):
+    return str(value)
+
+
+@value_to_sql.register(bool)
+def _(value: bool):
+    return str(value).upper()
+
+
+@value_to_sql.register(type(None))
+def _(value: type(None)):
+    return "NULL"
+
+
+@value_to_sql.register(datetime)
+def _(value: datetime):
+    return f"'{value.isoformat()}'"
+
+
+@value_to_sql.register(date)
+def _(value: date):
+    return f"'{value.isoformat()}'"
+
+
+@value_to_sql.register(list)
+def _(value: list):
+    return "[" + ", ".join(map(value_to_sql, value)) + "]"
+
+
+@value_to_sql.register(np.ndarray)
+def _(value: np.ndarray):
+    return value_to_sql(value.tolist())
--- a/python/lancedb/utils/events.py
+++ b/python/lancedb/utils/events.py
@@ -64,8 +64,10 @@ class _Events:
        Initializes the Events object with default values for events, rate_limit, and metadata.
        """
        self.events = []  # events list
-        self.max_events = 25  # max events to store in memory
-        self.rate_limit = 60.0  # rate limit (seconds)
+        self.throttled_event_names = ["search_table"]
+        self.throttled_events = set()
+        self.max_events = 5  # max events to store in memory
+        self.rate_limit = 60.0 * 5  # rate limit (seconds)
        self.time = 0.0

        if is_git_dir():
@@ -112,18 +114,21 @@ class _Events:
            return
        if (
            len(self.events) < self.max_events
-        ):  # Events list limited to 25 events (drop any events past this)
+        ):  # Events list limited to self.max_events (drop any events past this)
            params.update(self.metadata)
-            self.events.append(
-                {
-                    "event": event_name,
-                    "properties": params,
-                    "timestamp": datetime.datetime.now(
-                        tz=datetime.timezone.utc
-                    ).isoformat(),
-                    "distinct_id": CONFIG["uuid"],
-                }
-            )
+            event = {
+                "event": event_name,
+                "properties": params,
+                "timestamp": datetime.datetime.now(
+                    tz=datetime.timezone.utc
+                ).isoformat(),
+                "distinct_id": CONFIG["uuid"],
+            }
+            if event_name not in self.throttled_event_names:
+                self.events.append(event)
+            elif event_name not in self.throttled_events:
+                self.throttled_events.add(event_name)
+                self.events.append(event)

        # Check rate limit
        t = time.time()
@@ -135,7 +140,6 @@ class _Events:
            "distinct_id": CONFIG["uuid"],  # posthog needs this to accepts the event
            "batch": self.events,
        }
-
        # POST equivalent to requests.post(self.url, json=data).
        # threaded request is used to avoid blocking, retries are disabled, and verbose is disabled
        # to avoid any possible disruption in the console.
@@ -150,6 +154,7 @@ class _Events:

        # Flush & Reset
        self.events = []
+        self.throttled_events = set()
        self.time = t


--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -1,12 +1,12 @@
 [project]
 name = "lancedb"
-version = "0.3.4"
+version = "0.4.0"
 dependencies = [
    "deprecation",
-    "pylance==0.8.17",
+    "pylance==0.9.1",
    "ratelimiter~=1.0",
    "retry>=0.9.2",
-    "tqdm>=4.1.0",
+    "tqdm>=4.27.0",
    "aiohttp",
    "pydantic>=1.10",
    "attrs>=21.3.0",
--- a/python/tests/test_fts.py
+++ b/python/tests/test_fts.py
@@ -43,7 +43,15 @@ def table(tmp_path) -> ldb.table.LanceTable:
        for _ in range(100)
    ]
    table = db.create_table(
-        "test", data=pd.DataFrame({"vector": vectors, "text": text, "text2": text})
+        "test",
+        data=pd.DataFrame(
+            {
+                "vector": vectors,
+                "text": text,
+                "text2": text,
+                "nested": [{"text": t} for t in text],
+            }
+        ),
    )
    return table

@@ -75,6 +83,24 @@ def test_create_index_from_table(tmp_path, table):
    assert len(df) == 10
    assert "text" in df.columns

+    # Check whether it can be updated
+    table.add(
+        [
+            {
+                "vector": np.random.randn(128),
+                "text": "gorilla",
+                "text2": "gorilla",
+                "nested": {"text": "gorilla"},
+            }
+        ]
+    )
+
+    table.create_fts_index("text", replace=True)
+    assert len(table.search("gorilla").limit(1).to_pandas()) == 1
+
+    with pytest.raises(ValueError, match="already exists"):
+        table.create_fts_index("text")
+

 def test_create_index_multiple_columns(tmp_path, table):
    table.create_fts_index(["text", "text2"])
@@ -89,3 +115,9 @@ def test_empty_rs(tmp_path, table, mocker):
    mocker.patch("lancedb.fts.search_index", return_value=([], []))
    df = table.search("puppy").limit(10).to_pandas()
    assert len(df) == 0
+
+
+def test_nested_schema(tmp_path, table):
+    table.create_fts_index("nested.text")
+    rs = table.search("puppy").limit(10).to_list()
+    assert len(rs) == 10
--- a/python/tests/test_table.py
+++ b/python/tests/test_table.py
@@ -12,7 +12,7 @@
 #  limitations under the License.

 import functools
-from datetime import timedelta
+from datetime import date, datetime, timedelta
 from pathlib import Path
 from typing import List
 from unittest.mock import PropertyMock, patch
@@ -22,6 +22,7 @@ import numpy as np
 import pandas as pd
 import pyarrow as pa
 import pytest
+from pydantic import BaseModel

 from lancedb.conftest import MockTextEmbeddingFunction
 from lancedb.db import LanceDBConnection
@@ -141,14 +142,44 @@ def test_add(db):


 def test_add_pydantic_model(db):
-    class TestModel(LanceModel):
-        vector: Vector(16)
-        li: List[int]
+    # https://github.com/lancedb/lancedb/issues/562

-    data = TestModel(vector=list(range(16)), li=[1, 2, 3])
-    table = LanceTable.create(db, "test", data=[data])
-    assert len(table) == 1
-    assert table.schema == TestModel.to_arrow_schema()
+    class Metadata(BaseModel):
+        source: str
+        timestamp: datetime
+
+    class Document(BaseModel):
+        content: str
+        meta: Metadata
+
+    class LanceSchema(LanceModel):
+        id: str
+        vector: Vector(2)
+        li: List[int]
+        payload: Document
+
+    tbl = LanceTable.create(db, "mytable", schema=LanceSchema, mode="overwrite")
+    assert tbl.schema == LanceSchema.to_arrow_schema()
+
+    # add works
+    expected = LanceSchema(
+        id="id",
+        vector=[0.0, 0.0],
+        li=[1, 2, 3],
+        payload=Document(
+            content="foo", meta=Metadata(source="bar", timestamp=datetime.now())
+        ),
+    )
+    tbl.add([expected])
+
+    result = tbl.search([0.0, 0.0]).limit(1).to_pydantic(LanceSchema)[0]
+    assert result == expected
+
+    flattened = tbl.search([0.0, 0.0]).limit(1).to_pandas(flatten=1)
+    assert len(flattened.columns) == 6  # _distance is automatically added
+
+    really_flattened = tbl.search([0.0, 0.0]).limit(1).to_pandas(flatten=True)
+    assert len(really_flattened.columns) == 7


 def _add(table, schema):
@@ -195,39 +226,38 @@ def test_versioning(db):


 def test_create_index_method():
-    with patch.object(LanceTable, "_reset_dataset", return_value=None):
-        with patch.object(
-            LanceTable, "_dataset", new_callable=PropertyMock
-        ) as mock_dataset:
-            # Setup mock responses
-            mock_dataset.return_value.create_index.return_value = None
+    with patch.object(
+        LanceTable, "_dataset", new_callable=PropertyMock
+    ) as mock_dataset:
+        # Setup mock responses
+        mock_dataset.return_value.create_index.return_value = None

-            # Create a LanceTable object
-            connection = LanceDBConnection(uri="mock.uri")
-            table = LanceTable(connection, "test_table")
+        # Create a LanceTable object
+        connection = LanceDBConnection(uri="mock.uri")
+        table = LanceTable(connection, "test_table")

-            # Call the create_index method
-            table.create_index(
-                metric="L2",
-                num_partitions=256,
-                num_sub_vectors=96,
-                vector_column_name="vector",
-                replace=True,
-                index_cache_size=256,
-            )
+        # Call the create_index method
+        table.create_index(
+            metric="L2",
+            num_partitions=256,
+            num_sub_vectors=96,
+            vector_column_name="vector",
+            replace=True,
+            index_cache_size=256,
+        )

-            # Check that the _dataset.create_index method was called
-            # with the right parameters
-            mock_dataset.return_value.create_index.assert_called_once_with(
-                column="vector",
-                index_type="IVF_PQ",
-                metric="L2",
-                num_partitions=256,
-                num_sub_vectors=96,
-                replace=True,
-                accelerator=None,
-                index_cache_size=256,
-            )
+        # Check that the _dataset.create_index method was called
+        # with the right parameters
+        mock_dataset.return_value.create_index.assert_called_once_with(
+            column="vector",
+            index_type="IVF_PQ",
+            metric="L2",
+            num_partitions=256,
+            num_sub_vectors=96,
+            replace=True,
+            accelerator=None,
+            index_cache_size=256,
+        )


 def test_add_with_nans(db):
@@ -348,14 +378,79 @@ def test_update(db):
    assert len(table) == 2
    assert len(table.list_versions()) == 2
    table.update(where="id=0", values={"vector": [1.1, 1.1]})
-    assert len(table.list_versions()) == 4
-    assert table.version == 4
+    assert len(table.list_versions()) == 3
+    assert table.version == 3
    assert len(table) == 2
    v = table.to_arrow()["vector"].combine_chunks()
    v = v.values.to_numpy().reshape(2, 2)
    assert np.allclose(v, np.array([[1.2, 1.9], [1.1, 1.1]]))


+def test_update_types(db):
+    table = LanceTable.create(
+        db,
+        "my_table",
+        data=[
+            {
+                "id": 0,
+                "str": "foo",
+                "float": 1.1,
+                "timestamp": datetime(2021, 1, 1),
+                "date": date(2021, 1, 1),
+                "vector1": [1.0, 0.0],
+                "vector2": [1.0, 1.0],
+            }
+        ],
+    )
+    # Update with SQL
+    table.update(
+        values_sql=dict(
+            id="1",
+            str="'bar'",
+            float="2.2",
+            timestamp="TIMESTAMP '2021-01-02 00:00:00'",
+            date="DATE '2021-01-02'",
+            vector1="[2.0, 2.0]",
+            vector2="[3.0, 3.0]",
+        )
+    )
+    actual = table.to_arrow().to_pylist()[0]
+    expected = dict(
+        id=1,
+        str="bar",
+        float=2.2,
+        timestamp=datetime(2021, 1, 2),
+        date=date(2021, 1, 2),
+        vector1=[2.0, 2.0],
+        vector2=[3.0, 3.0],
+    )
+    assert actual == expected
+
+    # Update with values
+    table.update(
+        values=dict(
+            id=2,
+            str="baz",
+            float=3.3,
+            timestamp=datetime(2021, 1, 3),
+            date=date(2021, 1, 3),
+            vector1=[3.0, 3.0],
+            vector2=np.array([4.0, 4.0]),
+        )
+    )
+    actual = table.to_arrow().to_pylist()[0]
+    expected = dict(
+        id=2,
+        str="baz",
+        float=3.3,
+        timestamp=datetime(2021, 1, 3),
+        date=date(2021, 1, 3),
+        vector1=[3.0, 3.0],
+        vector2=[4.0, 4.0],
+    )
+    assert actual == expected
+
+
 def test_create_with_embedding_function(db):
    class MyTable(LanceModel):
        text: str
--- a/python/tests/test_util.py
+++ b/python/tests/test_util.py
@@ -11,7 +11,12 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.

-from lancedb.util import get_uri_scheme
+import os
+import pathlib
+
+import pytest
+
+from lancedb.util import get_uri_scheme, join_uri


 def test_normalize_uri():
@@ -28,3 +33,55 @@ def test_normalize_uri():
    for uri, expected_scheme in zip(uris, schemes):
        parsed_scheme = get_uri_scheme(uri)
        assert parsed_scheme == expected_scheme
+
+
+def test_join_uri_remote():
+    schemes = ["s3", "az", "gs"]
+    for scheme in schemes:
+        expected = f"{scheme}://bucket/path/to/table.lance"
+        base_uri = f"{scheme}://bucket/path/to/"
+        parts = ["table.lance"]
+        assert join_uri(base_uri, *parts) == expected
+
+        base_uri = f"{scheme}://bucket"
+        parts = ["path", "to", "table.lance"]
+        assert join_uri(base_uri, *parts) == expected
+
+
+# skip this test if on windows
+@pytest.mark.skipif(os.name == "nt", reason="Windows paths are not POSIX")
+def test_join_uri_posix():
+    for base in [
+        # relative path
+        "relative/path",
+        "relative/path/",
+        # an absolute path
+        "/absolute/path",
+        "/absolute/path/",
+        # a file URI
+        "file:///absolute/path",
+        "file:///absolute/path/",
+    ]:
+        joined = join_uri(base, "table.lance")
+        assert joined == str(pathlib.Path(base) / "table.lance")
+        joined = join_uri(pathlib.Path(base), "table.lance")
+        assert joined == pathlib.Path(base) / "table.lance"
+
+
+# skip this test if not on windows
+@pytest.mark.skipif(os.name != "nt", reason="Windows paths are not POSIX")
+def test_local_join_uri_windows():
+    # https://learn.microsoft.com/en-us/dotnet/standard/io/file-path-formats
+    for base in [
+        # windows relative path
+        "relative\\path",
+        "relative\\path\\",
+        # windows absolute path from current drive
+        "c:\\absolute\\path",
+        # relative path from root of current drive
+        "\\relative\\path",
+    ]:
+        joined = join_uri(base, "table.lance")
+        assert joined == str(pathlib.Path(base) / "table.lance")
+        joined = join_uri(pathlib.Path(base), "table.lance")
+        assert joined == pathlib.Path(base) / "table.lance"
--- a/rust/ffi/node/Cargo.toml
+++ b/rust/ffi/node/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "vectordb-node"
-version = "0.3.9"
+version = "0.4.0"
 description = "Serverless, low-latency vector database for AI applications"
 license = "Apache-2.0"
 edition = "2018"
--- a/rust/ffi/node/src/error.rs
+++ b/rust/ffi/node/src/error.rs
@@ -23,7 +23,7 @@ pub enum Error {
    #[snafu(display("column '{name}' is missing"))]
    MissingColumn { name: String },
    #[snafu(display("{name}: {message}"))]
-    RangeError { name: String, message: String },
+    OutOfRange { name: String, message: String },
    #[snafu(display("{index_type} is not a valid index type"))]
    InvalidIndexType { index_type: String },

--- a/rust/ffi/node/src/index/vector.rs
+++ b/rust/ffi/node/src/index/vector.rs
@@ -65,12 +65,10 @@ fn get_index_params_builder(
            obj.get_opt::<JsString, _, _>(cx, "index_name")?
                .map(|s| index_builder.index_name(s.value(cx)));

-            obj.get_opt::<JsString, _, _>(cx, "metric_type")?
-                .map(|s| MetricType::try_from(s.value(cx).as_str()))
-                .map(|mt| {
-                    let metric_type = mt.unwrap();
-                    index_builder.metric_type(metric_type);
-                });
+            if let Some(metric_type) = obj.get_opt::<JsString, _, _>(cx, "metric_type")? {
+                let metric_type = MetricType::try_from(metric_type.value(cx).as_str()).unwrap();
+                index_builder.metric_type(metric_type);
+            }

            let num_partitions = obj.get_opt_usize(cx, "num_partitions")?;
            let max_iters = obj.get_opt_usize(cx, "max_iters")?;
@@ -85,23 +83,29 @@ fn get_index_params_builder(
                index_builder.ivf_params(ivf_params)
            });

-            obj.get_opt::<JsBoolean, _, _>(cx, "use_opq")?
-                .map(|s| pq_params.use_opq = s.value(cx));
+            if let Some(use_opq) = obj.get_opt::<JsBoolean, _, _>(cx, "use_opq")? {
+                pq_params.use_opq = use_opq.value(cx);
+            }

-            obj.get_opt_usize(cx, "num_sub_vectors")?
-                .map(|s| pq_params.num_sub_vectors = s);
+            if let Some(num_sub_vectors) = obj.get_opt_usize(cx, "num_sub_vectors")? {
+                pq_params.num_sub_vectors = num_sub_vectors;
+            }

-            obj.get_opt_usize(cx, "num_bits")?
-                .map(|s| pq_params.num_bits = s);
+            if let Some(num_bits) = obj.get_opt_usize(cx, "num_bits")? {
+                pq_params.num_bits = num_bits;
+            }

-            obj.get_opt_usize(cx, "max_iters")?
-                .map(|s| pq_params.max_iters = s);
+            if let Some(max_iters) = obj.get_opt_usize(cx, "max_iters")? {
+                pq_params.max_iters = max_iters;
+            }

-            obj.get_opt_usize(cx, "max_opq_iters")?
-                .map(|s| pq_params.max_opq_iters = s);
+            if let Some(max_opq_iters) = obj.get_opt_usize(cx, "max_opq_iters")? {
+                pq_params.max_opq_iters = max_opq_iters;
+            }

-            obj.get_opt::<JsBoolean, _, _>(cx, "replace")?
-                .map(|s| index_builder.replace(s.value(cx)));
+            if let Some(replace) = obj.get_opt::<JsBoolean, _, _>(cx, "replace")? {
+                index_builder.replace(replace.value(cx));
+            }

            Ok(index_builder)
        }
--- a/rust/ffi/node/src/lib.rs
+++ b/rust/ffi/node/src/lib.rs
@@ -237,6 +237,7 @@ fn main(mut cx: ModuleContext) -> NeonResult<()> {
    cx.export_function("tableAdd", JsTable::js_add)?;
    cx.export_function("tableCountRows", JsTable::js_count_rows)?;
    cx.export_function("tableDelete", JsTable::js_delete)?;
+    cx.export_function("tableUpdate", JsTable::js_update)?;
    cx.export_function("tableCleanupOldVersions", JsTable::js_cleanup)?;
    cx.export_function("tableCompactFiles", JsTable::js_compact)?;
    cx.export_function("tableListIndices", JsTable::js_list_indices)?;
--- a/rust/ffi/node/src/neon_ext/js_object_ext.rs
+++ b/rust/ffi/node/src/neon_ext/js_object_ext.rs
@@ -47,15 +47,15 @@ fn f64_to_u32_safe(n: f64, key: &str) -> Result<u32> {
    use conv::*;

    n.approx_as::<u32>().map_err(|e| match e {
-        FloatError::NegOverflow(_) => Error::RangeError {
+        FloatError::NegOverflow(_) => Error::OutOfRange {
            name: key.into(),
            message: "must be > 0".to_string(),
        },
-        FloatError::PosOverflow(_) => Error::RangeError {
+        FloatError::PosOverflow(_) => Error::OutOfRange {
            name: key.into(),
            message: format!("must be < {}", u32::MAX),
        },
-        FloatError::NotANumber(_) => Error::RangeError {
+        FloatError::NotANumber(_) => Error::OutOfRange {
            name: key.into(),
            message: "not a valid number".to_string(),
        },
@@ -66,15 +66,15 @@ fn f64_to_usize_safe(n: f64, key: &str) -> Result<usize> {
    use conv::*;

    n.approx_as::<usize>().map_err(|e| match e {
-        FloatError::NegOverflow(_) => Error::RangeError {
+        FloatError::NegOverflow(_) => Error::OutOfRange {
            name: key.into(),
            message: "must be > 0".to_string(),
        },
-        FloatError::PosOverflow(_) => Error::RangeError {
+        FloatError::PosOverflow(_) => Error::OutOfRange {
            name: key.into(),
            message: format!("must be < {}", usize::MAX),
        },
-        FloatError::NotANumber(_) => Error::RangeError {
+        FloatError::NotANumber(_) => Error::OutOfRange {
            name: key.into(),
            message: "not a valid number".to_string(),
        },
--- a/rust/ffi/node/src/query.rs
+++ b/rust/ffi/node/src/query.rs
@@ -23,8 +23,14 @@ impl JsQuery {
        let query_obj = cx.argument::<JsObject>(0)?;

        let limit = query_obj
-            .get::<JsNumber, _, _>(&mut cx, "_limit")?
-            .value(&mut cx);
+            .get_opt::<JsNumber, _, _>(&mut cx, "_limit")?
+            .map(|value| {
+                let limit = value.value(&mut cx);
+                if limit <= 0.0 {
+                    panic!("Limit must be a positive integer");
+                }
+                limit as u64
+            });
        let select = query_obj
            .get_opt::<JsArray, _, _>(&mut cx, "_select")?
            .map(|arr| {
@@ -48,7 +54,9 @@ impl JsQuery {
            .map(|s| s.value(&mut cx))
            .map(|s| MetricType::try_from(s.as_str()).unwrap());

-        let prefilter = query_obj.get::<JsBoolean, _, _>(&mut cx, "_prefilter")?.value(&mut cx);
+        let prefilter = query_obj
+            .get::<JsBoolean, _, _>(&mut cx, "_prefilter")?
+            .value(&mut cx);

        let is_electron = cx
            .argument::<JsBoolean>(1)
@@ -59,20 +67,23 @@ impl JsQuery {

        let (deferred, promise) = cx.promise();
        let channel = cx.channel();
-        let query_vector = query_obj.get::<JsArray, _, _>(&mut cx, "_queryVector")?;
-        let query = convert::js_array_to_vec(query_vector.deref(), &mut cx);
+        let query_vector = query_obj.get_opt::<JsArray, _, _>(&mut cx, "_queryVector")?;
        let table = js_table.table.clone();
+        let query = query_vector.map(|q| convert::js_array_to_vec(q.deref(), &mut cx));

        rt.spawn(async move {
-            let builder = table
-                .search(Float32Array::from(query))
-                .limit(limit as usize)
+            let mut builder = table
+                .search(query.map(Float32Array::from))
                .refine_factor(refine_factor)
                .nprobes(nprobes)
                .filter(filter)
                .metric_type(metric_type)
                .select(select)
                .prefilter(prefilter);
+            if let Some(limit) = limit {
+                builder = builder.limit(limit as usize);
+            };
+
            let record_batch_stream = builder.execute();
            let results = record_batch_stream
                .and_then(|stream| {
--- a/rust/ffi/node/src/table.rs
+++ b/rust/ffi/node/src/table.rs
@@ -45,7 +45,7 @@ impl JsTable {
        let table_name = cx.argument::<JsString>(0)?.value(&mut cx);
        let buffer = cx.argument::<JsBuffer>(1)?;
        let (batches, schema) =
-            arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;
+            arrow_buffer_to_record_batch(buffer.as_slice(&cx)).or_throw(&mut cx)?;

        // Write mode
        let mode = match cx.argument::<JsString>(2)?.value(&mut cx).as_str() {
@@ -93,7 +93,7 @@ impl JsTable {
        let buffer = cx.argument::<JsBuffer>(0)?;
        let write_mode = cx.argument::<JsString>(1)?.value(&mut cx);
        let (batches, schema) =
-            arrow_buffer_to_record_batch(buffer.as_slice(&mut cx)).or_throw(&mut cx)?;
+            arrow_buffer_to_record_batch(buffer.as_slice(&cx)).or_throw(&mut cx)?;
        let rt = runtime(&mut cx)?;
        let channel = cx.channel();
        let mut table = js_table.table.clone();
@@ -165,6 +165,69 @@ impl JsTable {
        Ok(promise)
    }

+    pub(crate) fn js_update(mut cx: FunctionContext) -> JsResult<JsPromise> {
+        let js_table = cx.this().downcast_or_throw::<JsBox<JsTable>, _>(&mut cx)?;
+        let mut table = js_table.table.clone();
+
+        let rt = runtime(&mut cx)?;
+        let (deferred, promise) = cx.promise();
+        let channel = cx.channel();
+
+        // create a vector of updates from the passed map
+        let updates_arg = cx.argument::<JsObject>(1)?;
+        let properties = updates_arg.get_own_property_names(&mut cx)?;
+        let mut updates: Vec<(String, String)> =
+            Vec::with_capacity(properties.len(&mut cx) as usize);
+
+        let len_properties = properties.len(&mut cx);
+        for i in 0..len_properties {
+            let property = properties
+                .get_value(&mut cx, i)?
+                .downcast_or_throw::<JsString, _>(&mut cx)?;
+
+            let value = updates_arg
+                .get_value(&mut cx, property)?
+                .downcast_or_throw::<JsString, _>(&mut cx)?;
+
+            let property = property.value(&mut cx);
+            let value = value.value(&mut cx);
+            updates.push((property, value));
+        }
+
+        // get the filter/predicate if the user passed one
+        let predicate = cx.argument_opt(0);
+        let predicate = predicate.unwrap().downcast::<JsString, _>(&mut cx);
+        let predicate = match predicate {
+            Ok(_) => {
+                let val = predicate.map(|s| s.value(&mut cx)).unwrap();
+                Some(val)
+            }
+            Err(_) => {
+                // if the predicate is not string, check it's null otherwise an invalid
+                // type was passed
+                cx.argument::<JsNull>(0)?;
+                None
+            }
+        };
+
+        rt.spawn(async move {
+            let updates_arg = updates
+                .iter()
+                .map(|(k, v)| (k.as_str(), v.as_str()))
+                .collect::<Vec<_>>();
+
+            let predicate = predicate.as_deref();
+
+            let update_result = table.update(predicate, updates_arg).await;
+            deferred.settle_with(&channel, move |mut cx| {
+                update_result.or_throw(&mut cx)?;
+                Ok(cx.boxed(JsTable::from(table)))
+            })
+        });
+
+        Ok(promise)
+    }
+
    pub(crate) fn js_cleanup(mut cx: FunctionContext) -> JsResult<JsPromise> {
        let js_table = cx.this().downcast_or_throw::<JsBox<JsTable>, _>(&mut cx)?;
        let rt = runtime(&mut cx)?;
--- a/rust/vectordb/Cargo.toml
+++ b/rust/vectordb/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "vectordb"
-version = "0.3.9"
+version = "0.4.0"
 edition = "2021"
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license = "Apache-2.0"
--- a/rust/vectordb/src/io/object_store.rs
+++ b/rust/vectordb/src/io/object_store.rs
@@ -26,7 +26,7 @@ use futures::{stream::BoxStream, FutureExt, StreamExt};
 use lance::io::object_store::WrappingObjectStore;
 use object_store::{
    path::Path, Error, GetOptions, GetResult, ListResult, MultipartId, ObjectMeta, ObjectStore,
-    Result,
+    PutOptions, PutResult, Result,
 };

 use async_trait::async_trait;
@@ -72,13 +72,28 @@ impl PrimaryOnly for Path {
 /// Note: this object store does not mirror writes to *.manifest files
 #[async_trait]
 impl ObjectStore for MirroringObjectStore {
-    async fn put(&self, location: &Path, bytes: Bytes) -> Result<()> {
+    async fn put(&self, location: &Path, bytes: Bytes) -> Result<PutResult> {
        if location.primary_only() {
            self.primary.put(location, bytes).await
        } else {
            self.secondary.put(location, bytes.clone()).await?;
-            self.primary.put(location, bytes).await?;
-            Ok(())
+            self.primary.put(location, bytes).await
+        }
+    }
+
+    async fn put_opts(
+        &self,
+        location: &Path,
+        bytes: Bytes,
+        options: PutOptions,
+    ) -> Result<PutResult> {
+        if location.primary_only() {
+            self.primary.put_opts(location, bytes, options).await
+        } else {
+            self.secondary
+                .put_opts(location, bytes.clone(), options.clone())
+                .await?;
+            self.primary.put_opts(location, bytes, options).await
        }
    }

@@ -129,8 +144,8 @@ impl ObjectStore for MirroringObjectStore {
        self.primary.delete(location).await
    }

-    async fn list(&self, prefix: Option<&Path>) -> Result<BoxStream<'_, Result<ObjectMeta>>> {
-        self.primary.list(prefix).await
+    fn list(&self, prefix: Option<&Path>) -> BoxStream<'_, Result<ObjectMeta>> {
+        self.primary.list(prefix)
    }

    async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
@@ -359,7 +374,9 @@ mod test {
        assert_eq!(t.count_rows().await.unwrap(), 100);

        let q = t
-            .search(PrimitiveArray::from_iter_values(vec![0.1, 0.1, 0.1, 0.1]))
+            .search(Some(PrimitiveArray::from_iter_values(vec![
+                0.1, 0.1, 0.1, 0.1,
+            ])))
            .limit(10)
            .execute()
            .await
--- a/rust/vectordb/src/query.rs
+++ b/rust/vectordb/src/query.rs
@@ -24,8 +24,9 @@ use crate::error::Result;
 /// A builder for nearest neighbor queries for LanceDB.
 pub struct Query {
    pub dataset: Arc<Dataset>,
-    pub query_vector: Float32Array,
-    pub limit: usize,
+    pub query_vector: Option<Float32Array>,
+    pub column: String,
+    pub limit: Option<usize>,
    pub filter: Option<String>,
    pub select: Option<Vec<String>>,
    pub nprobes: usize,
@@ -46,11 +47,12 @@ impl Query {
    /// # Returns
    ///
    /// * A [Query] object.
-    pub(crate) fn new(dataset: Arc<Dataset>, vector: Float32Array) -> Self {
+    pub(crate) fn new(dataset: Arc<Dataset>, vector: Option<Float32Array>) -> Self {
        Query {
            dataset,
            query_vector: vector,
-            limit: 10,
+            column: crate::table::VECTOR_COLUMN_NAME.to_string(),
+            limit: None,
            nprobes: 20,
            refine_factor: None,
            metric_type: None,
@@ -69,11 +71,13 @@ impl Query {
    pub async fn execute(&self) -> Result<DatasetRecordBatchStream> {
        let mut scanner: Scanner = self.dataset.scan();

-        scanner.nearest(
-            crate::table::VECTOR_COLUMN_NAME,
-            &self.query_vector,
-            self.limit,
-        )?;
+        if let Some(query) = self.query_vector.as_ref() {
+            // If there is a vector query, default to limit=10 if unspecified
+            scanner.nearest(&self.column, query, self.limit.unwrap_or(10))?;
+        } else {
+            // If there is no vector query, it's ok to not have a limit
+            scanner.limit(self.limit.map(|limit| limit as i64), None)?;
+        }
        scanner.nprobs(self.nprobes);
        scanner.use_index(self.use_index);
        scanner.prefilter(self.prefilter);
@@ -85,13 +89,23 @@ impl Query {
        Ok(scanner.try_into_stream().await?)
    }

+    /// Set the column to query
+    ///
+    /// # Arguments
+    ///
+    /// * `column` - The column name
+    pub fn column(mut self, column: &str) -> Query {
+        self.column = column.into();
+        self
+    }
+
    /// Set the maximum number of results to return.
    ///
    /// # Arguments
    ///
    /// * `limit` - The maximum number of results to return.
    pub fn limit(mut self, limit: usize) -> Query {
-        self.limit = limit;
+        self.limit = Some(limit);
        self
    }

@@ -101,7 +115,7 @@ impl Query {
    ///
    /// * `vector` - The vector that will be used for search.
    pub fn query_vector(mut self, query_vector: Float32Array) -> Query {
-        self.query_vector = query_vector;
+        self.query_vector = Some(query_vector);
        self
    }

@@ -174,7 +188,10 @@ mod tests {
    use std::sync::Arc;

    use super::*;
-    use arrow_array::{Float32Array, RecordBatch, RecordBatchIterator, RecordBatchReader};
+    use arrow_array::{
+        cast::AsArray, Float32Array, Int32Array, RecordBatch, RecordBatchIterator,
+        RecordBatchReader,
+    };
    use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema};
    use futures::StreamExt;
    use lance::dataset::Dataset;
@@ -187,7 +204,7 @@ mod tests {
        let batches = make_test_batches();
        let ds = Dataset::write(batches, "memory://foo", None).await.unwrap();

-        let vector = Float32Array::from_iter_values([0.1, 0.2]);
+        let vector = Some(Float32Array::from_iter_values([0.1, 0.2]));
        let query = Query::new(Arc::new(ds), vector.clone());
        assert_eq!(query.query_vector, vector);

@@ -201,8 +218,8 @@ mod tests {
            .metric_type(Some(MetricType::Cosine))
            .refine_factor(Some(999));

-        assert_eq!(query.query_vector, new_vector);
-        assert_eq!(query.limit, 100);
+        assert_eq!(query.query_vector.unwrap(), new_vector);
+        assert_eq!(query.limit.unwrap(), 100);
        assert_eq!(query.nprobes, 1000);
        assert_eq!(query.use_index, true);
        assert_eq!(query.metric_type, Some(MetricType::Cosine));
@@ -214,7 +231,7 @@ mod tests {
        let batches = make_non_empty_batches();
        let ds = Arc::new(Dataset::write(batches, "memory://foo", None).await.unwrap());

-        let vector = Float32Array::from_iter_values([0.1; 4]);
+        let vector = Some(Float32Array::from_iter_values([0.1; 4]));

        let query = Query::new(ds.clone(), vector.clone());
        let result = query
@@ -244,6 +261,27 @@ mod tests {
        }
    }

+    #[tokio::test]
+    async fn test_execute_no_vector() {
+        // test that it's ok to not specify a query vector (just filter / limit)
+        let batches = make_non_empty_batches();
+        let ds = Arc::new(Dataset::write(batches, "memory://foo", None).await.unwrap());
+
+        let query = Query::new(ds.clone(), None);
+        let result = query
+            .filter(Some("id % 2 == 0".to_string()))
+            .execute()
+            .await;
+        let mut stream = result.expect("should have result");
+        // should only have one batch
+        while let Some(batch) = stream.next().await {
+            let b = batch.expect("should be Ok");
+            // cast arr into Int32Array
+            let arr: &Int32Array = b["id"].as_primitive();
+            assert!(arr.iter().all(|x| x.unwrap() % 2 == 0));
+        }
+    }
+
    fn make_non_empty_batches() -> impl RecordBatchReader + Send + 'static {
        let vec = Box::new(RandomVector::new().named("vector".to_string()));
        let id = Box::new(IncrementingInt32::new().named("id".to_string()));
--- a/rust/vectordb/src/table.rs
+++ b/rust/vectordb/src/table.rs
@@ -23,7 +23,7 @@ use lance::dataset::cleanup::RemovalStats;
 use lance::dataset::optimize::{
    compact_files, CompactionMetrics, CompactionOptions, IndexRemapperOptions,
 };
-use lance::dataset::{Dataset, WriteParams};
+use lance::dataset::{Dataset, UpdateBuilder, WriteParams};
 use lance::index::DatasetIndexExt;
 use lance::io::object_store::WrappingObjectStore;
 use std::path::Path;
@@ -308,10 +308,14 @@ impl Table {
    /// # Returns
    ///
    /// * A [Query] object.
-    pub fn search(&self, query_vector: Float32Array) -> Query {
+    pub fn search(&self, query_vector: Option<Float32Array>) -> Query {
        Query::new(self.dataset.clone(), query_vector)
    }

+    pub fn filter(&self, expr: String) -> Query {
+        Query::new(self.dataset.clone(), None).filter(Some(expr))
+    }
+
    /// Returns the number of rows in this Table
    pub async fn count_rows(&self) -> Result<usize> {
        Ok(self.dataset.count_rows().await?)
@@ -338,6 +342,27 @@ impl Table {
        Ok(())
    }

+    pub async fn update(
+        &mut self,
+        predicate: Option<&str>,
+        updates: Vec<(&str, &str)>,
+    ) -> Result<()> {
+        let mut builder = UpdateBuilder::new(self.dataset.clone());
+        if let Some(predicate) = predicate {
+            builder = builder.update_where(predicate)?;
+        }
+
+        for (column, value) in updates {
+            builder = builder.set(column, value)?;
+        }
+
+        let operation = builder.build()?;
+        let new_ds = operation.execute().await?;
+        self.dataset = new_ds;
+
+        Ok(())
+    }
+
    /// Remove old versions of the dataset from disk.
    ///
    /// # Arguments
@@ -413,11 +438,14 @@ mod tests {
    use std::sync::Arc;

    use arrow_array::{
-        Array, FixedSizeListArray, Float32Array, Int32Array, RecordBatch, RecordBatchIterator,
-        RecordBatchReader,
+        Array, BooleanArray, Date32Array, FixedSizeListArray, Float32Array, Float64Array,
+        Int32Array, Int64Array, LargeStringArray, RecordBatch, RecordBatchIterator,
+        RecordBatchReader, StringArray, TimestampMillisecondArray, TimestampNanosecondArray,
+        UInt32Array,
    };
    use arrow_data::ArrayDataBuilder;
-    use arrow_schema::{DataType, Field, Schema};
+    use arrow_schema::{DataType, Field, Schema, TimeUnit};
+    use futures::TryStreamExt;
    use lance::dataset::{Dataset, WriteMode};
    use lance::index::vector::pq::PQBuildParams;
    use lance::io::object_store::{ObjectStoreParams, WrappingObjectStore};
@@ -540,6 +568,272 @@ mod tests {
        assert_eq!(table.name, "test");
    }

+    #[tokio::test]
+    async fn test_update_with_predicate() {
+        let tmp_dir = tempdir().unwrap();
+        let dataset_path = tmp_dir.path().join("test.lance");
+        let uri = dataset_path.to_str().unwrap();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new("name", DataType::Utf8, false),
+        ]));
+
+        let record_batch_iter = RecordBatchIterator::new(
+            vec![RecordBatch::try_new(
+                schema.clone(),
+                vec![
+                    Arc::new(Int32Array::from_iter_values(0..10)),
+                    Arc::new(StringArray::from_iter_values(vec![
+                        "a", "b", "c", "d", "e", "f", "g", "h", "i", "j",
+                    ])),
+                ],
+            )
+            .unwrap()]
+            .into_iter()
+            .map(Ok),
+            schema.clone(),
+        );
+
+        Dataset::write(record_batch_iter, uri, None).await.unwrap();
+        let mut table = Table::open(uri).await.unwrap();
+
+        table
+            .update(Some("id > 5"), vec![("name", "'foo'")])
+            .await
+            .unwrap();
+
+        let ds_after = Dataset::open(uri).await.unwrap();
+        let mut batches = ds_after
+            .scan()
+            .project(&["id", "name"])
+            .unwrap()
+            .try_into_stream()
+            .await
+            .unwrap()
+            .try_collect::<Vec<_>>()
+            .await
+            .unwrap();
+
+        while let Some(batch) = batches.pop() {
+            let ids = batch
+                .column(0)
+                .as_any()
+                .downcast_ref::<Int32Array>()
+                .unwrap()
+                .iter()
+                .collect::<Vec<_>>();
+            let names = batch
+                .column(1)
+                .as_any()
+                .downcast_ref::<StringArray>()
+                .unwrap()
+                .iter()
+                .collect::<Vec<_>>();
+            for (i, name) in names.iter().enumerate() {
+                let id = ids[i].unwrap();
+                let name = name.unwrap();
+                if id > 5 {
+                    assert_eq!(name, "foo");
+                } else {
+                    assert_eq!(name, &format!("{}", (b'a' + id as u8) as char));
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn test_update_all_types() {
+        let tmp_dir = tempdir().unwrap();
+        let dataset_path = tmp_dir.path().join("test.lance");
+        let uri = dataset_path.to_str().unwrap();
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("int32", DataType::Int32, false),
+            Field::new("int64", DataType::Int64, false),
+            Field::new("uint32", DataType::UInt32, false),
+            Field::new("string", DataType::Utf8, false),
+            Field::new("large_string", DataType::LargeUtf8, false),
+            Field::new("float32", DataType::Float32, false),
+            Field::new("float64", DataType::Float64, false),
+            Field::new("bool", DataType::Boolean, false),
+            Field::new("date32", DataType::Date32, false),
+            Field::new(
+                "timestamp_ns",
+                DataType::Timestamp(TimeUnit::Nanosecond, None),
+                false,
+            ),
+            Field::new(
+                "timestamp_ms",
+                DataType::Timestamp(TimeUnit::Millisecond, None),
+                false,
+            ),
+            Field::new(
+                "vec_f32",
+                DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 2),
+                false,
+            ),
+            Field::new(
+                "vec_f64",
+                DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float64, true)), 2),
+                false,
+            ),
+        ]));
+
+        let record_batch_iter = RecordBatchIterator::new(
+            vec![RecordBatch::try_new(
+                schema.clone(),
+                vec![
+                    Arc::new(Int32Array::from_iter_values(0..10)),
+                    Arc::new(Int64Array::from_iter_values(0..10)),
+                    Arc::new(UInt32Array::from_iter_values(0..10)),
+                    Arc::new(StringArray::from_iter_values(vec![
+                        "a", "b", "c", "d", "e", "f", "g", "h", "i", "j",
+                    ])),
+                    Arc::new(LargeStringArray::from_iter_values(vec![
+                        "a", "b", "c", "d", "e", "f", "g", "h", "i", "j",
+                    ])),
+                    Arc::new(Float32Array::from_iter_values(
+                        (0..10).into_iter().map(|i| i as f32),
+                    )),
+                    Arc::new(Float64Array::from_iter_values(
+                        (0..10).into_iter().map(|i| i as f64),
+                    )),
+                    Arc::new(Into::<BooleanArray>::into(vec![
+                        true, false, true, false, true, false, true, false, true, false,
+                    ])),
+                    Arc::new(Date32Array::from_iter_values(0..10)),
+                    Arc::new(TimestampNanosecondArray::from_iter_values(0..10)),
+                    Arc::new(TimestampMillisecondArray::from_iter_values(0..10)),
+                    Arc::new(
+                        create_fixed_size_list(
+                            Float32Array::from_iter_values((0..20).into_iter().map(|i| i as f32)),
+                            2,
+                        )
+                        .unwrap(),
+                    ),
+                    Arc::new(
+                        create_fixed_size_list(
+                            Float64Array::from_iter_values((0..20).into_iter().map(|i| i as f64)),
+                            2,
+                        )
+                        .unwrap(),
+                    ),
+                ],
+            )
+            .unwrap()]
+            .into_iter()
+            .map(Ok),
+            schema.clone(),
+        );
+
+        Dataset::write(record_batch_iter, uri, None).await.unwrap();
+        let mut table = Table::open(uri).await.unwrap();
+
+        // check it can do update for each type
+        let updates: Vec<(&str, &str)> = vec![
+            ("string", "'foo'"),
+            ("large_string", "'large_foo'"),
+            ("int32", "1"),
+            ("int64", "1"),
+            ("uint32", "1"),
+            ("float32", "1.0"),
+            ("float64", "1.0"),
+            ("bool", "true"),
+            ("date32", "1"),
+            ("timestamp_ns", "1"),
+            ("timestamp_ms", "1"),
+            ("vec_f32", "[1.0, 1.0]"),
+            ("vec_f64", "[1.0, 1.0]"),
+        ];
+
+        // for (column, value) in test_cases {
+        table.update(None, updates).await.unwrap();
+
+        let ds_after = Dataset::open(uri).await.unwrap();
+        let mut batches = ds_after
+            .scan()
+            .project(&[
+                "string",
+                "large_string",
+                "int32",
+                "int64",
+                "uint32",
+                "float32",
+                "float64",
+                "bool",
+                "date32",
+                "timestamp_ns",
+                "timestamp_ms",
+                "vec_f32",
+                "vec_f64",
+            ])
+            .unwrap()
+            .try_into_stream()
+            .await
+            .unwrap()
+            .try_collect::<Vec<_>>()
+            .await
+            .unwrap();
+        let batch = batches.pop().unwrap();
+
+        macro_rules! assert_column {
+            ($column:expr, $array_type:ty, $expected:expr) => {
+                let array = $column
+                    .as_any()
+                    .downcast_ref::<$array_type>()
+                    .unwrap()
+                    .iter()
+                    .collect::<Vec<_>>();
+                for v in array {
+                    assert_eq!(v, Some($expected));
+                }
+            };
+        }
+
+        assert_column!(batch.column(0), StringArray, "foo");
+        assert_column!(batch.column(1), LargeStringArray, "large_foo");
+        assert_column!(batch.column(2), Int32Array, 1);
+        assert_column!(batch.column(3), Int64Array, 1);
+        assert_column!(batch.column(4), UInt32Array, 1);
+        assert_column!(batch.column(5), Float32Array, 1.0);
+        assert_column!(batch.column(6), Float64Array, 1.0);
+        assert_column!(batch.column(7), BooleanArray, true);
+        assert_column!(batch.column(8), Date32Array, 1);
+        assert_column!(batch.column(9), TimestampNanosecondArray, 1);
+        assert_column!(batch.column(10), TimestampMillisecondArray, 1);
+
+        let array = batch
+            .column(11)
+            .as_any()
+            .downcast_ref::<FixedSizeListArray>()
+            .unwrap()
+            .iter()
+            .collect::<Vec<_>>();
+        for v in array {
+            let v = v.unwrap();
+            let f32array = v.as_any().downcast_ref::<Float32Array>().unwrap();
+            for v in f32array {
+                assert_eq!(v, Some(1.0));
+            }
+        }
+
+        let array = batch
+            .column(12)
+            .as_any()
+            .downcast_ref::<FixedSizeListArray>()
+            .unwrap()
+            .iter()
+            .collect::<Vec<_>>();
+        for v in array {
+            let v = v.unwrap();
+            let f64array = v.as_any().downcast_ref::<Float64Array>().unwrap();
+            for v in f64array {
+                assert_eq!(v, Some(1.0));
+            }
+        }
+    }
+
    #[tokio::test]
    async fn test_search() {
        let tmp_dir = tempdir().unwrap();
@@ -554,8 +848,8 @@ mod tests {
        let table = Table::open(uri).await.unwrap();

        let vector = Float32Array::from_iter_values([0.1, 0.2]);
-        let query = table.search(vector.clone());
-        assert_eq!(vector, query.query_vector);
+        let query = table.search(Some(vector.clone()));
+        assert_eq!(vector, query.query_vector.unwrap());
    }

    #[derive(Default, Debug)]
Author	SHA1	Message	Date
Chang She	9eca8e7cd1	tests pass; still need catalog	2023-12-21 20:13:08 -08:00
Chang She	587fe6ffc1	almost	2023-12-21 19:45:10 -08:00
Chang She	89c8e5839b	initial changes to enable an in-memory dataset	2023-12-21 08:52:11 -08:00
Aidan	50c20af060	feat: node list tables pagination (#733 )	2023-12-21 11:37:19 -05:00
Chang She	0965d7dd5a	doc(javascript): minor improvement on docs for working with tables (#736 ) Closes #639 Closes #638	2023-12-20 20:05:22 -08:00
Chang She	7bbb2872de	bug(python): fix path handling in windows (#724 ) Use pathlib for local paths so that pathlib can handle the correct separator on windows. Closes #703 --------- Co-authored-by: Will Jones <willjones127@gmail.com>	2023-12-20 15:41:36 -08:00
Will Jones	e81d2975da	chore: add issue templates (#732 ) This PR adds issue templates, which help two recurring issues: * Users forget to tell us whether they are using the Node or Python SDK * Issues don't get appropriate tags This doesn't force the use of the templates. Because we set `blank_issues_enabled: true`, users can still create a custom issue.	2023-12-20 15:15:24 -08:00
Will Jones	2c7f96ba4f	ci: check formatting and clippy (#730 )	2023-12-20 13:37:51 -08:00
Will Jones	f9dd7a5d8a	fix: prevent duplicate data in FTS index (#728 ) This forces the user to replace the whole FTS directory when re-creating the index, prevent duplicate data from being created. Previously, the whole dataset was re-added to the existing index, duplicating existing rows in the index. This (in combination with lancedb/lance#1707) caused #726, since the duplicate data emitted duplicate indices for `take()` and an upstream issue caused those queries to fail. This solution isn't ideal, since it makes the FTS index temporarily unavailable while the index is built. In the future, we should have multiple FTS index directories, which would allow atomic commits of new indexes (as well as multiple indexes for different columns). Fixes #498. Fixes #726. --------- Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>	2023-12-20 13:07:07 -08:00
Will Jones	1d4943688d	upgrade lance to v0.9.1 (#727 ) This brings in some important bugfixes related to take and aarch64 Linux. See changes at: https://github.com/lancedb/lance/releases/tag/v0.9.1	2023-12-20 13:06:54 -08:00
Chang She	7856a94d2c	feat(python): support nested reference for fts (#723 ) https://github.com/lancedb/lance/issues/1739 Support nested field reference in full text search --------- Co-authored-by: Will Jones <willjones127@gmail.com>	2023-12-20 12:28:53 -08:00
Chang She	371d2f979e	feat(python): add option to flatten output in to_pandas (#722 ) Closes https://github.com/lancedb/lance/issues/1738 We add a `flatten` parameter to the signature of `to_pandas`. By default this is None and does nothing. If set to True or -1, then LanceDB will flatten structs before converting to a pandas dataframe. All nested structs are also flattened. If set to any positive integer, then LanceDB will flatten structs up to the specified level of nesting. --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>	2023-12-20 12:23:07 -08:00
Aidan	fff8e399a3	feat: Node create index API (#720 )	2023-12-20 15:22:35 -05:00
Aidan	73e4015797	feat: Node Schema API (#717 )	2023-12-20 12:16:40 -05:00
Lance Release	5142a27482	Updating package-lock.json	2023-12-18 18:15:50 +00:00
Lance Release	81df2a524e	Updating package-lock.json	2023-12-18 17:29:58 +00:00
Lance Release	40638e5515	Bump version: 0.3.11 → 0.4.0	2023-12-18 17:29:47 +00:00
Lance Release	018314a5c1	[python] Bump version: 0.3.6 → 0.4.0	2023-12-18 17:27:26 +00:00
Lei Xu	409eb30ea5	chore: bump lance version to 0.9 (#715 )	2023-12-17 22:11:42 -05:00
Lance Release	ff9872fd44	Updating package-lock.json	2023-12-15 18:25:06 +00:00
Lance Release	a0608044a1	[python] Bump version: 0.3.5 → 0.3.6	2023-12-15 18:20:55 +00:00
Lance Release	2e4ea7d2bc	Updating package-lock.json	2023-12-15 18:01:45 +00:00
Lance Release	57e5695a54	Bump version: 0.3.10 → 0.3.11	2023-12-15 18:01:34 +00:00
Bert	ce58ea7c38	chore: fix package lock (#711 )	2023-12-15 11:49:16 -05:00
Bert	57207eff4a	implement update for remote clients (#706 )	2023-12-15 09:06:40 -05:00
Rob Meng	2d78bff120	feat: pass vector column name to remote backend (#710 ) pass vector column name to remote as well. `vector_column` is already part of `Query` just declearing it as part to `remote.VectorQuery` as well	2023-12-15 00:19:08 -05:00
Rob Meng	7c09b9b9a9	feat: allow custom column name in query (#709 )	2023-12-14 23:29:26 -05:00
Chang She	bd0034a157	feat: support nested pydantic schema (#707 )	2023-12-14 18:20:45 -08:00
Will Jones	144b3b5d83	ci: fix broken npm publication (#704 ) Most recent release failed because `release` depends on `node-macos`, but we renamed `node-macos` to `node-macos-{x86,arm64}`. This fixes that by consolidating them back to a single `node-macos` job, which also has the side effect of making the file shorter.	2023-12-14 12:09:28 -08:00
Lance Release	b6f0a31686	Updating package-lock.json	2023-12-14 19:31:56 +00:00
Lance Release	9ec526f73f	Bump version: 0.3.9 → 0.3.10	2023-12-14 19:31:41 +00:00
Lance Release	600bfd7237	[python] Bump version: 0.3.4 → 0.3.5	2023-12-14 19:31:22 +00:00
Will Jones	d087e7891d	feat(python): add update query support for Python (#654 ) Closes #69 Will not pass until https://github.com/lancedb/lance/pull/1585 is released	2023-12-14 11:28:32 -08:00
Chang She	098e397cf0	feat: LocalTable for vectordb now supports filters without vector search (#693 ) Note this currently the filter/where is only implemented for LocalTable so that it requires an explicit cast to "enable" (see new unit test). The alternative is to add it to the Table interface, but since it's not available on RemoteTable this may cause some user experience issues.	2023-12-13 22:59:01 -08:00
Bert	63ee8fa6a1	Update in Node & Rust (#696 ) Co-authored-by: Will Jones <willjones127@gmail.com>	2023-12-13 14:53:06 -05:00
Ayush Chaurasia	693091db29	chore(python): Reduce posthog event count (#661 ) - Register open_table as event - Because we're dropping 'seach' event currently, changed the name to 'search_table' and introduced throttling - Throttled events will be counted once per time batch so that the user is registered but event count doesn't go up by a lot	2023-12-08 11:00:51 -08:00
Ayush Chaurasia	dca4533dbe	docs: Update roboflow tutorial position (#666 )	2023-12-08 11:00:11 -08:00
QianZhu	f6bbe199dc	Qian/minor fix doc (#695 )	2023-12-08 09:58:53 -08:00
Kaushal Kumar Choudhary	366e522c2b	docs: Add badges (#694 ) adding some badges added a gif to readme for the vectordb repo --------- Co-authored-by: kaushal07wick <kaushalc6@gmail.com>	2023-12-08 20:55:04 +05:30
Chang She	244b6919cc	chore: Use m1 runner for npm publish (#687 ) We had some build issues with npm publish for cross-compiling arm64 macos on an x86 macos runner. Switching to m1 runner for now until someone has time to deal with the feature flags. follow-up tracked here: #688	2023-12-07 15:49:52 -08:00
QianZhu	aca785ff98	saas python sdk doc (#692 ) <img width="256" alt="Screenshot 2023-12-07 at 11 55 41 AM" src="https://github.com/lancedb/lancedb/assets/1305083/259bf234-9b3b-4c5d-af45-c7f3fada2cc7">	2023-12-07 14:47:56 -08:00
Chang She	bbdebf2c38	chore: update package lock (#689 )	2023-12-06 17:14:56 -08:00