Bump version: 0.14.1-beta.0 → 0.14.1-beta.1

perf: eliminate iop in refresh (#1760 )
Closes #1741 If we checkout a version, we need to make a `HEAD` request to get the size of the manifest. The new `checkout_latest()` code path can skip this IOP. This makes the refresh slightly faster.
2025-12-23 13:29:57 +00:00 · 2024-10-23 00:33:40 +00:00 · 2024-10-18 13:40:24 -07:00 · 2024-10-18 10:53:28 -07:00 · 2024-10-17 18:59:18 +00:00 · 2024-10-17 18:59:01 +00:00
61 changed files with 1433 additions and 342 deletions
--- a/.bumpversion.toml
+++ b/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.11.0-beta.1"
+current_version = "0.11.1-beta.0"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -20,13 +20,15 @@ keywords = ["lancedb", "lance", "database", "vector", "search"]
 categories = ["database-implementations"]

 [workspace.dependencies]
-lance = { "version" = "=0.18.2", "features" = ["dynamodb"] }
-lance-index = { "version" = "=0.18.2" }
-lance-linalg = { "version" = "=0.18.2" }
-lance-table = { "version" = "=0.18.2" }
-lance-testing = { "version" = "=0.18.2" }
-lance-datafusion = { "version" = "=0.18.2" }
-lance-encoding = { "version" = "=0.18.2" }
+lance = { "version" = "=0.18.3", "features" = [
+    "dynamodb",
+], git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-index = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-linalg = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-table = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-testing = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-datafusion = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
+lance-encoding = { "version" = "=0.18.3", git = "https://github.com/lancedb/lance.git", tag = "v0.18.3-beta.2" }
 # Note that this one does not include pyarrow
 arrow = { version = "52.2", optional = false }
 arrow-array = "52.2"
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -90,6 +90,9 @@ markdown_extensions:
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg
+  - markdown.extensions.toc:
+      baselevel: 1
+      permalink: ""

 nav:
  - Home:
--- a/docs/src/guides/storage.md
+++ b/docs/src/guides/storage.md
@@ -498,7 +498,7 @@ This can also be done with the ``AWS_ENDPOINT`` and ``AWS_DEFAULT_REGION`` envir

 #### S3 Express

-LanceDB supports [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) endpoints, but requires additional configuration. Also, S3 Express endpoints only support connecting from an EC2 instance within the same region.
+LanceDB supports [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) endpoints, but requires additional infrastructure configuration for the compute service, such as EC2 or Lambda. Please refer to [Networking requirements for S3 Express One Zone](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-networking.html).

 To configure LanceDB to use an S3 Express endpoint, you must set the storage option `s3_express`. The bucket name in your table URI should **include the suffix**.

--- a/docs/src/javascript/README.md
+++ b/docs/src/javascript/README.md
@@ -41,7 +41,6 @@ To build everything fresh:

 ```bash
 npm install
-npm run tsc
 npm run build
 ```

@@ -51,18 +50,6 @@ Then you should be able to run the tests with:
 npm test
 ```

-### Rebuilding Rust library
-
-```bash
-npm run build
-```
-
-### Rebuilding Typescript
-
-```bash
-npm run tsc
-```
-
 ### Fix lints

 To run the linter and have it automatically fix all errors
--- a/docs/src/javascript/classes/DefaultWriteOptions.md
+++ b/docs/src/javascript/classes/DefaultWriteOptions.md
@@ -38,4 +38,4 @@ A [WriteMode](../enums/WriteMode.md) to use on this operation

 #### Defined in

-[index.ts:1019](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1019)
+[index.ts:1359](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1359)
--- a/docs/src/javascript/classes/LocalConnection.md
+++ b/docs/src/javascript/classes/LocalConnection.md
@@ -30,6 +30,7 @@ A connection to a LanceDB database.
 - [dropTable](LocalConnection.md#droptable)
 - [openTable](LocalConnection.md#opentable)
 - [tableNames](LocalConnection.md#tablenames)
+- [withMiddleware](LocalConnection.md#withmiddleware)

 ## Constructors

@@ -46,7 +47,7 @@ A connection to a LanceDB database.

 #### Defined in

-[index.ts:489](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L489)
+[index.ts:739](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L739)

 ## Properties

@@ -56,7 +57,7 @@ A connection to a LanceDB database.

 #### Defined in

-[index.ts:487](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L487)
+[index.ts:737](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L737)

 ___

@@ -74,7 +75,7 @@ ___

 #### Defined in

-[index.ts:486](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L486)
+[index.ts:736](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L736)

 ## Accessors

@@ -92,7 +93,7 @@ ___

 #### Defined in

-[index.ts:494](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L494)
+[index.ts:744](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L744)

 ## Methods

@@ -113,7 +114,7 @@ Creates a new Table, optionally initializing it with new data.
 | Name | Type |
 | :------ | :------ |
 | `name` | `string` \| [`CreateTableOptions`](../interfaces/CreateTableOptions.md)\<`T`\> |
-| `data?` | `Record`\<`string`, `unknown`\>[] |
+| `data?` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] |
 | `optsOrEmbedding?` | [`WriteOptions`](../interfaces/WriteOptions.md) \| [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)\<`T`\> |
 | `opt?` | [`WriteOptions`](../interfaces/WriteOptions.md) |

@@ -127,7 +128,7 @@ Creates a new Table, optionally initializing it with new data.

 #### Defined in

-[index.ts:542](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L542)
+[index.ts:788](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L788)

 ___

@@ -158,7 +159,7 @@ ___

 #### Defined in

-[index.ts:576](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L576)
+[index.ts:822](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L822)

 ___

@@ -184,7 +185,7 @@ Drop an existing table.

 #### Defined in

-[index.ts:630](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L630)
+[index.ts:876](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L876)

 ___

@@ -210,7 +211,7 @@ Open a table in the database.

 #### Defined in

-[index.ts:510](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L510)
+[index.ts:760](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L760)

 ▸ **openTable**\<`T`\>(`name`, `embeddings`): `Promise`\<[`Table`](../interfaces/Table.md)\<`T`\>\>

@@ -239,7 +240,7 @@ Connection.openTable

 #### Defined in

-[index.ts:518](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L518)
+[index.ts:768](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L768)

 ▸ **openTable**\<`T`\>(`name`, `embeddings?`): `Promise`\<[`Table`](../interfaces/Table.md)\<`T`\>\>

@@ -266,7 +267,7 @@ Connection.openTable

 #### Defined in

-[index.ts:522](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L522)
+[index.ts:772](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L772)

 ___

@@ -286,4 +287,36 @@ Get the names of all tables in the database.

 #### Defined in

-[index.ts:501](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L501)
+[index.ts:751](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L751)
+
+___
+
+### withMiddleware
+
+▸ **withMiddleware**(`middleware`): [`Connection`](../interfaces/Connection.md)
+
+Instrument the behavior of this Connection with middleware.
+
+The middleware will be called in the order they are added.
+
+Currently this functionality is only supported for remote Connections.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `middleware` | `HttpMiddleware` |
+
+#### Returns
+
+[`Connection`](../interfaces/Connection.md)
+
+- this Connection instrumented by the passed middleware
+
+#### Implementation of
+
+[Connection](../interfaces/Connection.md).[withMiddleware](../interfaces/Connection.md#withmiddleware)
+
+#### Defined in
+
+[index.ts:880](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L880)
--- a/docs/src/javascript/classes/LocalTable.md
+++ b/docs/src/javascript/classes/LocalTable.md
@@ -37,6 +37,8 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector
 ### Methods

 - [add](LocalTable.md#add)
+- [addColumns](LocalTable.md#addcolumns)
+- [alterColumns](LocalTable.md#altercolumns)
 - [checkElectron](LocalTable.md#checkelectron)
 - [cleanupOldVersions](LocalTable.md#cleanupoldversions)
 - [compactFiles](LocalTable.md#compactfiles)
@@ -44,13 +46,16 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector
 - [createIndex](LocalTable.md#createindex)
 - [createScalarIndex](LocalTable.md#createscalarindex)
 - [delete](LocalTable.md#delete)
+- [dropColumns](LocalTable.md#dropcolumns)
 - [filter](LocalTable.md#filter)
 - [getSchema](LocalTable.md#getschema)
 - [indexStats](LocalTable.md#indexstats)
 - [listIndices](LocalTable.md#listindices)
+- [mergeInsert](LocalTable.md#mergeinsert)
 - [overwrite](LocalTable.md#overwrite)
 - [search](LocalTable.md#search)
 - [update](LocalTable.md#update)
+- [withMiddleware](LocalTable.md#withmiddleware)

 ## Constructors

@@ -74,7 +79,7 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector

 #### Defined in

-[index.ts:642](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L642)
+[index.ts:892](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L892)

 • **new LocalTable**\<`T`\>(`tbl`, `name`, `options`, `embeddings`)

@@ -95,7 +100,7 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector

 #### Defined in

-[index.ts:649](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L649)
+[index.ts:899](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L899)

 ## Properties

@@ -105,7 +110,7 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector

 #### Defined in

-[index.ts:639](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L639)
+[index.ts:889](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L889)

 ___

@@ -115,7 +120,7 @@ ___

 #### Defined in

-[index.ts:638](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L638)
+[index.ts:888](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L888)

 ___

@@ -125,7 +130,7 @@ ___

 #### Defined in

-[index.ts:637](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L637)
+[index.ts:887](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L887)

 ___

@@ -143,7 +148,7 @@ ___

 #### Defined in

-[index.ts:640](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L640)
+[index.ts:890](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L890)

 ___

@@ -153,7 +158,7 @@ ___

 #### Defined in

-[index.ts:636](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L636)
+[index.ts:886](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L886)

 ___

@@ -179,7 +184,7 @@ Creates a filter query to find all rows matching the specified criteria

 #### Defined in

-[index.ts:688](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L688)
+[index.ts:938](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L938)

 ## Accessors

@@ -197,7 +202,7 @@ Creates a filter query to find all rows matching the specified criteria

 #### Defined in

-[index.ts:668](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L668)
+[index.ts:918](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L918)

 ___

@@ -215,7 +220,7 @@ ___

 #### Defined in

-[index.ts:849](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L849)
+[index.ts:1171](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1171)

 ## Methods

@@ -229,7 +234,7 @@ Insert records into this Table.

 | Name | Type | Description |
 | :------ | :------ | :------ |
-| `data` | `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |

 #### Returns

@@ -243,7 +248,59 @@ The number of rows added to the table

 #### Defined in

-[index.ts:696](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L696)
+[index.ts:946](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L946)
+
+___
+
+### addColumns
+
+▸ **addColumns**(`newColumnTransforms`): `Promise`\<`void`\>
+
+Add new columns with defined values.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `newColumnTransforms` | \{ `name`: `string` ; `valueSql`: `string`  }[] | pairs of column names and the SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns in the table. |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Implementation of
+
+[Table](../interfaces/Table.md).[addColumns](../interfaces/Table.md#addcolumns)
+
+#### Defined in
+
+[index.ts:1195](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1195)
+
+___
+
+### alterColumns
+
+▸ **alterColumns**(`columnAlterations`): `Promise`\<`void`\>
+
+Alter the name or nullability of columns.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `columnAlterations` | [`ColumnAlteration`](../interfaces/ColumnAlteration.md)[] | One or more alterations to apply to columns. |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Implementation of
+
+[Table](../interfaces/Table.md).[alterColumns](../interfaces/Table.md#altercolumns)
+
+#### Defined in
+
+[index.ts:1201](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1201)

 ___

@@ -257,7 +314,7 @@ ___

 #### Defined in

-[index.ts:861](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L861)
+[index.ts:1183](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1183)

 ___

@@ -280,7 +337,7 @@ Clean up old versions of the table, freeing disk space.

 #### Defined in

-[index.ts:808](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L808)
+[index.ts:1130](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1130)

 ___

@@ -307,16 +364,22 @@ Metrics about the compaction operation.

 #### Defined in

-[index.ts:831](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L831)
+[index.ts:1153](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1153)

 ___

 ### countRows

-▸ **countRows**(): `Promise`\<`number`\>
+▸ **countRows**(`filter?`): `Promise`\<`number`\>

 Returns the number of rows in this table.

+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `filter?` | `string` |
+
 #### Returns

 `Promise`\<`number`\>
@@ -327,7 +390,7 @@ Returns the number of rows in this table.

 #### Defined in

-[index.ts:749](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L749)
+[index.ts:1021](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1021)

 ___

@@ -357,13 +420,13 @@ VectorIndexParams.

 #### Defined in

-[index.ts:734](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L734)
+[index.ts:1003](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1003)

 ___

 ### createScalarIndex

-▸ **createScalarIndex**(`column`, `replace`): `Promise`\<`void`\>
+▸ **createScalarIndex**(`column`, `replace?`): `Promise`\<`void`\>

 Create a scalar index on this Table for the given column

@@ -372,7 +435,7 @@ Create a scalar index on this Table for the given column
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `column` | `string` | The column to index |
-| `replace` | `boolean` | If false, fail if an index already exists on the column Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column `my_col` has a scalar index: ```ts const con = await lancedb.connect('./.lancedb'); const table = await con.openTable('images'); const results = await table.where('my_col = 7').execute(); ``` Scalar indices can also speed up scans containing a vector search and a prefilter: ```ts const con = await lancedb.connect('././lancedb'); const table = await con.openTable('images'); const results = await table.search([1.0, 2.0]).where('my_col != 7').prefilter(true); ``` Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. `my_col BETWEEN 0 AND 100`), and set membership (e.g. `my_col IN (0, 1, 2)`) Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. `my_col < 0 AND other_col> 100`) Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column `not_indexed` does not have a scalar index then the filter `my_col = 0 OR not_indexed = 1` will not be able to use any scalar index on `my_col`. |
+| `replace?` | `boolean` | If false, fail if an index already exists on the column it is always set to true for remote connections Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column `my_col` has a scalar index: ```ts const con = await lancedb.connect('./.lancedb'); const table = await con.openTable('images'); const results = await table.where('my_col = 7').execute(); ``` Scalar indices can also speed up scans containing a vector search and a prefilter: ```ts const con = await lancedb.connect('././lancedb'); const table = await con.openTable('images'); const results = await table.search([1.0, 2.0]).where('my_col != 7').prefilter(true); ``` Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. `my_col BETWEEN 0 AND 100`), and set membership (e.g. `my_col IN (0, 1, 2)`) Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. `my_col < 0 AND other_col> 100`) Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column `not_indexed` does not have a scalar index then the filter `my_col = 0 OR not_indexed = 1` will not be able to use any scalar index on `my_col`. |

 #### Returns

@@ -392,7 +455,7 @@ await table.createScalarIndex('my_col')

 #### Defined in

-[index.ts:742](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L742)
+[index.ts:1011](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1011)

 ___

@@ -418,7 +481,38 @@ Delete rows from this table.

 #### Defined in

-[index.ts:758](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L758)
+[index.ts:1030](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1030)
+
+___
+
+### dropColumns
+
+▸ **dropColumns**(`columnNames`): `Promise`\<`void`\>
+
+Drop one or more columns from the dataset
+
+This is a metadata-only operation and does not remove the data from the
+underlying storage. In order to remove the data, you must subsequently
+call ``compact_files`` to rewrite the data without the removed columns and
+then call ``cleanup_files`` to remove the old files.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `columnNames` | `string`[] | The names of the columns to drop. These can be nested column references (e.g. "a.b.c") or top-level column names (e.g. "a"). |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Implementation of
+
+[Table](../interfaces/Table.md).[dropColumns](../interfaces/Table.md#dropcolumns)
+
+#### Defined in
+
+[index.ts:1205](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1205)

 ___

@@ -438,9 +532,13 @@ Creates a filter query to find all rows matching the specified criteria

 [`Query`](Query.md)\<`T`\>

+#### Implementation of
+
+[Table](../interfaces/Table.md).[filter](../interfaces/Table.md#filter)
+
 #### Defined in

-[index.ts:684](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L684)
+[index.ts:934](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L934)

 ___

@@ -454,13 +552,13 @@ ___

 #### Defined in

-[index.ts:854](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L854)
+[index.ts:1176](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1176)

 ___

 ### indexStats

-▸ **indexStats**(`indexUuid`): `Promise`\<[`IndexStats`](../interfaces/IndexStats.md)\>
+▸ **indexStats**(`indexName`): `Promise`\<[`IndexStats`](../interfaces/IndexStats.md)\>

 Get statistics about an index.

@@ -468,7 +566,7 @@ Get statistics about an index.

 | Name | Type |
 | :------ | :------ |
-| `indexUuid` | `string` |
+| `indexName` | `string` |

 #### Returns

@@ -480,7 +578,7 @@ Get statistics about an index.

 #### Defined in

-[index.ts:845](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L845)
+[index.ts:1167](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1167)

 ___

@@ -500,7 +598,57 @@ List the indicies on this table.

 #### Defined in

-[index.ts:841](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L841)
+[index.ts:1163](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1163)
+
+___
+
+### mergeInsert
+
+▸ **mergeInsert**(`on`, `data`, `args`): `Promise`\<`void`\>
+
+Runs a "merge insert" operation on the table
+
+This operation can add rows, update rows, and remove rows all in a single
+transaction. It is a very generic tool that can be used to create
+behaviors like "insert if not exists", "update or insert (i.e. upsert)",
+or even replace a portion of existing data with new data (e.g. replace
+all data where month="january")
+
+The merge insert operation works by combining new data from a
+**source table** with existing data in a **target table** by using a
+join.  There are three categories of records.
+
+"Matched" records are records that exist in both the source table and
+the target table. "Not matched" records exist only in the source table
+(e.g. these are new data) "Not matched by source" records exist only
+in the target table (this is old data)
+
+The MergeInsertArgs can be used to customize what should happen for
+each category of data.
+
+Please note that the data may appear to be reordered as part of this
+operation.  This is because updated rows will be deleted from the
+dataset and then reinserted at the end with the new values.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `on` | `string` | a column to join on. This is how records from the source table and target table are matched. |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | the new data to insert |
+| `args` | [`MergeInsertArgs`](../interfaces/MergeInsertArgs.md) | parameters controlling how the operation should behave |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Implementation of
+
+[Table](../interfaces/Table.md).[mergeInsert](../interfaces/Table.md#mergeinsert)
+
+#### Defined in
+
+[index.ts:1065](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1065)

 ___

@@ -514,7 +662,7 @@ Insert records into this Table, replacing its contents.

 | Name | Type | Description |
 | :------ | :------ | :------ |
-| `data` | `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |

 #### Returns

@@ -528,7 +676,7 @@ The number of rows added to the table

 #### Defined in

-[index.ts:716](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L716)
+[index.ts:977](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L977)

 ___

@@ -554,7 +702,7 @@ Creates a search query to find the nearest neighbors of the given search term

 #### Defined in

-[index.ts:676](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L676)
+[index.ts:926](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L926)

 ___

@@ -580,4 +728,36 @@ Update rows in this table.

 #### Defined in

-[index.ts:771](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L771)
+[index.ts:1043](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1043)
+
+___
+
+### withMiddleware
+
+▸ **withMiddleware**(`middleware`): [`Table`](../interfaces/Table.md)\<`T`\>
+
+Instrument the behavior of this Table with middleware.
+
+The middleware will be called in the order they are added.
+
+Currently this functionality is only supported for remote tables.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `middleware` | `HttpMiddleware` |
+
+#### Returns
+
+[`Table`](../interfaces/Table.md)\<`T`\>
+
+- this Table instrumented by the passed middleware
+
+#### Implementation of
+
+[Table](../interfaces/Table.md).[withMiddleware](../interfaces/Table.md#withmiddleware)
+
+#### Defined in
+
+[index.ts:1209](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1209)
--- a/docs/src/javascript/classes/MakeArrowTableOptions.md
+++ b/docs/src/javascript/classes/MakeArrowTableOptions.md
@@ -0,0 +1,82 @@
+[vectordb](../README.md) / [Exports](../modules.md) / MakeArrowTableOptions
+
+# Class: MakeArrowTableOptions
+
+Options to control the makeArrowTable call.
+
+## Table of contents
+
+### Constructors
+
+- [constructor](MakeArrowTableOptions.md#constructor)
+
+### Properties
+
+- [dictionaryEncodeStrings](MakeArrowTableOptions.md#dictionaryencodestrings)
+- [embeddings](MakeArrowTableOptions.md#embeddings)
+- [schema](MakeArrowTableOptions.md#schema)
+- [vectorColumns](MakeArrowTableOptions.md#vectorcolumns)
+
+## Constructors
+
+### constructor
+
+• **new MakeArrowTableOptions**(`values?`)
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `values?` | `Partial`\<[`MakeArrowTableOptions`](MakeArrowTableOptions.md)\> |
+
+#### Defined in
+
+[arrow.ts:98](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L98)
+
+## Properties
+
+### dictionaryEncodeStrings
+
+• **dictionaryEncodeStrings**: `boolean` = `false`
+
+If true then string columns will be encoded with dictionary encoding
+
+Set this to true if your string columns tend to repeat the same values
+often.  For more precise control use the `schema` property to specify the
+data type for individual columns.
+
+If `schema` is provided then this property is ignored.
+
+#### Defined in
+
+[arrow.ts:96](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L96)
+
+___
+
+### embeddings
+
+• `Optional` **embeddings**: [`EmbeddingFunction`](../interfaces/EmbeddingFunction.md)\<`any`\>
+
+#### Defined in
+
+[arrow.ts:85](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L85)
+
+___
+
+### schema
+
+• `Optional` **schema**: `Schema`\<`any`\>
+
+#### Defined in
+
+[arrow.ts:63](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L63)
+
+___
+
+### vectorColumns
+
+• **vectorColumns**: `Record`\<`string`, `VectorColumnOptions`\>
+
+#### Defined in
+
+[arrow.ts:81](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L81)
--- a/docs/src/javascript/classes/OpenAIEmbeddingFunction.md
+++ b/docs/src/javascript/classes/OpenAIEmbeddingFunction.md
@@ -40,7 +40,7 @@ An embedding function that automatically creates vector representation for a giv

 #### Defined in

-[embedding/openai.ts:21](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/openai.ts#L21)
+[embedding/openai.ts:22](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/openai.ts#L22)

 ## Properties

@@ -50,17 +50,17 @@ An embedding function that automatically creates vector representation for a giv

 #### Defined in

-[embedding/openai.ts:19](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/openai.ts#L19)
+[embedding/openai.ts:20](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/openai.ts#L20)

 ___

 ### \_openai

-• `Private` `Readonly` **\_openai**: `any`
+• `Private` `Readonly` **\_openai**: `OpenAI`

 #### Defined in

-[embedding/openai.ts:18](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/openai.ts#L18)
+[embedding/openai.ts:19](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/openai.ts#L19)

 ___

@@ -76,7 +76,7 @@ The name of the column that will be used as input for the Embedding Function.

 #### Defined in

-[embedding/openai.ts:50](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/openai.ts#L50)
+[embedding/openai.ts:56](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/openai.ts#L56)

 ## Methods

@@ -102,4 +102,4 @@ Creates a vector representation for the given values.

 #### Defined in

-[embedding/openai.ts:38](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/openai.ts#L38)
+[embedding/openai.ts:43](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/openai.ts#L43)
--- a/docs/src/javascript/classes/Query.md
+++ b/docs/src/javascript/classes/Query.md
@@ -19,6 +19,7 @@ A builder for nearest neighbor queries for LanceDB.
 ### Properties

 - [\_embeddings](Query.md#_embeddings)
+- [\_fastSearch](Query.md#_fastsearch)
 - [\_filter](Query.md#_filter)
 - [\_limit](Query.md#_limit)
 - [\_metricType](Query.md#_metrictype)
@@ -34,6 +35,7 @@ A builder for nearest neighbor queries for LanceDB.
 ### Methods

 - [execute](Query.md#execute)
+- [fastSearch](Query.md#fastsearch)
 - [filter](Query.md#filter)
 - [isElectron](Query.md#iselectron)
 - [limit](Query.md#limit)
@@ -65,7 +67,7 @@ A builder for nearest neighbor queries for LanceDB.

 #### Defined in

-[query.ts:38](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L38)
+[query.ts:39](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L39)

 ## Properties

@@ -75,7 +77,17 @@ A builder for nearest neighbor queries for LanceDB.

 #### Defined in

-[query.ts:36](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L36)
+[query.ts:37](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L37)
+
+___
+
+### \_fastSearch
+
+• `Private` **\_fastSearch**: `boolean`
+
+#### Defined in
+
+[query.ts:36](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L36)

 ___

@@ -85,7 +97,7 @@ ___

 #### Defined in

-[query.ts:33](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L33)
+[query.ts:33](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L33)

 ___

@@ -95,7 +107,7 @@ ___

 #### Defined in

-[query.ts:29](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L29)
+[query.ts:29](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L29)

 ___

@@ -105,7 +117,7 @@ ___

 #### Defined in

-[query.ts:34](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L34)
+[query.ts:34](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L34)

 ___

@@ -115,7 +127,7 @@ ___

 #### Defined in

-[query.ts:31](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L31)
+[query.ts:31](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L31)

 ___

@@ -125,7 +137,7 @@ ___

 #### Defined in

-[query.ts:35](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L35)
+[query.ts:35](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L35)

 ___

@@ -135,7 +147,7 @@ ___

 #### Defined in

-[query.ts:26](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L26)
+[query.ts:26](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L26)

 ___

@@ -145,7 +157,7 @@ ___

 #### Defined in

-[query.ts:28](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L28)
+[query.ts:28](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L28)

 ___

@@ -155,7 +167,7 @@ ___

 #### Defined in

-[query.ts:30](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L30)
+[query.ts:30](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L30)

 ___

@@ -165,7 +177,7 @@ ___

 #### Defined in

-[query.ts:32](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L32)
+[query.ts:32](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L32)

 ___

@@ -175,7 +187,7 @@ ___

 #### Defined in

-[query.ts:27](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L27)
+[query.ts:27](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L27)

 ___

@@ -201,7 +213,7 @@ A filter statement to be applied to this query.

 #### Defined in

-[query.ts:87](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L87)
+[query.ts:90](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L90)

 ## Methods

@@ -223,7 +235,30 @@ Execute the query and return the results as an Array of Objects

 #### Defined in

-[query.ts:115](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L115)
+[query.ts:127](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L127)
+
+___
+
+### fastSearch
+
+▸ **fastSearch**(`value`): [`Query`](Query.md)\<`T`\>
+
+Skip searching un-indexed data. This can make search faster, but will miss
+any data that is not yet indexed.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `value` | `boolean` |
+
+#### Returns
+
+[`Query`](Query.md)\<`T`\>
+
+#### Defined in
+
+[query.ts:119](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L119)

 ___

@@ -245,7 +280,7 @@ A filter statement to be applied to this query.

 #### Defined in

-[query.ts:82](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L82)
+[query.ts:85](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L85)

 ___

@@ -259,7 +294,7 @@ ___

 #### Defined in

-[query.ts:142](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L142)
+[query.ts:155](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L155)

 ___

@@ -268,6 +303,7 @@ ___
 ▸ **limit**(`value`): [`Query`](Query.md)\<`T`\>

 Sets the number of results that will be returned
+default value is 10

 #### Parameters

@@ -281,7 +317,7 @@ Sets the number of results that will be returned

 #### Defined in

-[query.ts:55](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L55)
+[query.ts:58](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L58)

 ___

@@ -307,7 +343,7 @@ MetricType for the different options

 #### Defined in

-[query.ts:102](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L102)
+[query.ts:105](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L105)

 ___

@@ -329,7 +365,7 @@ The number of probes used. A higher number makes search more accurate but also s

 #### Defined in

-[query.ts:73](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L73)
+[query.ts:76](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L76)

 ___

@@ -349,7 +385,7 @@ ___

 #### Defined in

-[query.ts:107](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L107)
+[query.ts:110](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L110)

 ___

@@ -371,7 +407,7 @@ Refine the results by reading extra elements and re-ranking them in memory.

 #### Defined in

-[query.ts:64](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L64)
+[query.ts:67](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L67)

 ___

@@ -393,4 +429,4 @@ Return only the specified columns.

 #### Defined in

-[query.ts:93](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/query.ts#L93)
+[query.ts:96](https://github.com/lancedb/lancedb/blob/92179835/node/src/query.ts#L96)
--- a/docs/src/javascript/enums/IndexStatus.md
+++ b/docs/src/javascript/enums/IndexStatus.md
@@ -0,0 +1,52 @@
+[vectordb](../README.md) / [Exports](../modules.md) / IndexStatus
+
+# Enumeration: IndexStatus
+
+## Table of contents
+
+### Enumeration Members
+
+- [Done](IndexStatus.md#done)
+- [Failed](IndexStatus.md#failed)
+- [Indexing](IndexStatus.md#indexing)
+- [Pending](IndexStatus.md#pending)
+
+## Enumeration Members
+
+### Done
+
+• **Done** = ``"done"``
+
+#### Defined in
+
+[index.ts:713](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L713)
+
+___
+
+### Failed
+
+• **Failed** = ``"failed"``
+
+#### Defined in
+
+[index.ts:714](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L714)
+
+___
+
+### Indexing
+
+• **Indexing** = ``"indexing"``
+
+#### Defined in
+
+[index.ts:712](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L712)
+
+___
+
+### Pending
+
+• **Pending** = ``"pending"``
+
+#### Defined in
+
+[index.ts:711](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L711)
--- a/docs/src/javascript/enums/MetricType.md
+++ b/docs/src/javascript/enums/MetricType.md
@@ -22,7 +22,7 @@ Cosine distance

 #### Defined in

-[index.ts:1041](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1041)
+[index.ts:1381](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1381)

 ___

@@ -34,7 +34,7 @@ Dot product

 #### Defined in

-[index.ts:1046](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1046)
+[index.ts:1386](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1386)

 ___

@@ -46,4 +46,4 @@ Euclidean distance

 #### Defined in

-[index.ts:1036](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1036)
+[index.ts:1376](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1376)
--- a/docs/src/javascript/enums/WriteMode.md
+++ b/docs/src/javascript/enums/WriteMode.md
@@ -22,7 +22,7 @@ Append new data to the table.

 #### Defined in

-[index.ts:1007](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1007)
+[index.ts:1347](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1347)

 ___

@@ -34,7 +34,7 @@ Create a new [Table](../interfaces/Table.md).

 #### Defined in

-[index.ts:1003](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1003)
+[index.ts:1343](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1343)

 ___

@@ -46,4 +46,4 @@ Overwrite the existing [Table](../interfaces/Table.md) if presented.

 #### Defined in

-[index.ts:1005](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1005)
+[index.ts:1345](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1345)
--- a/docs/src/javascript/interfaces/AwsCredentials.md
+++ b/docs/src/javascript/interfaces/AwsCredentials.md
@@ -18,7 +18,7 @@

 #### Defined in

-[index.ts:54](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L54)
+[index.ts:68](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L68)

 ___

@@ -28,7 +28,7 @@ ___

 #### Defined in

-[index.ts:56](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L56)
+[index.ts:70](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L70)

 ___

@@ -38,4 +38,4 @@ ___

 #### Defined in

-[index.ts:58](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L58)
+[index.ts:72](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L72)
--- a/docs/src/javascript/interfaces/CleanupStats.md
+++ b/docs/src/javascript/interfaces/CleanupStats.md
@@ -19,7 +19,7 @@ The number of bytes removed from disk.

 #### Defined in

-[index.ts:878](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L878)
+[index.ts:1218](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1218)

 ___

@@ -31,4 +31,4 @@ The number of old table versions removed.

 #### Defined in

-[index.ts:882](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L882)
+[index.ts:1222](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1222)
--- a/docs/src/javascript/interfaces/ColumnAlteration.md
+++ b/docs/src/javascript/interfaces/ColumnAlteration.md
@@ -0,0 +1,53 @@
+[vectordb](../README.md) / [Exports](../modules.md) / ColumnAlteration
+
+# Interface: ColumnAlteration
+
+A definition of a column alteration. The alteration changes the column at
+`path` to have the new name `name`, to be nullable if `nullable` is true,
+and to have the data type `data_type`. At least one of `rename` or `nullable`
+must be provided.
+
+## Table of contents
+
+### Properties
+
+- [nullable](ColumnAlteration.md#nullable)
+- [path](ColumnAlteration.md#path)
+- [rename](ColumnAlteration.md#rename)
+
+## Properties
+
+### nullable
+
+• `Optional` **nullable**: `boolean`
+
+Set the new nullability. Note that a nullable column cannot be made non-nullable.
+
+#### Defined in
+
+[index.ts:638](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L638)
+
+___
+
+### path
+
+• **path**: `string`
+
+The path to the column to alter. This is a dot-separated path to the column.
+If it is a top-level column then it is just the name of the column. If it is
+a nested column then it is the path to the column, e.g. "a.b.c" for a column
+`c` nested inside a column `b` nested inside a column `a`.
+
+#### Defined in
+
+[index.ts:633](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L633)
+
+___
+
+### rename
+
+• `Optional` **rename**: `string`
+
+#### Defined in
+
+[index.ts:634](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L634)
--- a/docs/src/javascript/interfaces/CompactionMetrics.md
+++ b/docs/src/javascript/interfaces/CompactionMetrics.md
@@ -22,7 +22,7 @@ fragments added.

 #### Defined in

-[index.ts:933](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L933)
+[index.ts:1273](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1273)

 ___

@@ -35,7 +35,7 @@ file.

 #### Defined in

-[index.ts:928](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L928)
+[index.ts:1268](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1268)

 ___

@@ -47,7 +47,7 @@ The number of new fragments that were created.

 #### Defined in

-[index.ts:923](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L923)
+[index.ts:1263](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1263)

 ___

@@ -59,4 +59,4 @@ The number of fragments that were removed.

 #### Defined in

-[index.ts:919](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L919)
+[index.ts:1259](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1259)
--- a/docs/src/javascript/interfaces/CompactionOptions.md
+++ b/docs/src/javascript/interfaces/CompactionOptions.md
@@ -24,7 +24,7 @@ Default is true.

 #### Defined in

-[index.ts:901](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L901)
+[index.ts:1241](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1241)

 ___

@@ -38,7 +38,7 @@ the deleted rows. Default is 10%.

 #### Defined in

-[index.ts:907](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L907)
+[index.ts:1247](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1247)

 ___

@@ -46,11 +46,11 @@ ___

 • `Optional` **maxRowsPerGroup**: `number`

-The maximum number of rows per group. Defaults to 1024.
+The maximum number of T per group. Defaults to 1024.

 #### Defined in

-[index.ts:895](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L895)
+[index.ts:1235](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1235)

 ___

@@ -63,7 +63,7 @@ the number of cores on the machine.

 #### Defined in

-[index.ts:912](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L912)
+[index.ts:1252](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1252)

 ___

@@ -77,4 +77,4 @@ Defaults to 1024 * 1024.

 #### Defined in

-[index.ts:891](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L891)
+[index.ts:1231](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1231)
--- a/docs/src/javascript/interfaces/Connection.md
+++ b/docs/src/javascript/interfaces/Connection.md
@@ -22,6 +22,7 @@ Connection could be local against filesystem or remote against a server.
 - [dropTable](Connection.md#droptable)
 - [openTable](Connection.md#opentable)
 - [tableNames](Connection.md#tablenames)
+- [withMiddleware](Connection.md#withmiddleware)

 ## Properties

@@ -31,7 +32,7 @@ Connection could be local against filesystem or remote against a server.

 #### Defined in

-[index.ts:183](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L183)
+[index.ts:261](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L261)

 ## Methods

@@ -59,7 +60,7 @@ Creates a new Table, optionally initializing it with new data.

 #### Defined in

-[index.ts:207](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L207)
+[index.ts:285](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L285)

 ▸ **createTable**(`name`, `data`): `Promise`\<[`Table`](Table.md)\<`number`[]\>\>

@@ -70,7 +71,7 @@ Creates a new Table and initialize it with new data.
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `name` | `string` | The name of the table. |
-| `data` | `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |

 #### Returns

@@ -78,7 +79,7 @@ Creates a new Table and initialize it with new data.

 #### Defined in

-[index.ts:221](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L221)
+[index.ts:299](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L299)

 ▸ **createTable**(`name`, `data`, `options`): `Promise`\<[`Table`](Table.md)\<`number`[]\>\>

@@ -89,7 +90,7 @@ Creates a new Table and initialize it with new data.
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `name` | `string` | The name of the table. |
-| `data` | `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
 | `options` | [`WriteOptions`](WriteOptions.md) | The write options to use when creating the table. |

 #### Returns
@@ -98,7 +99,7 @@ Creates a new Table and initialize it with new data.

 #### Defined in

-[index.ts:233](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L233)
+[index.ts:311](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L311)

 ▸ **createTable**\<`T`\>(`name`, `data`, `embeddings`): `Promise`\<[`Table`](Table.md)\<`T`\>\>

@@ -115,7 +116,7 @@ Creates a new Table and initialize it with new data.
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `name` | `string` | The name of the table. |
-| `data` | `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
 | `embeddings` | [`EmbeddingFunction`](EmbeddingFunction.md)\<`T`\> | An embedding function to use on this table |

 #### Returns
@@ -124,7 +125,7 @@ Creates a new Table and initialize it with new data.

 #### Defined in

-[index.ts:246](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L246)
+[index.ts:324](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L324)

 ▸ **createTable**\<`T`\>(`name`, `data`, `embeddings`, `options`): `Promise`\<[`Table`](Table.md)\<`T`\>\>

@@ -141,7 +142,7 @@ Creates a new Table and initialize it with new data.
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `name` | `string` | The name of the table. |
-| `data` | `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Non-empty Array of Records to be inserted into the table |
 | `embeddings` | [`EmbeddingFunction`](EmbeddingFunction.md)\<`T`\> | An embedding function to use on this table |
 | `options` | [`WriteOptions`](WriteOptions.md) | The write options to use when creating the table. |

@@ -151,7 +152,7 @@ Creates a new Table and initialize it with new data.

 #### Defined in

-[index.ts:259](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L259)
+[index.ts:337](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L337)

 ___

@@ -173,7 +174,7 @@ Drop an existing table.

 #### Defined in

-[index.ts:270](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L270)
+[index.ts:348](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L348)

 ___

@@ -202,7 +203,7 @@ Open a table in the database.

 #### Defined in

-[index.ts:193](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L193)
+[index.ts:271](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L271)

 ___

@@ -216,4 +217,32 @@ ___

 #### Defined in

-[index.ts:185](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L185)
+[index.ts:263](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L263)
+
+___
+
+### withMiddleware
+
+▸ **withMiddleware**(`middleware`): [`Connection`](Connection.md)
+
+Instrument the behavior of this Connection with middleware.
+
+The middleware will be called in the order they are added.
+
+Currently this functionality is only supported for remote Connections.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `middleware` | `HttpMiddleware` |
+
+#### Returns
+
+[`Connection`](Connection.md)
+
+- this Connection instrumented by the passed middleware
+
+#### Defined in
+
+[index.ts:360](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L360)
--- a/docs/src/javascript/interfaces/ConnectionOptions.md
+++ b/docs/src/javascript/interfaces/ConnectionOptions.md
@@ -10,7 +10,10 @@
 - [awsCredentials](ConnectionOptions.md#awscredentials)
 - [awsRegion](ConnectionOptions.md#awsregion)
 - [hostOverride](ConnectionOptions.md#hostoverride)
+- [readConsistencyInterval](ConnectionOptions.md#readconsistencyinterval)
 - [region](ConnectionOptions.md#region)
+- [storageOptions](ConnectionOptions.md#storageoptions)
+- [timeout](ConnectionOptions.md#timeout)
 - [uri](ConnectionOptions.md#uri)

 ## Properties
@@ -19,9 +22,13 @@

 • `Optional` **apiKey**: `string`

+API key for the remote connections
+
+Can also be passed by setting environment variable `LANCEDB_API_KEY`
+
 #### Defined in

-[index.ts:81](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L81)
+[index.ts:112](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L112)

 ___

@@ -33,9 +40,14 @@ User provided AWS crednetials.

 If not provided, LanceDB will use the default credentials provider chain.

+**`Deprecated`**
+
+Pass `aws_access_key_id`, `aws_secret_access_key`, and `aws_session_token`
+through `storageOptions` instead.
+
 #### Defined in

-[index.ts:75](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L75)
+[index.ts:92](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L92)

 ___

@@ -43,11 +55,15 @@ ___

 • `Optional` **awsRegion**: `string`

-AWS region to connect to. Default is defaultAwsRegion.
+AWS region to connect to. Default is defaultAwsRegion
+
+**`Deprecated`**
+
+Pass `region` through `storageOptions` instead.

 #### Defined in

-[index.ts:78](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L78)
+[index.ts:98](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L98)

 ___

@@ -55,13 +71,33 @@ ___

 • `Optional` **hostOverride**: `string`

-Override the host URL for the remote connections.
+Override the host URL for the remote connection.

 This is useful for local testing.

 #### Defined in

-[index.ts:91](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L91)
+[index.ts:122](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L122)
+
+___
+
+### readConsistencyInterval
+
+• `Optional` **readConsistencyInterval**: `number`
+
+(For LanceDB OSS only): The interval, in seconds, at which to check for
+updates to the table from other processes. If None, then consistency is not
+checked. For performance reasons, this is the default. For strong
+consistency, set this to zero seconds. Then every read will check for
+updates from other processes. As a compromise, you can set this to a
+non-zero value for eventual consistency. If more than that interval
+has passed since the last check, then the table will be checked for updates.
+Note: this consistency only applies to read operations. Write operations are
+always consistent.
+
+#### Defined in
+
+[index.ts:140](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L140)

 ___

@@ -69,11 +105,37 @@ ___

 • `Optional` **region**: `string`

-Region to connect
+Region to connect. Default is 'us-east-1'

 #### Defined in

-[index.ts:84](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L84)
+[index.ts:115](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L115)
+
+___
+
+### storageOptions
+
+• `Optional` **storageOptions**: `Record`\<`string`, `string`\>
+
+User provided options for object storage. For example, S3 credentials or request timeouts.
+
+The various options are described at https://lancedb.github.io/lancedb/guides/storage/
+
+#### Defined in
+
+[index.ts:105](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L105)
+
+___
+
+### timeout
+
+• `Optional` **timeout**: `number`
+
+Duration in milliseconds for request timeout. Default = 10,000 (10 seconds)
+
+#### Defined in
+
+[index.ts:127](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L127)

 ___

@@ -85,8 +147,8 @@ LanceDB database URI.

 - `/path/to/database` - local database
 - `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (SaaS)
+- `db://host:port` - remote database (LanceDB cloud)

 #### Defined in

-[index.ts:69](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L69)
+[index.ts:83](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L83)
--- a/docs/src/javascript/interfaces/CreateTableOptions.md
+++ b/docs/src/javascript/interfaces/CreateTableOptions.md
@@ -26,7 +26,7 @@

 #### Defined in

-[index.ts:116](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L116)
+[index.ts:163](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L163)

 ___

@@ -36,7 +36,7 @@ ___

 #### Defined in

-[index.ts:122](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L122)
+[index.ts:169](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L169)

 ___

@@ -46,7 +46,7 @@ ___

 #### Defined in

-[index.ts:113](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L113)
+[index.ts:160](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L160)

 ___

@@ -56,7 +56,7 @@ ___

 #### Defined in

-[index.ts:119](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L119)
+[index.ts:166](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L166)

 ___

@@ -66,4 +66,4 @@ ___

 #### Defined in

-[index.ts:125](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L125)
+[index.ts:172](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L172)
--- a/docs/src/javascript/interfaces/EmbeddingFunction.md
+++ b/docs/src/javascript/interfaces/EmbeddingFunction.md
@@ -18,11 +18,29 @@ An embedding function that automatically creates vector representation for a giv

 ### Properties

+- [destColumn](EmbeddingFunction.md#destcolumn)
 - [embed](EmbeddingFunction.md#embed)
+- [embeddingDataType](EmbeddingFunction.md#embeddingdatatype)
+- [embeddingDimension](EmbeddingFunction.md#embeddingdimension)
+- [excludeSource](EmbeddingFunction.md#excludesource)
 - [sourceColumn](EmbeddingFunction.md#sourcecolumn)

 ## Properties

+### destColumn
+
+• `Optional` **destColumn**: `string`
+
+The name of the column that will contain the embedding
+
+By default this is "vector"
+
+#### Defined in
+
+[embedding/embedding_function.ts:49](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L49)
+
+___
+
 ### embed

 • **embed**: (`data`: `T`[]) => `Promise`\<`number`[][]\>
@@ -45,7 +63,54 @@ Creates a vector representation for the given values.

 #### Defined in

-[embedding/embedding_function.ts:27](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/embedding_function.ts#L27)
+[embedding/embedding_function.ts:62](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L62)
+
+___
+
+### embeddingDataType
+
+• `Optional` **embeddingDataType**: `Float`\<`Floats`\>
+
+The data type of the embedding
+
+The embedding function should return `number`.  This will be converted into
+an Arrow float array.  By default this will be Float32 but this property can
+be used to control the conversion.
+
+#### Defined in
+
+[embedding/embedding_function.ts:33](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L33)
+
+___
+
+### embeddingDimension
+
+• `Optional` **embeddingDimension**: `number`
+
+The dimension of the embedding
+
+This is optional, normally this can be determined by looking at the results of
+`embed`.  If this is not specified, and there is an attempt to apply the embedding
+to an empty table, then that process will fail.
+
+#### Defined in
+
+[embedding/embedding_function.ts:42](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L42)
+
+___
+
+### excludeSource
+
+• `Optional` **excludeSource**: `boolean`
+
+Should the source column be excluded from the resulting table
+
+By default the source column is included.  Set this to true and
+only the embedding will be stored.
+
+#### Defined in
+
+[embedding/embedding_function.ts:57](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L57)

 ___

@@ -57,4 +122,4 @@ The name of the column that will be used as input for the Embedding Function.

 #### Defined in

-[embedding/embedding_function.ts:22](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/embedding/embedding_function.ts#L22)
+[embedding/embedding_function.ts:24](https://github.com/lancedb/lancedb/blob/92179835/node/src/embedding/embedding_function.ts#L24)
--- a/docs/src/javascript/interfaces/IndexStats.md
+++ b/docs/src/javascript/interfaces/IndexStats.md
@@ -6,18 +6,51 @@

 ### Properties

+- [distanceType](IndexStats.md#distancetype)
+- [indexType](IndexStats.md#indextype)
 - [numIndexedRows](IndexStats.md#numindexedrows)
+- [numIndices](IndexStats.md#numindices)
 - [numUnindexedRows](IndexStats.md#numunindexedrows)

 ## Properties

+### distanceType
+
+• `Optional` **distanceType**: `string`
+
+#### Defined in
+
+[index.ts:728](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L728)
+
+___
+
+### indexType
+
+• **indexType**: `string`
+
+#### Defined in
+
+[index.ts:727](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L727)
+
+___
+
 ### numIndexedRows

 • **numIndexedRows**: ``null`` \| `number`

 #### Defined in

-[index.ts:478](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L478)
+[index.ts:725](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L725)
+
+___
+
+### numIndices
+
+• `Optional` **numIndices**: `number`
+
+#### Defined in
+
+[index.ts:729](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L729)

 ___

@@ -27,4 +60,4 @@ ___

 #### Defined in

-[index.ts:479](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L479)
+[index.ts:726](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L726)
--- a/docs/src/javascript/interfaces/IvfPQIndexConfig.md
+++ b/docs/src/javascript/interfaces/IvfPQIndexConfig.md
@@ -29,7 +29,7 @@ The column to be indexed

 #### Defined in

-[index.ts:942](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L942)
+[index.ts:1282](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1282)

 ___

@@ -41,7 +41,7 @@ Cache size of the index

 #### Defined in

-[index.ts:991](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L991)
+[index.ts:1331](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1331)

 ___

@@ -53,7 +53,7 @@ A unique name for the index

 #### Defined in

-[index.ts:947](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L947)
+[index.ts:1287](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1287)

 ___

@@ -65,7 +65,7 @@ The max number of iterations for kmeans training.

 #### Defined in

-[index.ts:962](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L962)
+[index.ts:1302](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1302)

 ___

@@ -77,7 +77,7 @@ Max number of iterations to train OPQ, if `use_opq` is true.

 #### Defined in

-[index.ts:981](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L981)
+[index.ts:1321](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1321)

 ___

@@ -89,7 +89,7 @@ Metric type, L2 or Cosine

 #### Defined in

-[index.ts:952](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L952)
+[index.ts:1292](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1292)

 ___

@@ -101,7 +101,7 @@ The number of bits to present one PQ centroid.

 #### Defined in

-[index.ts:976](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L976)
+[index.ts:1316](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1316)

 ___

@@ -113,7 +113,7 @@ The number of partitions this index

 #### Defined in

-[index.ts:957](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L957)
+[index.ts:1297](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1297)

 ___

@@ -125,7 +125,7 @@ Number of subvectors to build PQ code

 #### Defined in

-[index.ts:972](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L972)
+[index.ts:1312](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1312)

 ___

@@ -137,7 +137,7 @@ Replace an existing index with the same name if it exists.

 #### Defined in

-[index.ts:986](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L986)
+[index.ts:1326](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1326)

 ___

@@ -147,7 +147,7 @@ ___

 #### Defined in

-[index.ts:993](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L993)
+[index.ts:1333](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1333)

 ___

@@ -159,4 +159,4 @@ Train as optimized product quantization.

 #### Defined in

-[index.ts:967](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L967)
+[index.ts:1307](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1307)
--- a/docs/src/javascript/interfaces/MergeInsertArgs.md
+++ b/docs/src/javascript/interfaces/MergeInsertArgs.md
@@ -0,0 +1,73 @@
+[vectordb](../README.md) / [Exports](../modules.md) / MergeInsertArgs
+
+# Interface: MergeInsertArgs
+
+## Table of contents
+
+### Properties
+
+- [whenMatchedUpdateAll](MergeInsertArgs.md#whenmatchedupdateall)
+- [whenNotMatchedBySourceDelete](MergeInsertArgs.md#whennotmatchedbysourcedelete)
+- [whenNotMatchedInsertAll](MergeInsertArgs.md#whennotmatchedinsertall)
+
+## Properties
+
+### whenMatchedUpdateAll
+
+• `Optional` **whenMatchedUpdateAll**: `string` \| `boolean`
+
+If true then rows that exist in both the source table (new data) and
+the target table (old data) will be updated, replacing the old row
+with the corresponding matching row.
+
+If there are multiple matches then the behavior is undefined.
+Currently this causes multiple copies of the row to be created
+but that behavior is subject to change.
+
+Optionally, a filter can be specified.  This should be an SQL
+filter where fields with the prefix "target." refer to fields
+in the target table (old data) and fields with the prefix
+"source." refer to fields in the source table (new data).  For
+example, the filter "target.lastUpdated < source.lastUpdated" will
+only update matched rows when the incoming `lastUpdated` value is
+newer.
+
+Rows that do not match the filter will not be updated.  Rows that
+do not match the filter do become "not matched" rows.
+
+#### Defined in
+
+[index.ts:690](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L690)
+
+___
+
+### whenNotMatchedBySourceDelete
+
+• `Optional` **whenNotMatchedBySourceDelete**: `string` \| `boolean`
+
+If true then rows that exist only in the target table (old data)
+will be deleted.
+
+If this is a string then it will be treated as an SQL filter and
+only rows that both do not match any row in the source table and
+match the given filter will be deleted.
+
+This can be used to replace a selection of existing data with
+new data.
+
+#### Defined in
+
+[index.ts:707](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L707)
+
+___
+
+### whenNotMatchedInsertAll
+
+• `Optional` **whenNotMatchedInsertAll**: `boolean`
+
+If true then rows that exist only in the source table (new data)
+will be inserted into the target table.
+
+#### Defined in
+
+[index.ts:695](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L695)
--- a/docs/src/javascript/interfaces/Table.md
+++ b/docs/src/javascript/interfaces/Table.md
@@ -25,17 +25,26 @@ A LanceDB Table is the collection of Records. Each Record has one or more vector
 - [delete](Table.md#delete)
 - [indexStats](Table.md#indexstats)
 - [listIndices](Table.md#listindices)
+- [mergeInsert](Table.md#mergeinsert)
 - [name](Table.md#name)
 - [overwrite](Table.md#overwrite)
 - [schema](Table.md#schema)
 - [search](Table.md#search)
 - [update](Table.md#update)

+### Methods
+
+- [addColumns](Table.md#addcolumns)
+- [alterColumns](Table.md#altercolumns)
+- [dropColumns](Table.md#dropcolumns)
+- [filter](Table.md#filter)
+- [withMiddleware](Table.md#withmiddleware)
+
 ## Properties

 ### add

-• **add**: (`data`: `Record`\<`string`, `unknown`\>[]) => `Promise`\<`number`\>
+• **add**: (`data`: `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[]) => `Promise`\<`number`\>

 #### Type declaration

@@ -47,7 +56,7 @@ Insert records into this Table.

 | Name | Type | Description |
 | :------ | :------ | :------ |
-| `data` | `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |

 ##### Returns

@@ -57,27 +66,33 @@ The number of rows added to the table

 #### Defined in

-[index.ts:291](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L291)
+[index.ts:381](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L381)

 ___

 ### countRows

-• **countRows**: () => `Promise`\<`number`\>
+• **countRows**: (`filter?`: `string`) => `Promise`\<`number`\>

 #### Type declaration

-▸ (): `Promise`\<`number`\>
+▸ (`filter?`): `Promise`\<`number`\>

 Returns the number of rows in this table.

+##### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `filter?` | `string` |
+
 ##### Returns

 `Promise`\<`number`\>

 #### Defined in

-[index.ts:361](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L361)
+[index.ts:454](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L454)

 ___

@@ -107,17 +122,17 @@ VectorIndexParams.

 #### Defined in

-[index.ts:306](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L306)
+[index.ts:398](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L398)

 ___

 ### createScalarIndex

-• **createScalarIndex**: (`column`: `string`, `replace`: `boolean`) => `Promise`\<`void`\>
+• **createScalarIndex**: (`column`: `string`, `replace?`: `boolean`) => `Promise`\<`void`\>

 #### Type declaration

-▸ (`column`, `replace`): `Promise`\<`void`\>
+▸ (`column`, `replace?`): `Promise`\<`void`\>

 Create a scalar index on this Table for the given column

@@ -126,7 +141,7 @@ Create a scalar index on this Table for the given column
 | Name | Type | Description |
 | :------ | :------ | :------ |
 | `column` | `string` | The column to index |
-| `replace` | `boolean` | If false, fail if an index already exists on the column Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column `my_col` has a scalar index: ```ts const con = await lancedb.connect('./.lancedb'); const table = await con.openTable('images'); const results = await table.where('my_col = 7').execute(); ``` Scalar indices can also speed up scans containing a vector search and a prefilter: ```ts const con = await lancedb.connect('././lancedb'); const table = await con.openTable('images'); const results = await table.search([1.0, 2.0]).where('my_col != 7').prefilter(true); ``` Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. `my_col BETWEEN 0 AND 100`), and set membership (e.g. `my_col IN (0, 1, 2)`) Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. `my_col < 0 AND other_col> 100`) Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column `not_indexed` does not have a scalar index then the filter `my_col = 0 OR not_indexed = 1` will not be able to use any scalar index on `my_col`. |
+| `replace?` | `boolean` | If false, fail if an index already exists on the column it is always set to true for remote connections Scalar indices, like vector indices, can be used to speed up scans. A scalar index can speed up scans that contain filter expressions on the indexed column. For example, the following scan will be faster if the column `my_col` has a scalar index: ```ts const con = await lancedb.connect('./.lancedb'); const table = await con.openTable('images'); const results = await table.where('my_col = 7').execute(); ``` Scalar indices can also speed up scans containing a vector search and a prefilter: ```ts const con = await lancedb.connect('././lancedb'); const table = await con.openTable('images'); const results = await table.search([1.0, 2.0]).where('my_col != 7').prefilter(true); ``` Scalar indices can only speed up scans for basic filters using equality, comparison, range (e.g. `my_col BETWEEN 0 AND 100`), and set membership (e.g. `my_col IN (0, 1, 2)`) Scalar indices can be used if the filter contains multiple indexed columns and the filter criteria are AND'd or OR'd together (e.g. `my_col < 0 AND other_col> 100`) Scalar indices may be used if the filter contains non-indexed columns but, depending on the structure of the filter, they may not be usable. For example, if the column `not_indexed` does not have a scalar index then the filter `my_col = 0 OR not_indexed = 1` will not be able to use any scalar index on `my_col`. |

 ##### Returns

@@ -142,7 +157,7 @@ await table.createScalarIndex('my_col')

 #### Defined in

-[index.ts:356](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L356)
+[index.ts:449](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L449)

 ___

@@ -194,17 +209,17 @@ await tbl.countRows() // Returns 1

 #### Defined in

-[index.ts:395](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L395)
+[index.ts:488](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L488)

 ___

 ### indexStats

-• **indexStats**: (`indexUuid`: `string`) => `Promise`\<[`IndexStats`](IndexStats.md)\>
+• **indexStats**: (`indexName`: `string`) => `Promise`\<[`IndexStats`](IndexStats.md)\>

 #### Type declaration

-▸ (`indexUuid`): `Promise`\<[`IndexStats`](IndexStats.md)\>
+▸ (`indexName`): `Promise`\<[`IndexStats`](IndexStats.md)\>

 Get statistics about an index.

@@ -212,7 +227,7 @@ Get statistics about an index.

 | Name | Type |
 | :------ | :------ |
-| `indexUuid` | `string` |
+| `indexName` | `string` |

 ##### Returns

@@ -220,7 +235,7 @@ Get statistics about an index.

 #### Defined in

-[index.ts:438](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L438)
+[index.ts:567](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L567)

 ___

@@ -240,7 +255,57 @@ List the indicies on this table.

 #### Defined in

-[index.ts:433](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L433)
+[index.ts:562](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L562)
+
+___
+
+### mergeInsert
+
+• **mergeInsert**: (`on`: `string`, `data`: `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[], `args`: [`MergeInsertArgs`](MergeInsertArgs.md)) => `Promise`\<`void`\>
+
+#### Type declaration
+
+▸ (`on`, `data`, `args`): `Promise`\<`void`\>
+
+Runs a "merge insert" operation on the table
+
+This operation can add rows, update rows, and remove rows all in a single
+transaction. It is a very generic tool that can be used to create
+behaviors like "insert if not exists", "update or insert (i.e. upsert)",
+or even replace a portion of existing data with new data (e.g. replace
+all data where month="january")
+
+The merge insert operation works by combining new data from a
+**source table** with existing data in a **target table** by using a
+join.  There are three categories of records.
+
+"Matched" records are records that exist in both the source table and
+the target table. "Not matched" records exist only in the source table
+(e.g. these are new data) "Not matched by source" records exist only
+in the target table (this is old data)
+
+The MergeInsertArgs can be used to customize what should happen for
+each category of data.
+
+Please note that the data may appear to be reordered as part of this
+operation.  This is because updated rows will be deleted from the
+dataset and then reinserted at the end with the new values.
+
+##### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `on` | `string` | a column to join on. This is how records from the source table and target table are matched. |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | the new data to insert |
+| `args` | [`MergeInsertArgs`](MergeInsertArgs.md) | parameters controlling how the operation should behave |
+
+##### Returns
+
+`Promise`\<`void`\>
+
+#### Defined in
+
+[index.ts:553](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L553)

 ___

@@ -250,13 +315,13 @@ ___

 #### Defined in

-[index.ts:277](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L277)
+[index.ts:367](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L367)

 ___

 ### overwrite

-• **overwrite**: (`data`: `Record`\<`string`, `unknown`\>[]) => `Promise`\<`number`\>
+• **overwrite**: (`data`: `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[]) => `Promise`\<`number`\>

 #### Type declaration

@@ -268,7 +333,7 @@ Insert records into this Table, replacing its contents.

 | Name | Type | Description |
 | :------ | :------ | :------ |
-| `data` | `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |
+| `data` | `Table`\<`any`\> \| `Record`\<`string`, `unknown`\>[] | Records to be inserted into the Table |

 ##### Returns

@@ -278,7 +343,7 @@ The number of rows added to the table

 #### Defined in

-[index.ts:299](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L299)
+[index.ts:389](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L389)

 ___

@@ -288,7 +353,7 @@ ___

 #### Defined in

-[index.ts:440](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L440)
+[index.ts:571](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L571)

 ___

@@ -314,7 +379,7 @@ Creates a search query to find the nearest neighbors of the given search term

 #### Defined in

-[index.ts:283](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L283)
+[index.ts:373](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L373)

 ___

@@ -365,4 +430,123 @@ let results = await tbl.search([1, 1]).execute();

 #### Defined in

-[index.ts:428](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L428)
+[index.ts:521](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L521)
+
+## Methods
+
+### addColumns
+
+▸ **addColumns**(`newColumnTransforms`): `Promise`\<`void`\>
+
+Add new columns with defined values.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `newColumnTransforms` | \{ `name`: `string` ; `valueSql`: `string`  }[] | pairs of column names and the SQL expression to use to calculate the value of the new column. These expressions will be evaluated for each row in the table, and can reference existing columns in the table. |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Defined in
+
+[index.ts:582](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L582)
+
+___
+
+### alterColumns
+
+▸ **alterColumns**(`columnAlterations`): `Promise`\<`void`\>
+
+Alter the name or nullability of columns.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `columnAlterations` | [`ColumnAlteration`](ColumnAlteration.md)[] | One or more alterations to apply to columns. |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Defined in
+
+[index.ts:591](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L591)
+
+___
+
+### dropColumns
+
+▸ **dropColumns**(`columnNames`): `Promise`\<`void`\>
+
+Drop one or more columns from the dataset
+
+This is a metadata-only operation and does not remove the data from the
+underlying storage. In order to remove the data, you must subsequently
+call ``compact_files`` to rewrite the data without the removed columns and
+then call ``cleanup_files`` to remove the old files.
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `columnNames` | `string`[] | The names of the columns to drop. These can be nested column references (e.g. "a.b.c") or top-level column names (e.g. "a"). |
+
+#### Returns
+
+`Promise`\<`void`\>
+
+#### Defined in
+
+[index.ts:605](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L605)
+
+___
+
+### filter
+
+▸ **filter**(`value`): [`Query`](../classes/Query.md)\<`T`\>
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `value` | `string` |
+
+#### Returns
+
+[`Query`](../classes/Query.md)\<`T`\>
+
+#### Defined in
+
+[index.ts:569](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L569)
+
+___
+
+### withMiddleware
+
+▸ **withMiddleware**(`middleware`): [`Table`](Table.md)\<`T`\>
+
+Instrument the behavior of this Table with middleware.
+
+The middleware will be called in the order they are added.
+
+Currently this functionality is only supported for remote tables.
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `middleware` | `HttpMiddleware` |
+
+#### Returns
+
+[`Table`](Table.md)\<`T`\>
+
+- this Table instrumented by the passed middleware
+
+#### Defined in
+
+[index.ts:617](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L617)
--- a/docs/src/javascript/interfaces/UpdateArgs.md
+++ b/docs/src/javascript/interfaces/UpdateArgs.md
@@ -20,7 +20,7 @@ new values to set

 #### Defined in

-[index.ts:454](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L454)
+[index.ts:652](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L652)

 ___

@@ -33,4 +33,4 @@ in which case all rows will be updated.

 #### Defined in

-[index.ts:448](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L448)
+[index.ts:646](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L646)
--- a/docs/src/javascript/interfaces/UpdateSqlArgs.md
+++ b/docs/src/javascript/interfaces/UpdateSqlArgs.md
@@ -20,7 +20,7 @@ new values to set as SQL expressions.

 #### Defined in

-[index.ts:468](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L468)
+[index.ts:666](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L666)

 ___

@@ -33,4 +33,4 @@ in which case all rows will be updated.

 #### Defined in

-[index.ts:462](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L462)
+[index.ts:660](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L660)
--- a/docs/src/javascript/interfaces/VectorIndex.md
+++ b/docs/src/javascript/interfaces/VectorIndex.md
@@ -8,6 +8,7 @@

 - [columns](VectorIndex.md#columns)
 - [name](VectorIndex.md#name)
+- [status](VectorIndex.md#status)
 - [uuid](VectorIndex.md#uuid)

 ## Properties
@@ -18,7 +19,7 @@

 #### Defined in

-[index.ts:472](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L472)
+[index.ts:718](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L718)

 ___

@@ -28,7 +29,17 @@ ___

 #### Defined in

-[index.ts:473](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L473)
+[index.ts:719](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L719)
+
+___
+
+### status
+
+• **status**: [`IndexStatus`](../enums/IndexStatus.md)
+
+#### Defined in
+
+[index.ts:721](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L721)

 ___

@@ -38,4 +49,4 @@ ___

 #### Defined in

-[index.ts:474](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L474)
+[index.ts:720](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L720)
--- a/docs/src/javascript/interfaces/WriteOptions.md
+++ b/docs/src/javascript/interfaces/WriteOptions.md
@@ -24,4 +24,4 @@ A [WriteMode](../enums/WriteMode.md) to use on this operation

 #### Defined in

-[index.ts:1015](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1015)
+[index.ts:1355](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1355)
--- a/docs/src/javascript/modules.md
+++ b/docs/src/javascript/modules.md
@@ -6,6 +6,7 @@

 ### Enumerations

+- [IndexStatus](enums/IndexStatus.md)
 - [MetricType](enums/MetricType.md)
 - [WriteMode](enums/WriteMode.md)

@@ -14,6 +15,7 @@
 - [DefaultWriteOptions](classes/DefaultWriteOptions.md)
 - [LocalConnection](classes/LocalConnection.md)
 - [LocalTable](classes/LocalTable.md)
+- [MakeArrowTableOptions](classes/MakeArrowTableOptions.md)
 - [OpenAIEmbeddingFunction](classes/OpenAIEmbeddingFunction.md)
 - [Query](classes/Query.md)

@@ -21,6 +23,7 @@

 - [AwsCredentials](interfaces/AwsCredentials.md)
 - [CleanupStats](interfaces/CleanupStats.md)
+- [ColumnAlteration](interfaces/ColumnAlteration.md)
 - [CompactionMetrics](interfaces/CompactionMetrics.md)
 - [CompactionOptions](interfaces/CompactionOptions.md)
 - [Connection](interfaces/Connection.md)
@@ -29,6 +32,7 @@
 - [EmbeddingFunction](interfaces/EmbeddingFunction.md)
 - [IndexStats](interfaces/IndexStats.md)
 - [IvfPQIndexConfig](interfaces/IvfPQIndexConfig.md)
+- [MergeInsertArgs](interfaces/MergeInsertArgs.md)
 - [Table](interfaces/Table.md)
 - [UpdateArgs](interfaces/UpdateArgs.md)
 - [UpdateSqlArgs](interfaces/UpdateSqlArgs.md)
@@ -42,7 +46,9 @@
 ### Functions

 - [connect](modules.md#connect)
+- [convertToTable](modules.md#converttotable)
 - [isWriteOptions](modules.md#iswriteoptions)
+- [makeArrowTable](modules.md#makearrowtable)

 ## Type Aliases

@@ -52,7 +58,7 @@

 #### Defined in

-[index.ts:996](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L996)
+[index.ts:1336](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1336)

 ## Functions

@@ -62,11 +68,11 @@

 Connect to a LanceDB instance at the given URI.

-Accpeted formats:
+Accepted formats:

 - `/path/to/database` - local database
 - `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud storage
- `db://host:port` - remote database (SaaS)
+- `db://host:port` - remote database (LanceDB cloud)

 #### Parameters

@@ -84,7 +90,7 @@ Accpeted formats:

 #### Defined in

-[index.ts:141](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L141)
+[index.ts:188](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L188)

 ▸ **connect**(`opts`): `Promise`\<[`Connection`](interfaces/Connection.md)\>

@@ -102,7 +108,35 @@ Connect to a LanceDB instance with connection options.

 #### Defined in

-[index.ts:147](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L147)
+[index.ts:194](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L194)
+
+___
+
+### convertToTable
+
+▸ **convertToTable**\<`T`\>(`data`, `embeddings?`, `makeTableOptions?`): `Promise`\<`ArrowTable`\>
+
+#### Type parameters
+
+| Name |
+| :------ |
+| `T` |
+
+#### Parameters
+
+| Name | Type |
+| :------ | :------ |
+| `data` | `Record`\<`string`, `unknown`\>[] |
+| `embeddings?` | [`EmbeddingFunction`](interfaces/EmbeddingFunction.md)\<`T`\> |
+| `makeTableOptions?` | `Partial`\<[`MakeArrowTableOptions`](classes/MakeArrowTableOptions.md)\> |
+
+#### Returns
+
+`Promise`\<`ArrowTable`\>
+
+#### Defined in
+
+[arrow.ts:465](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L465)

 ___

@@ -122,4 +156,116 @@ value is WriteOptions

 #### Defined in

-[index.ts:1022](https://github.com/lancedb/lancedb/blob/c89d5e6/node/src/index.ts#L1022)
+[index.ts:1362](https://github.com/lancedb/lancedb/blob/92179835/node/src/index.ts#L1362)
+
+___
+
+### makeArrowTable
+
+▸ **makeArrowTable**(`data`, `options?`): `ArrowTable`
+
+An enhanced version of the makeTable function from Apache Arrow
+that supports nested fields and embeddings columns.
+
+This function converts an array of Record<String, any> (row-major JS objects)
+to an Arrow Table (a columnar structure)
+
+Note that it currently does not support nulls.
+
+If a schema is provided then it will be used to determine the resulting array
+types.  Fields will also be reordered to fit the order defined by the schema.
+
+If a schema is not provided then the types will be inferred and the field order
+will be controlled by the order of properties in the first record.
+
+If the input is empty then a schema must be provided to create an empty table.
+
+When a schema is not specified then data types will be inferred.  The inference
+rules are as follows:
+
+ - boolean => Bool
+ - number => Float64
+ - String => Utf8
+ - Buffer => Binary
+ - Record<String, any> => Struct
+ - Array<any> => List
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `data` | `Record`\<`string`, `any`\>[] | input data |
+| `options?` | `Partial`\<[`MakeArrowTableOptions`](classes/MakeArrowTableOptions.md)\> | options to control the makeArrowTable call. |
+
+#### Returns
+
+`ArrowTable`
+
+**`Example`**
+
+```ts
+
+import { fromTableToBuffer, makeArrowTable } from "../arrow";
+import { Field, FixedSizeList, Float16, Float32, Int32, Schema } from "apache-arrow";
+
+const schema = new Schema([
+  new Field("a", new Int32()),
+  new Field("b", new Float32()),
+  new Field("c", new FixedSizeList(3, new Field("item", new Float16()))),
+ ]);
+ const table = makeArrowTable([
+   { a: 1, b: 2, c: [1, 2, 3] },
+   { a: 4, b: 5, c: [4, 5, 6] },
+   { a: 7, b: 8, c: [7, 8, 9] },
+ ], { schema });
+```
+
+By default it assumes that the column named `vector` is a vector column
+and it will be converted into a fixed size list array of type float32.
+The `vectorColumns` option can be used to support other vector column
+names and data types.
+
+```ts
+
+const schema = new Schema([
+   new Field("a", new Float64()),
+   new Field("b", new Float64()),
+   new Field(
+     "vector",
+     new FixedSizeList(3, new Field("item", new Float32()))
+   ),
+ ]);
+ const table = makeArrowTable([
+   { a: 1, b: 2, vector: [1, 2, 3] },
+   { a: 4, b: 5, vector: [4, 5, 6] },
+   { a: 7, b: 8, vector: [7, 8, 9] },
+ ]);
+ assert.deepEqual(table.schema, schema);
+```
+
+You can specify the vector column types and names using the options as well
+
+```typescript
+
+const schema = new Schema([
+   new Field('a', new Float64()),
+   new Field('b', new Float64()),
+   new Field('vec1', new FixedSizeList(3, new Field('item', new Float16()))),
+   new Field('vec2', new FixedSizeList(3, new Field('item', new Float16())))
+ ]);
+const table = makeArrowTable([
+   { a: 1, b: 2, vec1: [1, 2, 3], vec2: [2, 4, 6] },
+   { a: 4, b: 5, vec1: [4, 5, 6], vec2: [8, 10, 12] },
+   { a: 7, b: 8, vec1: [7, 8, 9], vec2: [14, 16, 18] }
+ ], {
+   vectorColumns: {
+     vec1: { type: new Float16() },
+     vec2: { type: new Float16() }
+   }
+ }
+assert.deepEqual(table.schema, schema)
+```
+
+#### Defined in
+
+[arrow.ts:198](https://github.com/lancedb/lancedb/blob/92179835/node/src/arrow.ts#L198)
--- a/docs/src/styles/extra.css
+++ b/docs/src/styles/extra.css
@@ -39,4 +39,46 @@
      height: 1.2rem;
      margin-top: -.1rem;
    }
-  }
+  }
+
+/* remove pilcrow as permanent link and add chain icon similar to github https://github.com/squidfunk/mkdocs-material/discussions/3535 */
+
+.headerlink {
+	--permalink-size: 16px; /* for font-relative sizes, 0.6em is a good choice */
+	--permalink-spacing: 4px;
+
+	width: calc(var(--permalink-size) + var(--permalink-spacing));
+	height: var(--permalink-size);
+	vertical-align: middle;
+	background-color: var(--md-default-fg-color--lighter);
+	background-size: var(--permalink-size);
+	mask-size: var(--permalink-size);
+	-webkit-mask-size: var(--permalink-size);
+	mask-repeat: no-repeat;
+	-webkit-mask-repeat: no-repeat;
+	visibility: visible;
+	mask-image: url('data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg>');
+	-webkit-mask-image: url('data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg>');
+}
+
+[id]:target .headerlink {
+	background-color: var(--md-typeset-a-color);
+}
+
+.headerlink:hover {
+	background-color: var(--md-accent-fg-color) !important;
+}
+
+@media screen and (min-width: 76.25em) {
+	h1, h2, h3, h4, h5, h6 {
+		display: flex;
+		align-items: center;
+		flex-direction: row;
+		column-gap: 0.2em; /* fixes spaces in titles */
+	}
+
+	.headerlink {
+		order: -1;
+		margin-left: calc(var(--permalink-size) * -1 - var(--permalink-spacing)) !important;
+	}
+}
--- a/java/core/pom.xml
+++ b/java/core/pom.xml
@@ -8,7 +8,7 @@
    <parent>
        <groupId>com.lancedb</groupId>
        <artifactId>lancedb-parent</artifactId>
-        <version>0.11.0-beta.1</version>
+        <version>0.11.1-beta.0</version>
        <relativePath>../pom.xml</relativePath>
    </parent>

--- a/java/pom.xml
+++ b/java/pom.xml
@@ -6,7 +6,7 @@

    <groupId>com.lancedb</groupId>
    <artifactId>lancedb-parent</artifactId>
-    <version>0.11.0-beta.1</version>
+    <version>0.11.1-beta.0</version>
    <packaging>pom</packaging>

    <name>LanceDB Parent</name>
--- a/node/package-lock.json
+++ b/node/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "vectordb",
-  "version": "0.11.0-beta.1",
+  "version": "0.11.1-beta.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "vectordb",
-      "version": "0.11.0-beta.1",
+      "version": "0.11.1-beta.0",
      "cpu": [
        "x64",
        "arm64"
@@ -52,11 +52,11 @@
        "uuid": "^9.0.0"
      },
      "optionalDependencies": {
-        "@lancedb/vectordb-darwin-arm64": "0.11.0-beta.1",
-        "@lancedb/vectordb-darwin-x64": "0.11.0-beta.1",
-        "@lancedb/vectordb-linux-arm64-gnu": "0.11.0-beta.1",
-        "@lancedb/vectordb-linux-x64-gnu": "0.11.0-beta.1",
-        "@lancedb/vectordb-win32-x64-msvc": "0.11.0-beta.1"
+        "@lancedb/vectordb-darwin-arm64": "0.11.1-beta.0",
+        "@lancedb/vectordb-darwin-x64": "0.11.1-beta.0",
+        "@lancedb/vectordb-linux-arm64-gnu": "0.11.1-beta.0",
+        "@lancedb/vectordb-linux-x64-gnu": "0.11.1-beta.0",
+        "@lancedb/vectordb-win32-x64-msvc": "0.11.1-beta.0"
      },
      "peerDependencies": {
        "@apache-arrow/ts": "^14.0.2",
@@ -326,66 +326,6 @@
        "@jridgewell/sourcemap-codec": "^1.4.10"
      }
    },
-    "node_modules/@lancedb/vectordb-darwin-arm64": {
-      "version": "0.11.0-beta.1",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-arm64/-/vectordb-darwin-arm64-0.11.0-beta.1.tgz",
-      "integrity": "sha512-qKQbFJwstMQEO2MVkkipyDxmH3/KafkuC4xfU8LjMtZ98ZGTQIW47t/OyftiUXYWcjsVxeXI3l2m9MCozFOdhg==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "darwin"
-      ]
-    },
-    "node_modules/@lancedb/vectordb-darwin-x64": {
-      "version": "0.11.0-beta.1",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-darwin-x64/-/vectordb-darwin-x64-0.11.0-beta.1.tgz",
-      "integrity": "sha512-245Q5hjQKljczBcDLbiq3N5fmUaY2zFRHoW6SBxOziQwyMphhLDSTNkAYkc3JnrQvf6dMolVYWigOsRVCFj56A==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "darwin"
-      ]
-    },
-    "node_modules/@lancedb/vectordb-linux-arm64-gnu": {
-      "version": "0.11.0-beta.1",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-arm64-gnu/-/vectordb-linux-arm64-gnu-0.11.0-beta.1.tgz",
-      "integrity": "sha512-B4z6sx4X6uqGDnQm3zL5mL47Agn4X4spf/nlxtrUWEfiOAyp9Iw465UQMmrbnodi+4k/BNjCNZNMFSjMOSsrcA==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ]
-    },
-    "node_modules/@lancedb/vectordb-linux-x64-gnu": {
-      "version": "0.11.0-beta.1",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-linux-x64-gnu/-/vectordb-linux-x64-gnu-0.11.0-beta.1.tgz",
-      "integrity": "sha512-0vWcPqpe3to78bYkc+3XWZToRu6TMrhLJAxC9cnV5d9GMuN1VbDoLqD8QPRWkoEr9Nk7jdIRKEBUwfq5yGOFLw==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "linux"
-      ]
-    },
-    "node_modules/@lancedb/vectordb-win32-x64-msvc": {
-      "version": "0.11.0-beta.1",
-      "resolved": "https://registry.npmjs.org/@lancedb/vectordb-win32-x64-msvc/-/vectordb-win32-x64-msvc-0.11.0-beta.1.tgz",
-      "integrity": "sha512-jU/+w2TfA4HKOZkib1UP4kIpaLgu+88S/t+Ccde67w/4qQuP0uAixTAls1WE4mtlf6pOnG0A1ILTY98nVkIQ3A==",
-      "cpu": [
-        "x64"
-      ],
-      "optional": true,
-      "os": [
-        "win32"
-      ]
-    },
    "node_modules/@neon-rs/cli": {
      "version": "0.0.160",
      "resolved": "https://registry.npmjs.org/@neon-rs/cli/-/cli-0.0.160.tgz",
--- a/node/package.json
+++ b/node/package.json
@@ -1,6 +1,6 @@
 {
  "name": "vectordb",
-  "version": "0.11.0-beta.1",
+  "version": "0.11.1-beta.0",
  "description": " Serverless, low-latency vector database for AI applications",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
@@ -88,10 +88,10 @@
    }
  },
  "optionalDependencies": {
-    "@lancedb/vectordb-darwin-arm64": "0.11.0-beta.1",
-    "@lancedb/vectordb-darwin-x64": "0.11.0-beta.1",
-    "@lancedb/vectordb-linux-arm64-gnu": "0.11.0-beta.1",
-    "@lancedb/vectordb-linux-x64-gnu": "0.11.0-beta.1",
-    "@lancedb/vectordb-win32-x64-msvc": "0.11.0-beta.1"
+    "@lancedb/vectordb-darwin-arm64": "0.11.1-beta.0",
+    "@lancedb/vectordb-darwin-x64": "0.11.1-beta.0",
+    "@lancedb/vectordb-linux-arm64-gnu": "0.11.1-beta.0",
+    "@lancedb/vectordb-linux-x64-gnu": "0.11.1-beta.0",
+    "@lancedb/vectordb-win32-x64-msvc": "0.11.1-beta.0"
  }
 }
--- a/node/src/index.ts
+++ b/node/src/index.ts
@@ -564,7 +564,7 @@ export interface Table<T = number[]> {
  /**
   * Get statistics about an index.
   */
-  indexStats: (indexUuid: string) => Promise<IndexStats>
+  indexStats: (indexName: string) => Promise<IndexStats>

  filter(value: string): Query<T>

@@ -1164,8 +1164,8 @@ export class LocalTable<T = number[]> implements Table<T> {
    return tableListIndices.call(this._tbl);
  }

-  async indexStats(indexUuid: string): Promise<IndexStats> {
-    return tableIndexStats.call(this._tbl, indexUuid);
+  async indexStats(indexName: string): Promise<IndexStats> {
+    return tableIndexStats.call(this._tbl, indexName);
  }

  get schema(): Promise<Schema> {
--- a/node/src/remote/index.ts
+++ b/node/src/remote/index.ts
@@ -517,9 +517,9 @@ export class RemoteTable<T = number[]> implements Table<T> {
    }))
  }

-  async indexStats (indexUuid: string): Promise<IndexStats> {
+  async indexStats (indexName: string): Promise<IndexStats> {
    const results = await this._client.post(
-      `/v1/table/${encodeURIComponent(this._name)}/index/${indexUuid}/stats/`
+      `/v1/table/${encodeURIComponent(this._name)}/index/${indexName}/stats/`
    )
    const body = await results.body()
    return {
--- a/nodejs/Cargo.toml
+++ b/nodejs/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "lancedb-nodejs"
 edition.workspace = true
-version = "0.11.0-beta.1"
+version = "0.11.1-beta.0"
 license.workspace = true
 description.workspace = true
 repository.workspace = true
--- a/nodejs/npm/darwin-arm64/package.json
+++ b/nodejs/npm/darwin-arm64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-arm64",
-	"version": "0.11.0-beta.1",
+	"version": "0.11.1-beta.0",
 	"os": ["darwin"],
 	"cpu": ["arm64"],
 	"main": "lancedb.darwin-arm64.node",
--- a/nodejs/npm/darwin-x64/package.json
+++ b/nodejs/npm/darwin-x64/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-darwin-x64",
-	"version": "0.11.0-beta.1",
+	"version": "0.11.1-beta.0",
 	"os": ["darwin"],
 	"cpu": ["x64"],
 	"main": "lancedb.darwin-x64.node",
--- a/nodejs/npm/linux-arm64-gnu/package.json
+++ b/nodejs/npm/linux-arm64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-arm64-gnu",
-	"version": "0.11.0-beta.1",
+	"version": "0.11.1-beta.0",
 	"os": ["linux"],
 	"cpu": ["arm64"],
 	"main": "lancedb.linux-arm64-gnu.node",
--- a/nodejs/npm/linux-x64-gnu/package.json
+++ b/nodejs/npm/linux-x64-gnu/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-linux-x64-gnu",
-	"version": "0.11.0-beta.1",
+	"version": "0.11.1-beta.0",
 	"os": ["linux"],
 	"cpu": ["x64"],
 	"main": "lancedb.linux-x64-gnu.node",
--- a/nodejs/npm/win32-x64-msvc/package.json
+++ b/nodejs/npm/win32-x64-msvc/package.json
@@ -1,6 +1,6 @@
 {
 	"name": "@lancedb/lancedb-win32-x64-msvc",
-	"version": "0.11.0-beta.1",
+	"version": "0.11.1-beta.0",
 	"os": ["win32"],
 	"cpu": ["x64"],
 	"main": "lancedb.win32-x64-msvc.node",
--- a/nodejs/package-lock.json
+++ b/nodejs/package-lock.json
@@ -1,12 +1,12 @@
 {
  "name": "@lancedb/lancedb",
-  "version": "0.11.0-beta.1",
+  "version": "0.11.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@lancedb/lancedb",
-      "version": "0.11.0-beta.1",
+      "version": "0.11.0",
      "cpu": [
        "x64",
        "arm64"
--- a/nodejs/package.json
+++ b/nodejs/package.json
@@ -10,7 +10,7 @@
    "vector database",
    "ann"
  ],
-  "version": "0.11.0-beta.1",
+  "version": "0.11.1-beta.0",
  "main": "dist/index.js",
  "exports": {
    ".": "./dist/index.js",
--- a/python/.bumpversion.toml
+++ b/python/.bumpversion.toml
@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.14.0"
+current_version = "0.14.1-beta.1"
 parse = """(?x)
    (?P<major>0|[1-9]\\d*)\\.
    (?P<minor>0|[1-9]\\d*)\\.
--- a/python/Cargo.toml
+++ b/python/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.14.0"
+version = "0.14.1-beta.1"
 edition.workspace = true
 description = "Python bindings for LanceDB"
 license.workspace = true
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -3,9 +3,8 @@ name = "lancedb"
 # version in Cargo.toml
 dependencies = [
    "deprecation",
-    "pylance==0.18.2",
+    "pylance==0.18.3-beta.2",
    "requests>=2.31.0",
-    "retry>=0.9.2",
    "tqdm>=4.27.0",
    "pydantic>=1.10",
    "attrs>=21.3.0",
--- a/python/python/lancedb/embeddings/utils.py
+++ b/python/python/lancedb/embeddings/utils.py
@@ -21,14 +21,35 @@ import time
 import urllib.error
 import weakref
 import logging
+from functools import wraps
 from typing import Callable, List, Union
 import numpy as np
 import pyarrow as pa
 from lance.vector import vec_to_table
-from retry import retry

 from ..util import deprecated, safe_import_pandas

+
+# ruff: noqa: PERF203
+def retry(tries=10, delay=1, max_delay=30, backoff=3, jitter=1):
+    def wrapper(fn):
+        @wraps(fn)
+        def wrapped(*args, **kwargs):
+            for i in range(tries):
+                try:
+                    return fn(*args, **kwargs)
+                except Exception:
+                    if i + 1 == tries:
+                        raise
+                    else:
+                        sleep = min(delay * (backoff**i) + jitter, max_delay)
+                        time.sleep(sleep)
+
+        return wrapped
+
+    return wrapper
+
+
 pd = safe_import_pandas()

 DATA = Union[pa.Table, "pd.DataFrame"]
--- a/python/python/lancedb/query.py
+++ b/python/python/lancedb/query.py
@@ -88,6 +88,11 @@ class Query(pydantic.BaseModel):
          tuning advice.
    offset: int
        The offset to start fetching results from
+    fast_search: bool
+        Skip a flat search of unindexed data. This will improve
+        search performance but search results will not include unindexed data.
+
+        - *default False*.
    """

    vector_column: Optional[str] = None
@@ -124,6 +129,8 @@ class Query(pydantic.BaseModel):

    offset: int = 0

+    fast_search: bool = False
+

 class LanceQueryBuilder(ABC):
    """An abstract query builder. Subclasses are defined for vector search,
@@ -139,6 +146,7 @@ class LanceQueryBuilder(ABC):
        vector_column_name: str,
        ordering_field_name: Optional[str] = None,
        fts_columns: Union[str, List[str]] = [],
+        fast_search: bool = False,
    ) -> LanceQueryBuilder:
        """
        Create a query builder based on the given query and query type.
@@ -155,6 +163,8 @@ class LanceQueryBuilder(ABC):
            If "auto", the query type is inferred based on the query.
        vector_column_name: str
            The name of the vector column to use for vector search.
+        fast_search: bool
+            Skip flat search of unindexed data.
        """
        # Check hybrid search first as it supports empty query pattern
        if query_type == "hybrid":
@@ -196,7 +206,9 @@ class LanceQueryBuilder(ABC):
        else:
            raise TypeError(f"Unsupported query type: {type(query)}")

-        return LanceVectorQueryBuilder(table, query, vector_column_name, str_query)
+        return LanceVectorQueryBuilder(
+            table, query, vector_column_name, str_query, fast_search
+        )

    @classmethod
    def _resolve_query(cls, table, query, query_type, vector_column_name):
@@ -565,6 +577,7 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
        query: Union[np.ndarray, list, "PIL.Image.Image"],
        vector_column: str,
        str_query: Optional[str] = None,
+        fast_search: bool = False,
    ):
        super().__init__(table)
        self._query = query
@@ -575,6 +588,7 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
        self._prefilter = False
        self._reranker = None
        self._str_query = str_query
+        self._fast_search = fast_search

    def metric(self, metric: Literal["L2", "cosine", "dot"]) -> LanceVectorQueryBuilder:
        """Set the distance metric to use.
@@ -675,6 +689,7 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
            vector_column=self._vector_column,
            with_row_id=self._with_row_id,
            offset=self._offset,
+            fast_search=self._fast_search,
        )
        result_set = self._table._execute_query(query, batch_size)
        if self._reranker is not None:
--- a/python/python/lancedb/remote/init.py
+++ b/python/python/lancedb/remote/init.py
@@ -50,6 +50,8 @@ class VectorQuery(BaseModel):

    vector_column: str = VECTOR_COLUMN_NAME

+    fast_search: bool = False
+

@attrs.define
 class VectorQueryResult:
--- a/python/python/lancedb/remote/table.py
+++ b/python/python/lancedb/remote/table.py
@@ -270,6 +270,7 @@ class RemoteTable(Table):
        vector_column_name: Optional[str] = None,
        query_type="auto",
        fts_columns: Optional[Union[str, List[str]]] = None,
+        fast_search: bool = False,
    ) -> LanceVectorQueryBuilder:
        """Create a search query to find the nearest neighbors
        of the given query vector. We currently support [vector search][search]
@@ -314,6 +315,12 @@ class RemoteTable(Table):
            - If the table has multiple vector columns then the *vector_column_name*
            needs to be specified. Otherwise, an error is raised.

+        fast_search: bool, optional
+            Skip a flat search of unindexed data. This may improve
+            search performance but search results will not include unindexed data.
+
+            - *default False*.
+
        Returns
        -------
        LanceQueryBuilder
@@ -343,6 +350,7 @@ class RemoteTable(Table):
            query_type,
            vector_column_name=vector_column_name,
            fts_columns=fts_columns,
+            fast_search=fast_search,
        )

    def _execute_query(
--- a/python/python/tests/test_embeddings.py
+++ b/python/python/tests/test_embeddings.py
@@ -11,6 +11,7 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 from typing import List, Union
+from unittest.mock import MagicMock, patch

 import lance
 import lancedb
@@ -25,6 +26,7 @@ from lancedb.embeddings import (
 )
 from lancedb.embeddings.base import TextEmbeddingFunction
 from lancedb.embeddings.registry import get_registry, register
+from lancedb.embeddings.utils import retry
 from lancedb.pydantic import LanceModel, Vector


@@ -225,3 +227,12 @@ def test_embedding_function_safe_model_dump(embedding_type):
                f"{embedding_type}: Private attribute '{key}' "
                f"is present in dumped model"
            )
+
+
+@patch("time.sleep")
+def test_retry(mock_sleep):
+    test_function = MagicMock(side_effect=[Exception] * 9 + ["result"])
+    test_function = retry()(test_function)
+    result = test_function()
+    assert mock_sleep.call_count == 9
+    assert result == "result"
--- a/rust/ffi/node/Cargo.toml
+++ b/rust/ffi/node/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-node"
-version = "0.11.0-beta.1"
+version = "0.11.1-beta.0"
 description = "Serverless, low-latency vector database for AI applications"
 license.workspace = true
 edition.workspace = true
--- a/rust/lancedb/Cargo.toml
+++ b/rust/lancedb/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "lancedb"
-version = "0.11.0-beta.1"
+version = "0.11.1-beta.0"
 edition.workspace = true
 description = "LanceDB: A serverless, low-latency vector database for AI applications"
 license.workspace = true
--- a/rust/lancedb/src/connection.rs
+++ b/rust/lancedb/src/connection.rs
@@ -196,22 +196,6 @@ impl<T: IntoArrow> CreateTableBuilder<true, T> {
        };
        Ok((data, builder))
    }
-
-    pub fn add_embedding(mut self, definition: EmbeddingDefinition) -> Result<Self> {
-        // Early verification of the embedding name
-        let embedding_func = self
-            .parent
-            .embedding_registry()
-            .get(&definition.embedding_name)
-            .ok_or_else(|| Error::EmbeddingFunctionNotFound {
-                name: definition.embedding_name.clone(),
-                reason: "No embedding function found in the connection's embedding_registry"
-                    .to_string(),
-            })?;
-
-        self.embeddings.push((definition, embedding_func));
-        Ok(self)
-    }
 }

 // Builder methods that only apply when we do not have initial data
@@ -329,6 +313,26 @@ impl<const HAS_DATA: bool, T: IntoArrow> CreateTableBuilder<HAS_DATA, T> {
        };
        self
    }
+
+    /// Add an embedding definition to the table.
+    ///
+    /// The `embedding_name` must match the name of an embedding function that
+    /// was previously registered with the connection's [`EmbeddingRegistry`].
+    pub fn add_embedding(mut self, definition: EmbeddingDefinition) -> Result<Self> {
+        // Early verification of the embedding name
+        let embedding_func = self
+            .parent
+            .embedding_registry()
+            .get(&definition.embedding_name)
+            .ok_or_else(|| Error::EmbeddingFunctionNotFound {
+                name: definition.embedding_name.clone(),
+                reason: "No embedding function found in the connection's embedding_registry"
+                    .to_string(),
+            })?;
+
+        self.embeddings.push((definition, embedding_func));
+        Ok(self)
+    }
 }

 #[derive(Clone, Debug)]
--- a/rust/lancedb/src/index.rs
+++ b/rust/lancedb/src/index.rs
@@ -144,7 +144,7 @@ impl std::str::FromStr for IndexType {
            "BTREE" => Ok(Self::BTree),
            "BITMAP" => Ok(Self::Bitmap),
            "LABEL_LIST" | "LABELLIST" => Ok(Self::LabelList),
-            "FTS" => Ok(Self::FTS),
+            "FTS" | "INVERTED" => Ok(Self::FTS),
            "IVF_PQ" => Ok(Self::IvfPq),
            "IVF_HNSW_PQ" => Ok(Self::IvfHnswPq),
            "IVF_HNSW_SQ" => Ok(Self::IvfHnswSq),
--- a/rust/lancedb/src/table.rs
+++ b/rust/lancedb/src/table.rs
@@ -2110,7 +2110,6 @@ mod tests {
    use arrow_schema::{DataType, Field, Schema, TimeUnit};
    use futures::TryStreamExt;
    use lance::dataset::{Dataset, WriteMode};
-    use lance::index::DatasetIndexInternalExt;
    use lance::io::{ObjectStoreParams, WrappingObjectStore};
    use rand::Rng;
    use tempfile::tempdir;
@@ -3002,22 +3001,8 @@ mod tests {
        let index_configs = table.list_indices().await.unwrap();
        assert_eq!(index_configs.len(), 1);
        let index = index_configs.into_iter().next().unwrap();
-        // TODO: Fix via https://github.com/lancedb/lance/issues/2039
-        // assert_eq!(index.index_type, crate::index::IndexType::Bitmap);
+        assert_eq!(index.index_type, crate::index::IndexType::Bitmap);
        assert_eq!(index.columns, vec!["category".to_string()]);
-
-        // For now, just open the index to verify its type
-        let lance_dataset = table.as_native().unwrap().dataset.get().await.unwrap();
-        let indices = lance_dataset
-            .load_indices_by_name(&index.name)
-            .await
-            .unwrap();
-        let index_meta = &indices[0];
-        let idx = lance_dataset
-            .open_scalar_index("category", &index_meta.uuid.to_string())
-            .await
-            .unwrap();
-        assert_eq!(idx.index_type(), IndexType::Bitmap);
    }

    #[tokio::test]
@@ -3086,22 +3071,57 @@ mod tests {
        let index_configs = table.list_indices().await.unwrap();
        assert_eq!(index_configs.len(), 1);
        let index = index_configs.into_iter().next().unwrap();
-        // TODO: Fix via https://github.com/lancedb/lance/issues/2039
-        // assert_eq!(index.index_type, crate::index::IndexType::LabelList);
+        assert_eq!(index.index_type, crate::index::IndexType::LabelList);
        assert_eq!(index.columns, vec!["tags".to_string()]);
+    }

-        // For now, just open the index to verify its type
-        let lance_dataset = table.as_native().unwrap().dataset.get().await.unwrap();
-        let indices = lance_dataset
-            .load_indices_by_name(&index.name)
+    #[tokio::test]
+    async fn test_create_inverted_index() {
+        let tmp_dir = tempdir().unwrap();
+        let uri = tmp_dir.path().to_str().unwrap();
+
+        let conn = ConnectBuilder::new(uri).execute().await.unwrap();
+        const WORDS: [&str; 3] = ["cat", "dog", "fish"];
+        let mut text_builder = StringBuilder::new();
+        let num_rows = 120;
+        for i in 0..num_rows {
+            text_builder.append_value(WORDS[i % 3]);
+        }
+        let text = Arc::new(text_builder.finish());
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("id", DataType::Int32, false),
+            Field::new("text", DataType::Utf8, true),
+        ]));
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(Int32Array::from_iter_values(0..num_rows as i32)),
+                text,
+            ],
+        )
+        .unwrap();
+
+        let table = conn
+            .create_table(
+                "test_bitmap",
+                RecordBatchIterator::new(vec![Ok(batch.clone())], batch.schema()),
+            )
+            .execute()
            .await
            .unwrap();
-        let index_meta = &indices[0];
-        let idx = lance_dataset
-            .open_scalar_index("tags", &index_meta.uuid.to_string())
+
+        table
+            .create_index(&["text"], Index::FTS(Default::default()))
+            .execute()
            .await
            .unwrap();
-        assert_eq!(idx.index_type(), IndexType::LabelList);
+        let index_configs = table.list_indices().await.unwrap();
+        assert_eq!(index_configs.len(), 1);
+        let index = index_configs.into_iter().next().unwrap();
+        assert_eq!(index.index_type, crate::index::IndexType::FTS);
+        assert_eq!(index.columns, vec!["text".to_string()]);
+        assert_eq!(index.name, "text_idx");
    }

    #[tokio::test]
--- a/rust/lancedb/src/table/dataset.rs
+++ b/rust/lancedb/src/table/dataset.rs
@@ -54,9 +54,7 @@ impl DatasetRef {
                last_consistency_check,
                ..
            } => {
-                *dataset = dataset
-                    .checkout_version(dataset.latest_version_id().await?)
-                    .await?;
+                dataset.checkout_latest().await?;
                last_consistency_check.replace(Instant::now());
            }
            Self::TimeTravel { dataset, version } => {
Author	SHA1	Message	Date
Lance Release	5517e102c3	Bump version: 0.14.1-beta.0 → 0.14.1-beta.1	2024-10-23 00:33:40 +00:00
Will Jones	82197c54e4	perf: eliminate iop in refresh (#1760 ) Closes #1741 If we checkout a version, we need to make a `HEAD` request to get the size of the manifest. The new `checkout_latest()` code path can skip this IOP. This makes the refresh slightly faster.	2024-10-18 13:40:24 -07:00
Will Jones	48f46d4751	docs(node): update `indexStats` signature and regenerate docs (#1742 ) `indexStats` still referenced UUID even though in https://github.com/lancedb/lancedb/pull/1702 we changed it to take name instead.	2024-10-18 10:53:28 -07:00
Lance Release	437316cbbc	Updating package-lock.json	2024-10-17 18:59:18 +00:00
Lance Release	d406eab2c8	Bump version: 0.11.0 → 0.11.1-beta.0	2024-10-17 18:59:01 +00:00
Lance Release	1f41101897	Bump version: 0.14.0 → 0.14.1-beta.0	2024-10-17 18:58:45 +00:00
Will Jones	99e4db0d6a	feat(rust): allow `add_embedding` on `create_empty_table` (#1754 ) Fixes https://github.com/lancedb/lancedb/issues/1750	2024-10-17 11:58:15 -07:00
Will Jones	46486d4d22	fix: list_indices can handle fts indexes (#1753 ) Fixes #1752	2024-10-16 10:39:40 -07:00
Weston Pace	f43cb8bba1	feat: upgrade lance to 0.18.3 (#1748 )	2024-10-16 00:48:31 -07:00
James Wu	38eb05f297	fix(python): remove dependency on retry package (#1749 ) ## user story fixes https://github.com/lancedb/lancedb/issues/1480 https://github.com/invl/retry has not had an update in 8 years, one if its sub-dependencies via requirements.txt (https://github.com/pytest-dev/py) is no longer maintained and has a high severity vulnerability (CVE-2022-42969). retry is only used for a single function in the python codebase for a deprecated helper function `with_embeddings`, which was created for an older tutorial (https://github.com/lancedb/lancedb/pull/12) [but is now deprecated](https://lancedb.github.io/lancedb/embeddings/legacy/). ## changes i backported a limited range of functionality of the `@retry()` decorator directly into lancedb so that we no longer have a dependency to the `retry` package. ## tests ``` /Users/james/src/lancedb/python $ ruff check . All checks passed! /Users/james/src/lancedb/python $ pytest python/tests/test_embeddings.py python/tests/test_embeddings.py .......s.... [100%] ================================================================ 11 passed, 1 skipped, 2 warnings in 7.08s ================================================================ ```	2024-10-15 15:13:57 -07:00
Ryan Green	679a70231e	feat: allow fast_search on python remote table (#1747 ) Add `fast_search` parameter to query builder and remote table to support skipping flat search in remote search	2024-10-14 14:39:54 -06:00
Dominik Weckmüller	e7b56b7b2a	docs: add permanent link chain icon to headings without impacting SEO (#1746 ) I noted that there are no permanent links in the docs. Adapted the current best solution from https://github.com/squidfunk/mkdocs-material/discussions/3535. It adds a GitHub-like chain icon to the left of each heading (right on mobile) and does not impact SEO unlike the default solution with pilcrow char `¶` that might show up on google search results. <img alt="image" src="https://user-images.githubusercontent.com/182589/153004627-6df3f8e9-c747-4f43-bd62-a8dabaa96c3f.gif">	2024-10-14 11:58:23 -07:00
Olzhas Alexandrov	5ccd0edec2	docs: clarify infrastructure requirements for S3 Express One Zone (#1745 )	2024-10-11 14:06:28 -06:00
Will Jones	9c74c435e0	ci: update package lock (#1740 )	2024-10-09 15:14:08 -06:00
Lance Release	6de53ce393	Updating package-lock.json	2024-10-09 18:54:29 +00:00
Lance Release	9f42fbba96	Bump version: 0.11.0-beta.2 → 0.11.0	2024-10-09 18:54:09 +00:00
Lance Release	d892f7a622	Bump version: 0.11.0-beta.1 → 0.11.0-beta.2	2024-10-09 18:54:04 +00:00