Prior to this PR, to get a new version of a table, we need to re-open
the table. This has a few downsides w.r.t. performance:
* Object store is recreated, which takes time and throws away existing
warm connections
* Commit handler is thrown aways as well, which also may contain warm
connections
in #486 `connect` started converting path into uri. However, the PR
didn't handle relative path and appended `file://` to relative path.
This PR changes the parsing strat to be more rational. If a path is
provided instead of url, we do not try anythinng special.
engine and engine params may only be specified when a url with schema is
provided
Co-authored-by: rmeng <rob@lancedb.com>
# WARNING: specifying engine is NOT a publicly supported feature in
lancedb yet. THE API WILL CHANGE.
This PR exposes dynamodb based commit to `vectordb` and JS SDK (will do
python in another PR since it's on a different release track)
This PR also added aws integration test using `localstack`
## What?
This PR adds uri parameters to DB connection string. User may specify
`engine` in the connection string to let LanceDB know that the user
wants to use an external store when reading and writing a table. User
may also pass any parameters required by the commitStore in the
connection string, these parameters will be propagated to lance.
e.g.
```
vectordb.connect("s3://my-db-bucket?engine=ddb&ddbTableName=my-commit-table")
```
will automatically convert table path to
```
s3+ddb://my-db-bucket/my_table.lance?&ddbTableName=my-commit-table
```
This reverts commit 87e9a0250f.
I triggered the nodejs release commit GHA by mistake. Reverting it.
The tag will be removed manually.
Co-authored-by: Chang She <chang@lancedb.com>
- Supports creating an empty table as long as an Arrow Schema is provided
- Supports creating a table from an Arrow Table (can be passed as data)
- Simplified some Arrow code in the TS/FFI side
- removed createTableArrow method, it was never documented / tested.
- Callers / SDKs are responsible for keeping track of the last version of the Table
- Remove the mpsc from Table and make all Table operations non-blocking
- Moved computation out of JS main thread by using a mpsc
- Removes the Arc/Mutex since Table is owned by JsTable now
- Moved table / query methods to their own files
- Fixed js-transformers example
- Implement Result/Error in the node FFI
- Implement a trait (ResultExt) to make error handling less verbose
- Refactor some parts of the code that touch arrow into arrow.rs
