rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-01-09 05:12:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
798 Commits 115 Branches 397 Tags
562d1af1ede86fc66d40d906cc9333ec34dd20a0
Commit Graph

97 Commits

Author SHA1 Message Date
Will Jones
c5b0934bfb feat(node): add read_consistency_interval to Node and Rust (#1002) 
This PR adds the same consistency semantics as was added in #828. It
*does not* add the same lazy-loading of tables, since that breaks some
existing tests.

This closes #998.

---------

Co-authored-by: Weston Pace <weston.pace@gmail.com>
 2024-04-05 16:30:40 -07:00
Weston Pace
9241f47f0e feat: make it easier to create empty tables (#942) 
This PR also reworks the table creation utilities significantly so that
they are more consistent, built on top of each other, and thoroughly
documented.
 2024-04-05 16:30:30 -07:00
Will Jones
68115f1369 fix: wrap in BigInt to avoid upstream bug (#962) 
Closes #960
 2024-04-05 16:30:30 -07:00
Weston Pace
41ccb48160 feat: add support for filter during merge insert when matched (#948) 
Closes #940
 2024-04-05 16:29:58 -07:00
Weston Pace
138fc3f66b feat: add a filterable count_rows to all the lancedb APIs (#913) 
A `count_rows` method that takes a filter was recently added to
`LanceTable`. This PR adds it everywhere else except `RemoteTable` (that
will come soon).
 2024-04-05 16:29:58 -07:00
Weston Pace
18f7bad3dd feat: add merge_insert to the node and rust APIs (#915) 2024-04-05 16:29:05 -07:00
Lei Xu
8e139012e2 fix(node): pass AWS credentials to db level operations (#908) 
Passed the following tests

```ts
const keyId = process.env.AWS_ACCESS_KEY_ID;
const secretKey = process.env.AWS_SECRET_ACCESS_KEY;
const sessionToken = process.env.AWS_SESSION_TOKEN;
const region = process.env.AWS_REGION;

const db = await lancedb.connect({
  uri: "s3://bucket/path",
  awsCredentials: {
    accessKeyId: keyId,
    secretKey: secretKey,
    sessionToken: sessionToken,
  },
  awsRegion: region,
} as lancedb.ConnectionOptions);

  console.log(await db.createTable("test", [{ vector: [1, 2, 3] }]));
  console.log(await db.tableNames());
  console.log(await db.dropTable("test"))
```
 2024-04-05 16:28:56 -07:00
Lei Xu
e7fdb931de chore: convert all js doc test to use snippet. (#881) 2024-04-05 16:28:56 -07:00
Lei Xu
a192c1a9b1 chore(rust): simplified version of optimize (#869) 
Consolidate various optimize() into one method, similar to postgres
VACCUM in the process of preparing Rust API for public use
 2024-04-05 16:28:18 -07:00
Lei Xu
65c1d8bc4c feat: change create table to accept Arrow table (#845) 2024-04-05 16:27:50 -07:00
Bert
a409000c6f allow passing api key as env var (#841) 
Allow passing API key as env var:
```shell
export LANCEDB_API_KEY=sh_123...
```

with this set, apiKey argument can omitted from `connect`
```js
    const db = await vectordb.connect({
        uri: "db://test-proj-01-ae8343",
        region: "us-east-1",
  })
```
```py
    db = lancedb.connect(
        uri="db://test-proj-01-ae8343",
        region="us-east-1",
    )
```
 2024-04-05 16:27:42 -07:00
Lei Xu
d8befeeea2 feat(js): add helper function to create Arrow Table with schema (#838) 
Support to make Apache Arrow Table from an array of javascript Records,
with optionally provided Schema.
 2024-04-05 16:27:42 -07:00
Chang She
b699b5c42b chore(js): remove errant console.log (#834) 2024-04-05 16:27:42 -07:00
Chang She
d19bf80375 Merge branch 'tecmie/embeddings-openai' of github.com:tecmie/lancedb into tecmie-tecmie/embeddings-openai 2024-04-05 16:27:41 -07:00
Lei Xu
5b2c602fb3 doc: improve docs for nodejs connect functions (#833) 
* improve the docstring for NodeJS connect functions and
`ConnectOptions` parameters.
* Simplify `npm run build` steps.
 2024-04-05 16:27:32 -07:00
Andrew Miracle
8daed93a91 eslint fix 2024-04-05 16:25:52 -07:00
Andrew Miracle
fa13fb9392 rebase from lancedb/main 2024-04-05 16:25:14 -07:00
Chang She
118a11c9b3 feat(node): align incoming data to table schema (#802) 2024-04-05 16:25:14 -07:00
Chang She
a758876a65 feat(node): support table.schema for LocalTable (#789) 
Close #773 

we pass an empty table over IPC so we don't need to manually deal with
serde. Then we just return the schema attribute from the empty table.

---------

Co-authored-by: albertlockett <albert.lockett@gmail.com>
 2024-04-05 16:25:14 -07:00
Chang She
81487f10fe feat(js): support list of string input (#755) 
Add support for adding lists of string input (e.g., list of categorical
labels)

Follow-up items: #757 #758
 2024-04-05 16:25:02 -07:00
Aidan
a76b5755ff fix: createIndex index cache size (#741) 2024-04-05 16:25:02 -07:00
Andrew Miracle
5948f11641 eslint fix 2024-04-05 16:25:02 -07:00
Andrew Miracle
9efc3fa6d8 remove console logs 2024-04-05 16:25:02 -07:00
Andrew Miracle
453bf113ae add support for openai SDK version ^4.24.1 2024-04-05 16:25:02 -07:00
Chang She
4b243c5ff8 feat(node): align incoming data to table schema (#802) 2024-04-05 16:25:01 -07:00
Chang She
175ad9223b feat(node): support table.schema for LocalTable (#789) 
Close #773 

we pass an empty table over IPC so we don't need to manually deal with
serde. Then we just return the schema attribute from the empty table.

---------

Co-authored-by: albertlockett <albert.lockett@gmail.com>
 2024-04-05 16:25:01 -07:00
Bert
5bb128a24d docs: fix JS api docs for update method (#738) 2024-04-05 16:24:47 -07:00
Weston Pace
94e81ff84b feat: add the ability to create scalar indices (#679) 
This is a pretty direct binding to the underlying lance capability
 2024-04-05 16:24:47 -07:00
Aidan
b4ae3f3097 feat: node list tables pagination (#733) 2024-04-05 16:24:47 -07:00
Chang She
cd791a366b feat(js): support list of string input (#755) 
Add support for adding lists of string input (e.g., list of categorical
labels)

Follow-up items: #757 #758
 2024-04-05 16:24:47 -07:00
Aidan
e74c203e6f fix: createIndex index cache size (#741) 2024-04-05 16:24:47 -07:00
Aidan
3232b55218 feat: Node create index API (#720) 2024-04-05 16:24:30 -07:00
Aidan
ee2034db23 feat: Node Schema API (#717) 2024-04-05 16:24:30 -07:00
Bert
eb62ddfb0c implement update for remote clients (#706) 2024-04-05 16:24:30 -07:00
Chang She
b02370cacd feat: LocalTable for vectordb now supports filters without vector search (#693) 
Note this currently the filter/where is only implemented for LocalTable
so that it requires an explicit cast to "enable" (see new unit test).
The alternative is to add it to the Table interface, but since it's not
available on RemoteTable this may cause some user experience issues.
 2024-04-05 16:24:15 -07:00
Bert
e479acc1bd Update in Node & Rust (#696) 
Co-authored-by: Will Jones <willjones127@gmail.com>
 2024-04-05 16:24:15 -07:00
Rob Meng
d74e188f80 fix: fix passing prefilter flag to remote client (#677) 
was passing this at the wrong position
 2024-04-05 16:23:49 -07:00
Rob Meng
59c25574f0 feat: enable prefilter in node js (#675) 
enable prefiltering in node js, both native and remote
 2024-04-05 16:23:49 -07:00
Will Jones
6d76fe80b8 chore: upgrade lance to v0.8.17 (#656) 
Readying for the next Lance release.
 2024-04-05 16:23:49 -07:00
Bert
22fc0eaaf6 fix: node remote implement table.countRows (#648) 2024-04-05 16:23:49 -07:00
Bert
49a366bc74 fix: node send db header for GET requests (#646) 2024-04-05 16:23:49 -07:00
Bert
20ab85171b fix: node remote connection handles non http errors (#624) 
https://github.com/lancedb/lancedb/issues/623

Fixes issue trying to print response status when using remote client. If
the error is not an HTTP error (e.g. dns/network failure), there won't
be a response.
 2024-04-05 16:23:14 -07:00
Bert
4d086d63eb feat: added dataset stats api to node (#604) 2024-04-05 16:22:59 -07:00
Rob Meng
2826bc7f1a feat: include manifest files in mirrow store (#589) 2024-04-05 16:22:59 -07:00
Rob Meng
c14ccbd334 implement remote api calls for table mutation (#567) 
Add more APIs to remote table for Node SDK
* `add` rows
* `overwrite` table with rows
* `create` table

This has been tested against dev stack
 2024-04-05 16:22:59 -07:00
Lei Xu
8dc10180b0 feat(python,js): deletion operation on remote tables (#568) 2024-04-05 16:22:59 -07:00
Will Jones
c07207c661 feat: cleanup and compaction (#518) 
#488
 2024-04-05 16:22:59 -07:00
Rob Meng
1db66c6980 implement mirroring object store (#537) 
This PR implements a mirroring object store and allows and table to be
mirrored to a local path when param `mirroredStore` is set in the url
 2023-10-04 21:23:34 -04:00
Rob Meng
dbf37a0434 fix: upgrade lance to 0.7.5 and add tests for searching empty dataset (#505) 
This PR upgrade lance to `0.7.5`, which include fixes for searching an
empty dataset.

This PR also adds two tests in node SDK to make sure searching empty
dataset do no throw

Co-authored-by: rmeng <rob@lancedb.com>
 2023-09-18 22:12:11 -07:00
Rob Meng
731f86e44c add health check to wait for all service ready before next step (#501) 
aws integration tests are flaky because we didn't wait for the services
to become healthy. (we only waited for the localstack service, this PR
adds wait for sub services)
 2023-09-18 15:17:45 -04:00