rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2025-12-27 15:12:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,057 Commits 117 Branches 395 Tags
python-v0.9.0-patch
Commit Graph

1057 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Weston Pace
9fee384d2c chore(node): restore package-lock.json lost during rebase 2024-04-05 16:36:29 -07:00
Ayush Chaurasia
b2952acca7 chore(python): remove redundant files (#1203) 2024-04-05 16:35:10 -07:00
Pranav Maddi
2b132a0bef Fix markdown formatting (#1188) 2024-04-05 16:35:10 -07:00
Will Jones
ba56208a34 ci: fix job (#1193) 2024-04-05 16:35:10 -07:00
Ayush Chaurasia
2d2042d59e chore(python): Remove settings manager and telemetry. (#1198) 
This PR is intended to remove settings manager. But because telemetry
and CLI depends on settings manager those need to go too.
 2024-04-05 16:35:09 -07:00
Raghav Dixit
1c41a00d87 Embeddings: HF model hub support added via transformers (#1154) 2024-04-05 16:34:56 -07:00
Lance Release
ac63d4066b Updating package-lock.json 2024-04-05 16:34:53 -07:00
Lance Release
be2074b90d [python] Bump version: 0.6.6 → 0.6.7 2024-04-05 16:34:53 -07:00
Lance Release
6c452f29e9 Bump version: 0.4.15 → 0.4.16 2024-04-05 16:34:50 -07:00
Will Jones
8a7ded23b2 chore: upgrade to lance-0.10.9 (#1192) 2024-04-05 16:34:50 -07:00
QianZhu
871500db70 add a default value for search.limit to be consistent with python sdk (#1191) 
Changed the default value for search.limit to be 10
 2024-04-05 16:34:50 -07:00
Bert
a900bc0827 ensure table names are uri encoded for tables (#1189) 
This prevents an issue where users can do something like:
```js
db.createTable('my-table#123123')
```
The server has logic to determine that '#' character is not allowed in
the table name, but currently this is being returned as 404 error
because it routes to `/v1/my-table#123123/create` and `#123123/create`
will not be parsed as part of path
 2024-04-05 16:34:50 -07:00
Will Jones
47cff963c5 feat: ship fp16kernels in Python wheels (#1148) 
Same deal as https://github.com/lancedb/lance/pull/2098
 2024-04-05 16:34:50 -07:00
Lei Xu
e6ff3d848b chore: bump to 0.10.8 (#1187) 2024-04-05 16:34:50 -07:00
QianZhu
44d799ebb8 bug: fix the return value of countRows (#1186) 2024-04-05 16:34:50 -07:00
Lei Xu
1d3325dcc5 chore: bump lance version (#1185) 
Bump lance version to `0.10.7`
 2024-04-05 16:34:50 -07:00
Bert
ff45f25cf2 fix error decoding in nodejs client (#1184) 
fixes: #1183
 2024-04-05 16:34:50 -07:00
QianZhu
a34cc770c5 remote count_rows need to return the number (#1181) 2024-04-05 16:34:50 -07:00
eduardjbotha
f749b8808f SQL Documentation includes DataFusion functions (#1179) 
Show that it is possible to use the DataFusion functions in the `WHERE`
clause.

Co-authored-by: Eduard Botha <eduard.botha@inovex.de>
 2024-04-05 16:34:50 -07:00
Lei Xu
7e5a54b76a chore: add social link footer (#1177) 2024-04-05 16:34:50 -07:00
Lei Xu
3f14938392 chore: pass str instead of String to build table names (#1178) 2024-04-05 16:34:50 -07:00
Lance Release
3bd16e1b14 Updating package-lock.json 2024-04-05 16:34:46 -07:00
QianZhu
2f89fc26f1 feat: add filterable countRows to remote API (#1169) 2024-04-05 16:34:46 -07:00
Lance Release
e5bfec4318 [python] Bump version: 0.6.5 → 0.6.6 2024-04-05 16:34:46 -07:00
Lance Release
e0f50013ea Bump version: 0.4.14 → 0.4.15 2024-04-05 16:34:39 -07:00
Weston Pace
e4e64f9d6b chore: bump lance version to 0.10.6 (#1175) 2024-04-05 16:34:39 -07:00
Bert
6c9f4c4304 Update LanceDB Logo in README (#1167) 
<img width="1034" alt="image"
src="https://github.com/lancedb/lancedb/assets/5846846/5b8aa53c-4d93-4c0e-bed4-80c238b319ba">
 2024-04-05 16:34:39 -07:00
Weston Pace
e21b56293c docs: add a reference to @lancedb/lance in the docs (#1166) 
We aren't yet ready to switch over the examples since almost all JS
examples rely on embeddings and we haven't yet ported those over.
However, this makes it possible for those that are interested to start
using `@lancedb/lancedb`
 2024-04-05 16:34:39 -07:00
Will Jones
1b0aaf9ec3 ci: fix name collision in npm artifacts for vectordb (#1164) 
Fixes #1163
 2024-04-05 16:34:39 -07:00
Weston Pace
01239da082 chore: add nodejs to bumpversion (#1161) 
The previous release failed to release nodejs because the nodejs version
wasn't bumped. This should fix that.
 2024-04-05 16:34:39 -07:00
Weston Pace
6060c0cd36 chore: fix clippy (#1162) 2024-04-05 16:34:38 -07:00
Bert
bb179981dd added new logo to vercel example gif (#1158) 2024-04-05 16:34:38 -07:00
Bert
2e1f1c6d5d New logo on docs site (#1157) 2024-04-05 16:34:38 -07:00
Ayush Chaurasia
b916f5f132 docs: Add all available HF/sentence transformers embedding models list (#1134) 
Solves -  https://github.com/lancedb/lancedb/issues/968
 2024-04-05 16:34:38 -07:00
Weston Pace
f97c7dad8c docs: add the async python API to the docs (#1156) 2024-04-05 16:34:37 -07:00
Lance Release
ccf13f15d4 Bump version: 0.4.13 → 0.4.14 2024-04-05 16:33:37 -07:00
Weston Pace
287c5ca2f9 feat: add publish step for nodejs (#1155) 
This will start publishing `@lancedb/lancedb` with the new nodejs
package on our releases.
 2024-04-05 16:33:37 -07:00
Pranav Maddi
479289dd38 Adds a Ask LanceDB button to docs. (#1150) 
This links out to the new [asklancedb.com](https://asklancedb.com) page.

Screenshots of the change:

![Quick start - LanceDB · 10 20am ·
03-22](https://github.com/lancedb/lancedb/assets/2371511/c45ba893-fc74-4957-bdd3-3712b351aff3)
![Quick start -
LanceDB](https://github.com/lancedb/lancedb/assets/2371511/d4762eb6-52af-4fd5-857e-3ed280716999)
 2024-04-05 16:33:37 -07:00
Bert
1e41232f28 Node SDK Client middleware for HTTP Requests (#1130) 
Adds client-side middleware to LanceDB Node SDK to instrument HTTP
Requests

Example - adding `x-request-id` request header:
```js
class HttpMiddleware {
    constructor({ requestId }) {
        this.requestId = requestId
    }

    onRemoteRequest(req, next) {
        req.headers['x-request-id'] = this.requestId
        return next(req)
    }
}

const db = await lancedb.connect({
  uri: 'db://remote-123',
  apiKey: 'sk_...',
})

let tables = await db.withMiddleware(new HttpMiddleware({ requestId: '123' })).tableNames();

```

---------

Co-authored-by: Weston Pace <weston.pace@gmail.com>
 2024-04-05 16:33:37 -07:00
QianZhu
db2631c2ad remove warnings (#1147) 2024-04-05 16:33:37 -07:00
Lei Xu
473ef7e426 chore: validate table name (#1146) 
Closes #1129
 2024-04-05 16:33:37 -07:00
Lance Release
d32dc84653 [python] Bump version: 0.6.4 → 0.6.5 2024-04-05 16:33:37 -07:00
Lei Xu
1aaaeff511 chore: bump lance to 0.10.5 (#1145) 2024-04-05 16:33:37 -07:00
QianZhu
bdd07a5dfa fix nodejs test (#1141) 
changed the error msg for query with wrong vector dim thus need this
change to pass the nodejs tests.
 2024-04-05 16:33:37 -07:00
QianZhu
63db51c90d better error msg for query vector with wrong dim (#1140) 2024-04-05 16:33:37 -07:00
Ishani Ghose
0838e12b30 feat: add to_batches API #805 (#1048) 
SDK
Python

Description
Exposes pyarrow batch api during query execution - relevant when there
is no vector search query, dataset is large and the filtered result is
larger than memory.

---------

Co-authored-by: Ishani Ghose <isghose@amazon.com>
Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>
 2024-04-05 16:33:37 -07:00
Weston Pace
968c62cb8f feat: introduce ArrowNative wrapper struct for adding data that is already a RecordBatchReader (#1139) 
In
2de226220b
I added a new `IntoArrow` trait for adding data into a table.
Unfortunately, it seems my approach for implementing the trait for
"things that are already record batch readers" was flawed. This PR
corrects that flaw and, conveniently, removes the need to box readers at
all (though it is ok if you do).
 2024-04-05 16:33:37 -07:00
natcharacter
f6e9f8e3f4 Order by field support FTS (#1132) 
This PR adds support for passing through a set of ordering fields at
index time (unsigned ints that tantivity can use as fast_fields) that at
query time you can sort your results on. This is useful for cases where
you want to get related hits, i.e by keyword, but order those hits by
some other score, such as popularity.

I.e search for songs descriptions that match on "sad AND jazz AND 1920"
and then order those by number of times played. Example usage can be
seen in the fts tests.

---------

Co-authored-by: Nat Roth <natroth@Nats-MacBook-Pro.local>
Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>
 2024-04-05 16:33:36 -07:00
Chang She
4466cfa958 feat(python): support writing huggingface dataset and dataset dict (#1110) 
HuggingFace Dataset is written as arrow batches.
For DatasetDict, all splits are written with a "split" column appended.

- [x] what if the dataset schema already has a `split` column
- [x] add unit tests
 2024-04-05 16:33:06 -07:00
Ayush Chaurasia
42fad84ec8 feat(python): Support reranking for vector and fts (#1103) 
solves https://github.com/lancedb/lancedb/issues/1086

Usage Reranking with FTS:
```
retriever = db.create_table("fine-tuning", schema=Schema, mode="overwrite")
pylist = [{"text": "Carson City is the capital city of the American state of Nevada. At the  2010 United States Census, Carson City had a population of 55,274."},
          {"text": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan."},
        {"text": "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas."},
        {"text": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. "},
        {"text": "Capital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."},
        {"text": "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck."},
        ]
retriever.add(pylist)
retriever.create_fts_index("text", replace=True)

query = "What is the capital of the United States?"
reranker = CohereReranker(return_score="all")
print(retriever.search(query, query_type="fts").limit(10).to_pandas())
print(retriever.search(query, query_type="fts").rerank(reranker=reranker).limit(10).to_pandas())
```
Result
```
                                                text                                             vector     score
0  Capital punishment (the death penalty) has exi...  [0.099975586, 0.047943115, -0.16723633, -0.183...  0.729602
1  Charlotte Amalie is the capital and largest ci...  [-0.021255493, 0.03363037, -0.027450562, -0.17...  0.678046
2  The Commonwealth of the Northern Mariana Islan...  [0.3684082, 0.30493164, 0.004600525, -0.049407...  0.671521
3  Carson City is the capital city of the America...  [0.13989258, 0.14990234, 0.14172363, 0.0546569...  0.667898
4  Washington, D.C. (also known as simply Washing...  [-0.0090408325, 0.42578125, 0.3798828, -0.3574...  0.653422
5  North Dakota is a state in the United States. ...  [0.55859375, -0.2109375, 0.14526367, 0.1634521...  0.639346
                                                text                                             vector     score  _relevance_score
0  Washington, D.C. (also known as simply Washing...  [-0.0090408325, 0.42578125, 0.3798828, -0.3574...  0.653422          0.979977
1  The Commonwealth of the Northern Mariana Islan...  [0.3684082, 0.30493164, 0.004600525, -0.049407...  0.671521          0.299105
2  Capital punishment (the death penalty) has exi...  [0.099975586, 0.047943115, -0.16723633, -0.183...  0.729602          0.284874
3  Carson City is the capital city of the America...  [0.13989258, 0.14990234, 0.14172363, 0.0546569...  0.667898          0.089614
4  North Dakota is a state in the United States. ...  [0.55859375, -0.2109375, 0.14526367, 0.1634521...  0.639346          0.063832
5  Charlotte Amalie is the capital and largest ci...  [-0.021255493, 0.03363037, -0.027450562, -0.17...  0.678046          0.041462
```

## Vector Search usage:
```
query = "What is the capital of the United States?"
reranker = CohereReranker(return_score="all")
print(retriever.search(query).limit(10).to_pandas())
print(retriever.search(query).rerank(reranker=reranker, query=query).limit(10).to_pandas()) # <-- Note: passing extra string query here
```

Results
```
                                                text                                             vector  _distance
0  Capital punishment (the death penalty) has exi...  [0.099975586, 0.047943115, -0.16723633, -0.183...  39.728973
1  Washington, D.C. (also known as simply Washing...  [-0.0090408325, 0.42578125, 0.3798828, -0.3574...  41.384884
2  Carson City is the capital city of the America...  [0.13989258, 0.14990234, 0.14172363, 0.0546569...  55.220200
3  Charlotte Amalie is the capital and largest ci...  [-0.021255493, 0.03363037, -0.027450562, -0.17...  58.345654
4  The Commonwealth of the Northern Mariana Islan...  [0.3684082, 0.30493164, 0.004600525, -0.049407...  60.060867
5  North Dakota is a state in the United States. ...  [0.55859375, -0.2109375, 0.14526367, 0.1634521...  64.260544
                                                text                                             vector  _distance  _relevance_score
0  Washington, D.C. (also known as simply Washing...  [-0.0090408325, 0.42578125, 0.3798828, -0.3574...  41.384884          0.979977
1  The Commonwealth of the Northern Mariana Islan...  [0.3684082, 0.30493164, 0.004600525, -0.049407...  60.060867          0.299105
2  Capital punishment (the death penalty) has exi...  [0.099975586, 0.047943115, -0.16723633, -0.183...  39.728973          0.284874
3  Carson City is the capital city of the America...  [0.13989258, 0.14990234, 0.14172363, 0.0546569...  55.220200          0.089614
4  North Dakota is a state in the United States. ...  [0.55859375, -0.2109375, 0.14526367, 0.1634521...  64.260544          0.063832
5  Charlotte Amalie is the capital and largest ci...  [-0.021255493, 0.03363037, -0.027450562, -0.17...  58.345654          0.041462
```
 2024-04-05 16:33:06 -07:00