916 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
albertlockett	ada28d3f81	implement display for copy behavior	2024-04-02 15:16:22 -04:00
albertlockett	f6bd785398	tests	2024-04-02 12:07:23 -04:00
albertlockett	5b490d08dd	add checked copy and new secondary copy behaviour	2024-04-02 10:30:09 -04:00
Lei Xu	46e95f2c4c	chore: add social link footer (#1177)	2024-04-01 22:09:27 -07:00
Lei Xu	73810b4410	chore: pass str instead of String to build table names (#1178)	2024-04-01 21:31:07 -07:00
Lance Release	09280bc54a	Updating package-lock.json	2024-04-02 03:03:07 +00:00
Lance Release	5603f1e57f	Updating package-lock.json	2024-04-02 02:28:04 +00:00
QianZhu	1d67615cff	feat: add filterable countRows to remote API (#1169)	2024-04-01 14:31:15 -07:00
Lance Release	05f484b716	[python] Bump version: 0.6.5 → 0.6.6 python-v0.6.6	2024-04-01 19:09:01 +00:00
Lance Release	7e92aa657a	Updating package-lock.json	2024-04-01 18:36:25 +00:00
Lance Release	e5f40a4b09	Bump version: 0.4.14 → 0.4.15 v0.4.15	2024-04-01 18:36:13 +00:00
Weston Pace	6779c1c192	chore: bump lance version to 0.10.6 (#1175)	2024-04-01 11:35:47 -07:00
Bert	e0bf6d9bd0	Update LanceDB Logo in README (#1167) ... <img width="1034" alt="image" src="https://github.com/lancedb/lancedb/assets/5846846/5b8aa53c-4d93-4c0e-bed4-80c238b319ba">	2024-03-29 10:10:43 -04:00
Weston Pace	67f041be91	docs: add a reference to @lancedb/lance in the docs (#1166) ... We aren't yet ready to switch over the examples since almost all JS examples rely on embeddings and we haven't yet ported those over. However, this makes it possible for those that are interested to start using `@lancedb/lancedb`	2024-03-29 04:55:03 -07:00
Will Jones	d388ef2f55	ci: fix name collision in npm artifacts for vectordb (#1164) ... Fixes #1163	2024-03-28 14:07:27 -05:00
Weston Pace	e52dc877e3	chore: add nodejs to bumpversion (#1161) ... The previous release failed to release nodejs because the nodejs version wasn't bumped. This should fix that.	2024-03-28 08:54:32 -07:00
Weston Pace	ca4fdf5499	chore: fix clippy (#1162)	2024-03-28 08:54:17 -07:00
Bert	0e9ad764b0	added new logo to vercel example gif (#1158)	2024-03-26 16:25:36 -04:00
Bert	cae0348c51	New logo on docs site (#1157)	2024-03-26 20:50:13 +05:30
Ayush Chaurasia	e9e0a37ca8	docs: Add all available HF/sentence transformers embedding models list (#1134) ... Solves - https://github.com/lancedb/lancedb/issues/968	2024-03-26 19:04:09 +05:30
Weston Pace	c37a28abbd	docs: add the async python API to the docs (#1156)	2024-03-26 07:54:16 -05:00
Lance Release	98c1e635b3	Updating package-lock.json	2024-03-25 20:38:37 +00:00
Lance Release	9992b927fd	Updating package-lock.json	2024-03-25 15:43:00 +00:00
Lance Release	80d501011c	Bump version: 0.4.13 → 0.4.14 v0.4.14	2024-03-25 15:42:49 +00:00
Weston Pace	6e3a9d08e0	feat: add publish step for nodejs (#1155) ... This will start publishing `@lancedb/lancedb` with the new nodejs package on our releases.	2024-03-25 11:23:30 -04:00
Pranav Maddi	268d8e057b	Adds a Ask LanceDB button to docs. (#1150) ... This links out to the new [asklancedb.com](https://asklancedb.com) page. Screenshots of the change:  	2024-03-23 01:09:44 +05:30
Bert	dfc518b8fb	Node SDK Client middleware for HTTP Requests (#1130) ... Adds client-side middleware to LanceDB Node SDK to instrument HTTP Requests Example - adding `x-request-id` request header: ```js class HttpMiddleware { constructor({ requestId }) { this.requestId = requestId } onRemoteRequest(req, next) { req.headers['x-request-id'] = this.requestId return next(req) } } const db = await lancedb.connect({ uri: 'db://remote-123', apiKey: 'sk_...', }) let tables = await db.withMiddleware(new HttpMiddleware({ requestId: '123' })).tableNames(); ``` --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>	2024-03-22 11:58:05 -04:00
QianZhu	98acf34ae8	remove warnings (#1147)	2024-03-21 14:49:01 -07:00
Lei Xu	25988d23cd	chore: validate table name (#1146) ... Closes #1129	2024-03-21 14:46:13 -07:00
Lance Release	c0dd98c798	[python] Bump version: 0.6.4 → 0.6.5 python-v0.6.5	2024-03-21 19:53:38 +00:00
Lei Xu	ee73a3bcb8	chore: bump lance to 0.10.5 (#1145)	2024-03-21 12:53:02 -07:00
QianZhu	c07989ac29	fix nodejs test (#1141) ... changed the error msg for query with wrong vector dim thus need this change to pass the nodejs tests.	2024-03-21 07:21:39 -07:00
QianZhu	8f7ef26f5f	better error msg for query vector with wrong dim (#1140)	2024-03-20 21:01:05 -07:00
Ishani Ghose	e14f079fe2	feat: add to_batches API #805 (#1048) ... SDK Python Description Exposes pyarrow batch api during query execution - relevant when there is no vector search query, dataset is large and the filtered result is larger than memory. --------- Co-authored-by: Ishani Ghose <isghose@amazon.com> Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>	2024-03-20 13:38:06 -07:00
Weston Pace	7d790bd9e7	feat: introduce ArrowNative wrapper struct for adding data that is already a RecordBatchReader (#1139) ... In 2de226220b I added a new `IntoArrow` trait for adding data into a table. Unfortunately, it seems my approach for implementing the trait for "things that are already record batch readers" was flawed. This PR corrects that flaw and, conveniently, removes the need to box readers at all (though it is ok if you do).	2024-03-20 13:28:17 -07:00
natcharacter	dbdd0a7b4b	Order by field support FTS (#1132) ... This PR adds support for passing through a set of ordering fields at index time (unsigned ints that tantivity can use as fast_fields) that at query time you can sort your results on. This is useful for cases where you want to get related hits, i.e by keyword, but order those hits by some other score, such as popularity. I.e search for songs descriptions that match on "sad AND jazz AND 1920" and then order those by number of times played. Example usage can be seen in the fts tests. --------- Co-authored-by: Nat Roth <natroth@Nats-MacBook-Pro.local> Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>	2024-03-20 01:27:37 -07:00
Chang She	befb79c5f9	feat(python): support writing huggingface dataset and dataset dict (#1110) ... HuggingFace Dataset is written as arrow batches. For DatasetDict, all splits are written with a "split" column appended. - [x] what if the dataset schema already has a `split` column - [x] add unit tests	2024-03-20 00:22:03 -07:00
Ayush Chaurasia	0a387a5429	feat(python): Support reranking for vector and fts (#1103) ... solves https://github.com/lancedb/lancedb/issues/1086 Usage Reranking with FTS: ``` retriever = db.create_table("fine-tuning", schema=Schema, mode="overwrite") pylist = [{"text": "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274."}, {"text": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan."}, {"text": "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas."}, {"text": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. "}, {"text": "Capital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."}, {"text": "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck."}, ] retriever.add(pylist) retriever.create_fts_index("text", replace=True) query = "What is the capital of the United States?" reranker = CohereReranker(return_score="all") print(retriever.search(query, query_type="fts").limit(10).to_pandas()) print(retriever.search(query, query_type="fts").rerank(reranker=reranker).limit(10).to_pandas()) ``` Result ``` text vector score 0 Capital punishment (the death penalty) has exi... [0.099975586, 0.047943115, -0.16723633, -0.183... 0.729602 1 Charlotte Amalie is the capital and largest ci... [-0.021255493, 0.03363037, -0.027450562, -0.17... 0.678046 2 The Commonwealth of the Northern Mariana Islan... [0.3684082, 0.30493164, 0.004600525, -0.049407... 0.671521 3 Carson City is the capital city of the America... [0.13989258, 0.14990234, 0.14172363, 0.0546569... 0.667898 4 Washington, D.C. (also known as simply Washing... [-0.0090408325, 0.42578125, 0.3798828, -0.3574... 0.653422 5 North Dakota is a state in the United States. ... [0.55859375, -0.2109375, 0.14526367, 0.1634521... 0.639346 text vector score _relevance_score 0 Washington, D.C. (also known as simply Washing... [-0.0090408325, 0.42578125, 0.3798828, -0.3574... 0.653422 0.979977 1 The Commonwealth of the Northern Mariana Islan... [0.3684082, 0.30493164, 0.004600525, -0.049407... 0.671521 0.299105 2 Capital punishment (the death penalty) has exi... [0.099975586, 0.047943115, -0.16723633, -0.183... 0.729602 0.284874 3 Carson City is the capital city of the America... [0.13989258, 0.14990234, 0.14172363, 0.0546569... 0.667898 0.089614 4 North Dakota is a state in the United States. ... [0.55859375, -0.2109375, 0.14526367, 0.1634521... 0.639346 0.063832 5 Charlotte Amalie is the capital and largest ci... [-0.021255493, 0.03363037, -0.027450562, -0.17... 0.678046 0.041462 ``` ## Vector Search usage: ``` query = "What is the capital of the United States?" reranker = CohereReranker(return_score="all") print(retriever.search(query).limit(10).to_pandas()) print(retriever.search(query).rerank(reranker=reranker, query=query).limit(10).to_pandas()) # <-- Note: passing extra string query here ``` Results ``` text vector _distance 0 Capital punishment (the death penalty) has exi... [0.099975586, 0.047943115, -0.16723633, -0.183... 39.728973 1 Washington, D.C. (also known as simply Washing... [-0.0090408325, 0.42578125, 0.3798828, -0.3574... 41.384884 2 Carson City is the capital city of the America... [0.13989258, 0.14990234, 0.14172363, 0.0546569... 55.220200 3 Charlotte Amalie is the capital and largest ci... [-0.021255493, 0.03363037, -0.027450562, -0.17... 58.345654 4 The Commonwealth of the Northern Mariana Islan... [0.3684082, 0.30493164, 0.004600525, -0.049407... 60.060867 5 North Dakota is a state in the United States. ... [0.55859375, -0.2109375, 0.14526367, 0.1634521... 64.260544 text vector _distance _relevance_score 0 Washington, D.C. (also known as simply Washing... [-0.0090408325, 0.42578125, 0.3798828, -0.3574... 41.384884 0.979977 1 The Commonwealth of the Northern Mariana Islan... [0.3684082, 0.30493164, 0.004600525, -0.049407... 60.060867 0.299105 2 Capital punishment (the death penalty) has exi... [0.099975586, 0.047943115, -0.16723633, -0.183... 39.728973 0.284874 3 Carson City is the capital city of the America... [0.13989258, 0.14990234, 0.14172363, 0.0546569... 55.220200 0.089614 4 North Dakota is a state in the United States. ... [0.55859375, -0.2109375, 0.14526367, 0.1634521... 64.260544 0.063832 5 Charlotte Amalie is the capital and largest ci... [-0.021255493, 0.03363037, -0.027450562, -0.17... 58.345654 0.041462 ```	2024-03-19 22:20:31 +05:30
Weston Pace	5a173e1d54	fix: fix compile error in example caused by merge conflict (#1135)	2024-03-19 08:55:15 -07:00
Weston Pace	51bdbcad98	feat: change DistanceType to be independent thing instead of resuing lance_linalg (#1133) ... This PR originated from a request to add `Serialize` / `Deserialize` to `lance_linalg::distance::DistanceType`. However, that is a strange request for `lance_linalg` which shouldn't really have to worry about `Serialize` / `Deserialize`. The problem is that `lancedb` is re-using `DistanceType` and things in `lancedb` do need to worry about `Serialize`/`Deserialize` (because `lancedb` needs to support remote client). On the bright side, separating the two types allows us to independently document distance type and allows `lance_linalg` to make changes to `DistanceType` in the future without having to worry about backwards compatibility concerns.	2024-03-19 07:27:51 -07:00
Weston Pace	0c7809c7a0	docs: add links to rust SDK docs, remove references to rust SDK being unstable / experimental (#1131)	2024-03-19 07:16:48 -07:00
Weston Pace	2de226220b	feat(rust): add trait for incoming data (#1128) ... This will make it easier for 3rd party integrations. They simply need to implement `IntoArrow` for their types in order for those types to be used in ingestion.	2024-03-19 07:15:49 -07:00
vincent d warmerdam	bd5b6f21e2	Unhide Pydantic guides in Docs (#1122) ... @wjones127 after fixing https://github.com/lancedb/lancedb/issues/1112 I noticed something else on the docs. There's an odd chunk of the docs missing [here](https://lancedb.github.io/lancedb/guides/tables/#from-a-polars-dataframe). I can see the heading, but after clicking it the contents don't show.  Apon inspection it was a markdown issue, one tab too many on a whole segment. This PR fixes it. It looks like this now and the sections appear again: 	2024-03-18 23:47:51 -07:00
Weston Pace	6331807b95	feat: refactor the query API and add query support to the python async API (#1113) ... In addition, there are also a number of changes in nodejs to the docstrings of existing methods because this PR adds a jsdoc linter.	2024-03-18 12:36:49 -07:00
Lance Release	83cb3f01a4	Updating package-lock.json	2024-03-18 18:05:55 +00:00
Lance Release	81f2cdf736	[python] Bump version: 0.6.3 → 0.6.4 python-v0.6.4	2024-03-16 18:59:14 +00:00
Lance Release	d404a3590c	Updating package-lock.json	2024-03-16 05:21:58 +00:00
Lance Release	e688484bd3	Bump version: 0.4.12 → 0.4.13 v0.4.13	2024-03-16 05:21:44 +00:00
Weston Pace	3bcd61c8de	feat: bump lance to 0.10.4 (#1123)	2024-03-15 22:21:04 -07:00
vincent d warmerdam	c76ec48603	Explain vonoroi seed initalisation (#1114) ... This PR fixes https://github.com/lancedb/lancedb/issues/1112. It turned out that K-means is currently used internally, so I figured adding that context to the docs would be nice.	2024-03-15 14:16:05 -07:00