rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-01-08 04:42:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: page_token / limit to native table_names function. Use async table_names function from sync table_names function (#1059)

Browse Source 
The synchronous table_names function in python lancedb relies on arrow's
filesystem which behaves slightly differently than object_store. As a
result, the function would not work properly in GCS.

However, the async table_names function uses object_store directly and
thus is accurate. In most cases we can fallback to using the async
table_names function and so this PR does so. The one case we cannot is
if the user is already in an async context (we can't start a new async
event loop). Soon, we can just redirect those users to use the async API
instead of the sync API and so that case will eventually go away. For
now, we fallback to the old behavior.
This commit is contained in:
Weston Pace
2024-03-05 08:38:18 -08:00
committed by GitHub
parent 47dbb988bf
commit 9148cd6d47
21 changed files with 250 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
nodejs/__test__/table.test.ts
View File

@@ -103,12 +103,12 @@ describe("Test creating index", () => {
// TODO: check index type.
// Search without specifying the column
const query_vector = data.toArray()[5].vec.toJSON();
const rst = await tbl.query().nearestTo(query_vector).limit(2).toArrow();
const queryVector = data.toArray()[5].vec.toJSON();
const rst = await tbl.query().nearestTo(queryVector).limit(2).toArrow();
expect(rst.numRows).toBe(2);
// Search with specifying the column
const rst2 = await tbl.search(query_vector, "vec").limit(2).toArrow();
const rst2 = await tbl.search(queryVector, "vec").limit(2).toArrow();
expect(rst2.numRows).toBe(2);
expect(rst.toString()).toEqual(rst2.toString());
});
@@ -169,6 +169,7 @@ describe("Test creating index", () => {
);
tbl
.createIndex("vec")
// eslint-disable-next-line @typescript-eslint/naming-convention
.ivf_pq({ num_partitions: 2, num_sub_vectors: 2 })
.build();
@@ -199,10 +200,10 @@ describe("Test creating index", () => {
const query64 = Array(64)
.fill(1)
.map(() => Math.random());
const rst64_1 = await tbl.query().nearestTo(query64).limit(2).toArrow();
const rst64_2 = await tbl.search(query64, "vec2").limit(2).toArrow();
expect(rst64_1.toString()).toEqual(rst64_2.toString());
expect(rst64_1.numRows).toBe(2);
const rst64Query = await tbl.query().nearestTo(query64).limit(2).toArrow();
const rst64Search = await tbl.search(query64, "vec2").limit(2).toArrow();
expect(rst64Query.toString()).toEqual(rst64Search.toString());
expect(rst64Query.numRows).toBe(2);
});
test("create scalar index", async () => {