rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-26 05:26:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat!: return lazy iterators for table and namespace listings (#4599)

Browse Source 
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.

In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
    print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
This commit is contained in:
Colin Patrick McCabe
2026-10-22 15:41:30 -07:00
committed by GitHub
parent 88203d2515
commit 631ea41a63
27 changed files with 814 additions and 713 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/js/classes/Connection.md
+17 -63
View File
@@ -653,94 +653,48 @@ See [Listing](Listing.md) for pagination state and error behavior.
### listNamespaces()
```ts
abstract listNamespaces(namespacePath?, options?): Promise<ListNamespacesResponse>
abstract listNamespaces(namespacePath?, options?): Listing<string>
```
List the immediate child namespaces under the given parent.
Results may be paginated. To retrieve subsequent pages, pass the
`pageToken` returned by a previous call.
Lazily list immediate child namespaces, 100 per page by default.
#### Parameters
* **namespacePath?**: `string`[]
The parent namespace path. Defaults
to the root namespace if omitted.
* **options?**: `Partial`&lt;[`ListNamespacesOptions`](../interfaces/ListNamespacesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
`Promise`&lt;[`ListNamespacesResponse`](../interfaces/ListNamespacesResponse.md)&gt;
Child namespace names and
an optional token for fetching the next page.
[`Listing`](Listing.md)&lt;`string`&gt;
***
### listTables()
#### listTables(options)
```ts
abstract listTables(options?): Promise<ListTablesResponse>
abstract listTables(namespacePath?, options?): Listing<string>
```
List a page of the tables in this database.
Lazily list tables in a namespace, requesting 100 per page by default.
To retrieve the tables after the page, pass the `pageToken` the response
carries back in. A page can be shorter than `limit` without being the last
one, so walk until a response carries no page token:
```ts
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
```
##### Parameters
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
##### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
A page of table names and an
optional token for the tables after it.
#### listTables(namespacePath, options)
```ts
abstract listTables(namespacePath?, options?): Promise<ListTablesResponse>
```
List a page of the tables in this database.
##### Parameters
#### Parameters
* **namespacePath?**: `string`[]
The namespace path to list tables from
(defaults to root namespace)
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
##### Returns
#### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
[`Listing`](Listing.md)&lt;`string`&gt;
A page of table names and an
optional token for the tables after it.
#### Example
```ts
for await (const name of conn.listTables([], { pageLimit: 20 })) {
console.log(name);
}
```
***
docs/src/js/globals.md
-4
View File
@@ -108,10 +108,6 @@
- [JobEventsOptions](interfaces/JobEventsOptions.md)
- [JobFailureInfo](interfaces/JobFailureInfo.md)
- [JobInfo](interfaces/JobInfo.md)
- [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
- [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
- [ListTablesOptions](interfaces/ListTablesOptions.md)
- [ListTablesResponse](interfaces/ListTablesResponse.md)
- [ListingOptions](interfaces/ListingOptions.md)
- [MaterializedViewDefinition](interfaces/MaterializedViewDefinition.md)
- [MemtableStats](interfaces/MemtableStats.md)
docs/src/js/interfaces/ListNamespacesOptions.md
-27
View File
@@ -1,27 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListNamespacesOptions
# Interface: ListNamespacesOptions
## Properties
### limit?
```ts
optional limit: number;
```
An optional limit to the number of results to return.
***
### pageToken?
```ts
optional pageToken: string;
```
Token from a previous response for pagination.
docs/src/js/interfaces/ListNamespacesResponse.md
-23
View File
@@ -1,23 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListNamespacesResponse
# Interface: ListNamespacesResponse
## Properties
### namespaces
```ts
namespaces: string[];
```
***
### pageToken?
```ts
optional pageToken: string;
```
docs/src/js/interfaces/ListTablesOptions.md
-35
View File
@@ -1,35 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListTablesOptions
# Interface: ListTablesOptions
## Properties
### limit?
```ts
optional limit: number;
```
An upper bound on how many tables to return.
Defaults to 100. Zero returns an empty page without a continuation token.
A page may hold fewer than this and still not be the last one, so keep
going while the response carries a page token rather than while pages are
full.
***
### pageToken?
```ts
optional pageToken: string;
```
Token from a previous response, to resume listing where it left off.
The token is opaque: it carries whatever the database needs to resume, and
callers should not construct or interpret one.
docs/src/js/interfaces/ListTablesResponse.md
-23
View File
@@ -1,23 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListTablesResponse
# Interface: ListTablesResponse
## Properties
### pageToken?
```ts
optional pageToken: string;
```
***
### tables
```ts
tables: string[];
```
docs/src/js/interfaces/TableNamesOptions.md
+1 -1
View File
@@ -8,7 +8,7 @@
## Deprecated
Use [ListTablesOptions](ListTablesOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
Use [ListingOptions](ListingOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
instead.
## Properties
docs/src/python/python.md
+18 -5
View File
@@ -26,14 +26,17 @@ is also an [asynchronous API client](#connections-asynchronous).
::: lancedb.db.DBConnection
Resource listings (`list_secrets`, `list_views`, `list_jobs`, `list_functions`, and
`list_materialized_views`) return lazy iterators. Use `list(db.list_views())` to
collect synchronous results, or `[name async for name in db.list_views()]` with
an asynchronous connection. `page_limit` controls each request, and `page_token`
resumes a listing. Drain the iterator's cached results before saving its token.
Resource listings return iterators. Use `list(db.list_tables())` to collect all
table names. `db.list_tables().tables` is a list of the first page; `limit`
controls its page size and `page_token` resumes at a saved token. The same
patterns apply to `list_namespaces()`. Other resource listings use `page_limit`.
::: lancedb.listing.Listing
::: lancedb.listing.TableListing
::: lancedb.listing.NamespaceListing
::: lancedb.Session
## Catalogs (Synchronous)
@@ -631,6 +634,16 @@ can be used to create, list, or open tables.
::: lancedb.listing.AsyncListing
`async for name in db.list_tables()` loads pages as needed. To inspect the
first page, use `page = await db.list_tables()` and then `page.tables`. Reading
the page list does not consume the iterator. `page.page_token` is the token for
the next request; drain cached results before using it to resume in a new
listing. The same behavior applies to namespaces.
::: lancedb.listing.AsyncTableListing
::: lancedb.listing.AsyncNamespaceListing
## Namespaces (Asynchronous)
::: lancedb.connect_namespace_async