Will Jones b015dca20f fix(listing): paginate table listing instead of enumerating the database ...

`ListingDatabase::table_names` and `list_tables` listed every table directory
under the database prefix before applying `limit` and `page_token`. The cost of
a request was set by the size of the database rather than the size of the page,
so listing one table out of ten thousand took ten S3 round trips instead of one.

List through `ObjectStore::read_dir_stream`, which pushes the resume position and
the page size into the store's list request. Stores with no paginated list API
fall back to a full listing, which is what every store did before.

Two behaviour changes come with it:

- Names are reported in the order the store lists directories, which differs
  from sorting by name only between a name and one that extends it: `users-archive`
  now precedes `users`, because `-` sorts below the `.` of `users.lance`.
  Pagination cannot report an order other than the one it resumes in.
- `list_tables` returned the first name of the *next* page as its `page_token`,
  and `page_token` resumes *after* the name it is given, so paging dropped one
  table per page boundary. The token is now the last name of the page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-03 16:24:44 -07:00

..

lancedb

fix(listing): paginate table listing instead of enumerating the database

2026-08-03 16:24:44 -07:00

license_header.txt

ci: check license headers (#2076)

2025-01-29 08:27:07 -08:00