fix: align table listing pagination across backends (#4412)

Fixes #4379

Remote `table_names` discarded the continuation token after the first
root-listing request, silently returning the server's default page of 10
names. Remote `list_tables` also inherited that server default, while
local listings had no SDK page limit. Python's synchronous and namespace
wrappers still imposed a default limit of 10 names. The namespace
adapter also discarded continuation tokens, so an unlimited REST
namespace listing could stop at the server's page cap.

Follow server-provided root listing tokens until the requested name
limit or the end of the listing, preserving `start_after`, cache
population, and guards against empty or repeated tokens. The namespace
adapter collects every server page, then sorts names and applies the
name cursor and total limit: namespace tokens are opaque, and page order
need not be name order. Empty tokens end the listing; token cycles
return an error. Python sync and namespace `table_names` wrappers now
default to `None`, matching the asynchronous and TypeScript APIs.
Namespace overrides accept the base connection's optional limit,
avoiding import-time signature errors on Python 3.10 and 3.11. Set the
`list_tables` default to 100 at the shared connection boundary,
including namespaces, and return empty results for a zero limit without
sending an invalid server request. Update the SDK documentation to
describe these defaults.

`list_tables()` is now a paginated call in both modes: callers with more
than 100 tables must follow the returned token or supply an explicit
larger limit. `table_names()` without a limit returns every name.
Namespace table-name calls enumerate the namespace before applying name
cursors and limits; use `list_tables()` for paginated access.

Validation:

- Regression cases reproduce the original root-listing truncation and
cover 130 tables, opaque tokens, short pages, explicit limits,
`start_after`, and zero limits.
- Relevant Rust remote database, connection, and local listing test
groups: 175 passed. Namespace adapter tests: 23 passed, including opaque
tokens, unordered and empty pages, name cursors, limits, zero requests,
and token cycles.
- Rebuilt the Python extension; namespace and local/mocked remote
listing tests using `uv.lock`: 46 passed on Python 3.10 and 46 passed on
Python 3.13.
- REST namespace regressions reproduced the truncation before the fix
and now return all 12 names from a server capped at 10 per response, for
sync/async connections at root and child paths. They also cover
unordered pages, name cursors, explicit limits, and zero requests. The
exact Python 3.10 import error also reproduced before the
optional-signature fix.
- TypeScript connection test file: 38 passed; `pnpm build`, `pnpm lint`,
and `pnpm run docs` passed.
- `cargo check --quiet --features remote --tests --examples` and `cargo
clippy --quiet --features remote --tests --examples` passed.
- Rust formatting, Python Ruff formatting/lint, and diff whitespace
checks passed.

<!-- lance-gatekeeper-fix:v1 agent=43f92f1f05ac9ca74229af590945cb88
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>

nodejs/__test__/connection.test.ts

@@ -1,7 +1,8 @@
 // SPDX-License-Identifier: Apache-2.0
 // SPDX-FileCopyrightText: Copyright The LanceDB Authors
 
-import { readdirSync } from "fs";
+import { mkdirSync, readdirSync } from "fs";
+import { join } from "path";
 import { Field, Float64, Schema } from "apache-arrow";
 import * as tmp from "tmp";
 import {
@@ -180,6 +181,32 @@ describe("given a connection", () => {
     expect(seen).toEqual(created);
   });
 
+  it("should list all names and use a default page size of 100", async () => {
+    const names = Array.from(
+      { length: 130 },
+      (_, i) => `t${i.toString().padStart(3, "0")}`,
+    );
+    for (const name of names) {
+      mkdirSync(join(tmpDir.name, `${name}.lance`));
+    }
+    await expect(db.tableNames()).resolves.toEqual(names);
+    await expect(db.tableNames({ limit: 0 })).resolves.toEqual([]);
+    const first = await db.listTables();
+    expect(first.tables).toEqual(names.slice(0, 100));
+    expect(first.pageToken).toBeDefined();
+    const second = await db.listTables({ pageToken: first.pageToken });
+    expect(second.tables).toEqual(names.slice(100));
+    expect(second.pageToken).toBeUndefined();
+    await expect(db.listTables({ limit: 200 })).resolves.toEqual({
+      tables: names,
+      pageToken: undefined,
+    });
+    await expect(db.listTables({ limit: 0 })).resolves.toEqual({
+      tables: [],
+      pageToken: undefined,
+    });
+  });
+
   it("should list tables in a namespace", async () => {
     const db = await connect(tmpDir.name, {
       // biome-ignore lint/style/useNamingConvention: opaque backend property key, must match Rust

nodejs/lancedb/connection.ts

@@ -150,7 +150,7 @@ export interface TableNamesOptions {
    * the last table name from the previous page.
    */
   startAfter?: string;
-  /** An optional limit to the number of results to return. */
+  /** The maximum number of names to return. Omitted returns all names; zero returns none. */
   limit?: number;
 }
 
@@ -164,6 +164,7 @@ export interface ListTablesOptions {
   pageToken?: string;
   /**
    * An upper bound on how many tables to return.
+   * Defaults to 100. Zero returns an empty page without a continuation token.
    *
    * A page may hold fewer than this and still not be the last one, so keep
    * going while the response carries a page token rather than while pages are