feat(sql): use server-issued catalog listing tokens (#5026)

Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM
'<page-token>'` and server-issued Arrow continuation metadata.
Namespace-specific APIs send an explicit `IN` schema clause, including
`IN "public"` for root, so raw SQL can default to all namespaces.

```sql
SHOW VIEWS LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>';
SHOW VIEWS IN other_db.team$ml LIMIT 100;
```

SDK clients forward opaque tokens unchanged, preserve empty
continuations, and decode identifier columns by name. Removes legacy
cursor reconstruction and updates the SQL reference and Python Flight
fixture. Public docs cover optional direct schema paths, all-namespace
defaults, and the shared pagination syntax across every catalog listing.
This commit is contained in:
Jack Ye
2026-11-15 08:55:42 +08:00
committed by GitHub
parent e2f836da99
commit e8a8dd1b2a
22 changed files with 663 additions and 695 deletions
+13 -7
View File
@@ -57,15 +57,21 @@ properties and requires system `USAGE`.
## SHOW DATABASES {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW DATABASES
```sql
SHOW DATABASES LIMIT 100;
SHOW DATABASES LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
Returns a sorted `name` column containing the databases the caller has
`USAGE` on, including `system` when permitted. `LIKE` and `WHERE` filters
are unsupported.
Returns one non-null `name` column, sorted by database name and filtered to
databases the caller has `USAGE` on, including `system` when permitted.
Databases are above namespaces, so this command does not accept `IN` or
return a `namespace_path` column.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## ATTACH DATABASE {#attach}
+25 -12
View File
@@ -135,20 +135,33 @@ from it.
## SHOW FUNCTIONS {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW FUNCTIONS
SHOW FUNCTIONS HISTORY IN "analytics" WITH (name = 'embed', limit = 100)
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '1', limit = 100)
```sql
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100;
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
`SHOW FUNCTIONS` returns the highest published numeric revision for each name.
Add `HISTORY` to list every published revision, as Python's `list_functions()`
does. Rows are ordered by name and numeric revision, so `2` comes before `10`.
`after_name` and `after_version` must be supplied together for keyset pagination.
`limit` bounds
the response; `include_definition = false` omits the optional definition JSON.
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
`SHOW FUNCTIONS` lists each function once. Add `HISTORY` to list every
published revision with its metadata and definition, as `list_functions()`
does. History uses the same pagination suffix; numeric revision order is
encoded in the token, so revision `2` precedes `10`.
```sql
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100;
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## DROP FUNCTION {#drop}
+17 -8
View File
@@ -70,16 +70,25 @@ automatic execution needs the same server-side switch enabled. Omitting
## SHOW MATERIALIZED VIEWS {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW MATERIALIZED VIEWS
```sql
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100;
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
<Note>
This asks the server for its views, as the SDK's `list_materialized_views` does
on a `db://` connection.
</Note>
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
The listing contains identifiers; use `DESCRIBE MATERIALIZED VIEW` for its defining query and refresh metadata.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## Refreshing one {#refresh}
+21 -19
View File
@@ -17,7 +17,7 @@ backtick-quoted identifier with `$` between components, such as
one path component, relative to an explicit parent schema or the session's
current schema -- it cannot contain `$` or be qualified with a database.
`SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
For single-object commands, `SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
schema identifier. Names are lowercased unless quoted.
## CREATE SCHEMA and CREATE NAMESPACE {#create}
@@ -126,27 +126,30 @@ Requires `USAGE` on the database and the namespace.
## SHOW SCHEMAS and SHOW NAMESPACES {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW SCHEMAS
SHOW NAMESPACES
SHOW NAMESPACES IN analytics.team
```sql
SHOW SCHEMAS LIMIT 100;
SHOW SCHEMAS LIMIT 100 FROM '<page-token>';
SHOW NAMESPACES IN analytics.team LIMIT 100;
SHOW NAMESPACES IN analytics.team LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
`SHOW SCHEMAS` returns complete paths in the current database, recursively,
including `public` when the caller can use it. It does not accept `IN`.
Both commands return two non-null string columns: `namespace_path`, the
parent path as a compact JSON array, and `name`, the unqualified child name.
Results are ordered by parent path components, then name, and omit namespaces
the caller cannot use.
`SHOW NAMESPACES` lists only immediate child names under the session's
current schema. `IN` selects a different parent schema, optionally in
another database -- use `IN public` for the current database's root, or
`IN analytics.public` for another database's root. A missing parent is an
error.
`SHOW SCHEMAS` and `SHOW NAMESPACES` are equivalent listings. Without `IN`,
both list recursively in the current database, include the implicit `public`
root, and ignore the session namespace. With `IN`, both list immediate children
of the selected parent. A schema path can appear directly: `IN team$features`;
a database qualifier is optional. `IN public` selects the current database root,
and `IN analytics.public` selects another database's root.
Both forms return a sorted `name` column and omit namespaces the caller
cannot use. Large listings that exceed 10,000 namespaces, or recursive
listings deeper than 32 levels, return an error suggesting a narrower
listing.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## DROP SCHEMA and DROP NAMESPACE {#drop}
@@ -187,7 +190,6 @@ with `IF EXISTS`. Inspect a cascade drop's child jobs with
| `CREATE SCHEMA child IN parent` | `CREATE NAMESPACE child IN parent` or `` CREATE SCHEMA `parent$child` `` |
| `` DESCRIBE NAMESPACE `parent$child` `` | `` DESCRIBE SCHEMA `parent$child` `` or `DESCRIBE NAMESPACE child IN parent` |
| `DROP NAMESPACE analytics.child` | `DROP NAMESPACE child IN analytics.public` |
| `SHOW SCHEMAS IN parent` | `SHOW NAMESPACES IN parent` |
## What to read next {#next}
+17 -13
View File
@@ -98,21 +98,25 @@ picked up by every Function still bound to that name.
## SHOW SECRETS {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW SECRETS
SHOW SECRETS IN platform
SHOW SECRETS IN analytics.`ml$vision`
SHOW SECRETS IN public
SHOW SECRETS IN platform WHERE name > 'model-api-key' LIMIT 50
```sql
SHOW SECRETS IN analytics.`team$ml` LIMIT 100;
SHOW SECRETS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
Lists the Secret names in one namespace as a sorted `name` column. Without
`IN`, it lists the session's namespace; `IN public` lists the root. `WHERE
name > '<cursor>'` pages past the last name already seen, and `LIMIT`
bounds the page size; `WHERE` must compare `name` with `>`. Requires
`USAGE` on the database and the namespace.
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
Secret listings contain no values or Secret metadata.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## DESCRIBE SECRET {#describe}
+17 -8
View File
@@ -194,16 +194,25 @@ only the reference naming it.
## SHOW STORAGE INTEGRATIONS {#show}
<CodeGroup>
```sql SQL icon="database"
SHOW STORAGE INTEGRATIONS
SHOW STORAGE INTEGRATIONS IN platform
LIST STORAGE INTEGRATIONS
```sql
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100;
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
</CodeGroup>
`LIST` is an alias for `SHOW` here. Returns one `name` column; `IN`
narrows the listing to one namespace.
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
`LIST STORAGE INTEGRATIONS` is an exact alias, including pagination.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## DROP STORAGE INTEGRATION {#drop}
+20
View File
@@ -268,3 +268,23 @@ deployment stores its catalog. See
Renaming, retyping and dropping columns from the SDK.
</Card>
</CardGroup>
## SHOW TABLES {#show}
```sql
SHOW TABLES IN analytics.`team$ml` LIMIT 100;
SHOW TABLES IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
+13 -20
View File
@@ -81,29 +81,22 @@ was none to begin with.
## Listing {#listing}
```sql
SHOW VIEWS
SHOW VIEWS IN "analytics" WHERE name > 'previous_view' LIMIT 100
SHOW VIEWS IN analytics.`team$ml` LIMIT 100;
SHOW VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
`SHOW VIEWS` returns ascending `name` values. The `name` cursor is scoped to
the selected namespace.
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
Object commands retain their small results in shared storage even when general
query-result copying is disabled, so asynchronous polling can use any query node.
Views and tables share a namespace, and both are things you can query. The
catalog distinguishes them:
| | |
| --- | --- |
| view names | the views in the database |
| table names | the tables |
| relation names | everything queryable (both) |
Use the relation listing when the question is "what can I select from," and the
separate listings when the distinction matters: before dropping, for
instance, since dropping a view removes only the saved query while dropping a
table removes data.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## Choosing between a view and a materialized view {#choosing}
+68
View File
@@ -0,0 +1,68 @@
---
title: "Catalog listing pagination"
description: "Use one continuation-token syntax for SQL catalog listings."
---
Catalog `SHOW` statements share one pagination suffix:
```sql
SHOW <objects> [IN <scope>] [LIMIT <count> [FROM '<page-token>']];
```
This applies to tables, views, materialized views, functions and function
history, secrets, storage integrations, namespaces/schemas, and databases.
```sql
SHOW TABLES LIMIT 100;
SHOW TABLES IN team$ml LIMIT 100;
SHOW TABLES IN team$ml LIMIT 100 FROM '<page-token>';
SHOW TABLES IN analytics.team$ml LIMIT 100;
SHOW DATABASES LIMIT 100 FROM '<page-token>';
```
## Namespace scope
`IN` is optional. Without it, listings include root and every authorized
namespace in the current database, regardless of the session's default namespace.
With `IN`, objects are listed only in that exact namespace. The scope accepts a
schema directly: `IN team$ml` means the absolute namespace path `["team", "ml"]`
in the current database. Use `IN public` for root, or `IN other_db.team$ml` to
select a different database. Quote schema identifiers containing punctuation or
mixed case, for example `IN "Team$ML"`.
`SHOW SCHEMAS` and `SHOW NAMESPACES` use the same rules: without `IN` they
include all namespace paths and the public root; with `IN` they list the
selected parent's immediate children. `SHOW DATABASES` remains global and
does not accept `IN`.
SDK APIs that take a namespace send an explicit `IN` clause and continue to
list that one namespace.
## Continuation metadata
The Arrow result schema carries `lancedb.next_page_token` when more results
remain. Pass that opaque value unchanged in `FROM` on the next request.
The token identifies the object type, database, namespace scope, and complete
position, including namespace path and any numeric revision. It cannot be
reused with another object type or scope.
The token is metadata, not an additional result column or a synthetic row.
Namespace-scoped object listings contain `namespace_path` and `name`;
databases contain only `name`. Function history retains revision metadata.
SDK listing APIs expose this server-issued value as the page's `page_token`.
Continue until no token is returned. A short or empty page may still have a
token. Pages observe the live catalog; continuation does not create a snapshot.
## Page size and ordering
The default page limit is 1,000, with a maximum of 10,000. `LIMIT 0` returns
no rows. Object catalogs order identifiers by namespace path components and
unqualified name; database names are ordered by name. Table page boundaries
follow native catalog order, which may differ from logical name order for
directory-backed tables; table names are sorted within each returned page. Function history uses
numeric revision order within each name. The token preserves that complete
position, including when names repeat in different namespaces.
The old name-comparison `WHERE` and Function `WITH` pagination forms are no
longer supported. Use the same `LIMIT ... FROM ...` suffix for every listing.