mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
feat(sql): use server-issued catalog listing tokens (#5026)
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM '<page-token>'` and server-issued Arrow continuation metadata. Namespace-specific APIs send an explicit `IN` schema clause, including `IN "public"` for root, so raw SQL can default to all namespaces. ```sql SHOW VIEWS LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>'; SHOW VIEWS IN other_db.team$ml LIMIT 100; ``` SDK clients forward opaque tokens unchanged, preserve empty continuations, and decode identifier columns by name. Removes legacy cursor reconstruction and updates the SQL reference and Python Flight fixture. Public docs cover optional direct schema paths, all-namespace defaults, and the shared pagination syntax across every catalog listing.
This commit is contained in:
@@ -57,15 +57,21 @@ properties and requires system `USAGE`.
|
||||
|
||||
## SHOW DATABASES {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW DATABASES
|
||||
```sql
|
||||
SHOW DATABASES LIMIT 100;
|
||||
SHOW DATABASES LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Returns a sorted `name` column containing the databases the caller has
|
||||
`USAGE` on, including `system` when permitted. `LIKE` and `WHERE` filters
|
||||
are unsupported.
|
||||
Returns one non-null `name` column, sorted by database name and filtered to
|
||||
databases the caller has `USAGE` on, including `system` when permitted.
|
||||
Databases are above namespaces, so this command does not accept `IN` or
|
||||
return a `namespace_path` column.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## ATTACH DATABASE {#attach}
|
||||
|
||||
|
||||
@@ -135,20 +135,33 @@ from it.
|
||||
|
||||
## SHOW FUNCTIONS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW FUNCTIONS
|
||||
SHOW FUNCTIONS HISTORY IN "analytics" WITH (name = 'embed', limit = 100)
|
||||
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '1', limit = 100)
|
||||
```sql
|
||||
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`SHOW FUNCTIONS` returns the highest published numeric revision for each name.
|
||||
Add `HISTORY` to list every published revision, as Python's `list_functions()`
|
||||
does. Rows are ordered by name and numeric revision, so `2` comes before `10`.
|
||||
`after_name` and `after_version` must be supplied together for keyset pagination.
|
||||
`limit` bounds
|
||||
the response; `include_definition = false` omits the optional definition JSON.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
`SHOW FUNCTIONS` lists each function once. Add `HISTORY` to list every
|
||||
published revision with its metadata and definition, as `list_functions()`
|
||||
does. History uses the same pagination suffix; numeric revision order is
|
||||
encoded in the token, so revision `2` precedes `10`.
|
||||
|
||||
```sql
|
||||
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP FUNCTION {#drop}
|
||||
|
||||
|
||||
@@ -70,16 +70,25 @@ automatic execution needs the same server-side switch enabled. Omitting
|
||||
|
||||
## SHOW MATERIALIZED VIEWS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW MATERIALIZED VIEWS
|
||||
```sql
|
||||
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
This asks the server for its views, as the SDK's `list_materialized_views` does
|
||||
on a `db://` connection.
|
||||
</Note>
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
The listing contains identifiers; use `DESCRIBE MATERIALIZED VIEW` for its defining query and refresh metadata.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## Refreshing one {#refresh}
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ backtick-quoted identifier with `$` between components, such as
|
||||
one path component, relative to an explicit parent schema or the session's
|
||||
current schema -- it cannot contain `$` or be qualified with a database.
|
||||
|
||||
`SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
|
||||
For single-object commands, `SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
|
||||
schema identifier. Names are lowercased unless quoted.
|
||||
|
||||
## CREATE SCHEMA and CREATE NAMESPACE {#create}
|
||||
@@ -126,27 +126,30 @@ Requires `USAGE` on the database and the namespace.
|
||||
|
||||
## SHOW SCHEMAS and SHOW NAMESPACES {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW SCHEMAS
|
||||
SHOW NAMESPACES
|
||||
SHOW NAMESPACES IN analytics.team
|
||||
```sql
|
||||
SHOW SCHEMAS LIMIT 100;
|
||||
SHOW SCHEMAS LIMIT 100 FROM '<page-token>';
|
||||
SHOW NAMESPACES IN analytics.team LIMIT 100;
|
||||
SHOW NAMESPACES IN analytics.team LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`SHOW SCHEMAS` returns complete paths in the current database, recursively,
|
||||
including `public` when the caller can use it. It does not accept `IN`.
|
||||
Both commands return two non-null string columns: `namespace_path`, the
|
||||
parent path as a compact JSON array, and `name`, the unqualified child name.
|
||||
Results are ordered by parent path components, then name, and omit namespaces
|
||||
the caller cannot use.
|
||||
|
||||
`SHOW NAMESPACES` lists only immediate child names under the session's
|
||||
current schema. `IN` selects a different parent schema, optionally in
|
||||
another database -- use `IN public` for the current database's root, or
|
||||
`IN analytics.public` for another database's root. A missing parent is an
|
||||
error.
|
||||
`SHOW SCHEMAS` and `SHOW NAMESPACES` are equivalent listings. Without `IN`,
|
||||
both list recursively in the current database, include the implicit `public`
|
||||
root, and ignore the session namespace. With `IN`, both list immediate children
|
||||
of the selected parent. A schema path can appear directly: `IN team$features`;
|
||||
a database qualifier is optional. `IN public` selects the current database root,
|
||||
and `IN analytics.public` selects another database's root.
|
||||
|
||||
Both forms return a sorted `name` column and omit namespaces the caller
|
||||
cannot use. Large listings that exceed 10,000 namespaces, or recursive
|
||||
listings deeper than 32 levels, return an error suggesting a narrower
|
||||
listing.
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP SCHEMA and DROP NAMESPACE {#drop}
|
||||
|
||||
@@ -187,7 +190,6 @@ with `IF EXISTS`. Inspect a cascade drop's child jobs with
|
||||
| `CREATE SCHEMA child IN parent` | `CREATE NAMESPACE child IN parent` or `` CREATE SCHEMA `parent$child` `` |
|
||||
| `` DESCRIBE NAMESPACE `parent$child` `` | `` DESCRIBE SCHEMA `parent$child` `` or `DESCRIBE NAMESPACE child IN parent` |
|
||||
| `DROP NAMESPACE analytics.child` | `DROP NAMESPACE child IN analytics.public` |
|
||||
| `SHOW SCHEMAS IN parent` | `SHOW NAMESPACES IN parent` |
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
|
||||
+17
-13
@@ -98,21 +98,25 @@ picked up by every Function still bound to that name.
|
||||
|
||||
## SHOW SECRETS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW SECRETS
|
||||
SHOW SECRETS IN platform
|
||||
SHOW SECRETS IN analytics.`ml$vision`
|
||||
SHOW SECRETS IN public
|
||||
SHOW SECRETS IN platform WHERE name > 'model-api-key' LIMIT 50
|
||||
```sql
|
||||
SHOW SECRETS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW SECRETS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Lists the Secret names in one namespace as a sorted `name` column. Without
|
||||
`IN`, it lists the session's namespace; `IN public` lists the root. `WHERE
|
||||
name > '<cursor>'` pages past the last name already seen, and `LIMIT`
|
||||
bounds the page size; `WHERE` must compare `name` with `>`. Requires
|
||||
`USAGE` on the database and the namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Secret listings contain no values or Secret metadata.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DESCRIBE SECRET {#describe}
|
||||
|
||||
|
||||
@@ -194,16 +194,25 @@ only the reference naming it.
|
||||
|
||||
## SHOW STORAGE INTEGRATIONS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW STORAGE INTEGRATIONS
|
||||
SHOW STORAGE INTEGRATIONS IN platform
|
||||
LIST STORAGE INTEGRATIONS
|
||||
```sql
|
||||
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`LIST` is an alias for `SHOW` here. Returns one `name` column; `IN`
|
||||
narrows the listing to one namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
`LIST STORAGE INTEGRATIONS` is an exact alias, including pagination.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP STORAGE INTEGRATION {#drop}
|
||||
|
||||
|
||||
@@ -268,3 +268,23 @@ deployment stores its catalog. See
|
||||
Renaming, retyping and dropping columns from the SDK.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## SHOW TABLES {#show}
|
||||
|
||||
```sql
|
||||
SHOW TABLES IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW TABLES IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
+13
-20
@@ -81,29 +81,22 @@ was none to begin with.
|
||||
## Listing {#listing}
|
||||
|
||||
```sql
|
||||
SHOW VIEWS
|
||||
SHOW VIEWS IN "analytics" WHERE name > 'previous_view' LIMIT 100
|
||||
SHOW VIEWS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
`SHOW VIEWS` returns ascending `name` values. The `name` cursor is scoped to
|
||||
the selected namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Object commands retain their small results in shared storage even when general
|
||||
query-result copying is disabled, so asynchronous polling can use any query node.
|
||||
|
||||
Views and tables share a namespace, and both are things you can query. The
|
||||
catalog distinguishes them:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| view names | the views in the database |
|
||||
| table names | the tables |
|
||||
| relation names | everything queryable (both) |
|
||||
|
||||
Use the relation listing when the question is "what can I select from," and the
|
||||
separate listings when the distinction matters: before dropping, for
|
||||
instance, since dropping a view removes only the saved query while dropping a
|
||||
table removes data.
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## Choosing between a view and a materialized view {#choosing}
|
||||
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: "Catalog listing pagination"
|
||||
description: "Use one continuation-token syntax for SQL catalog listings."
|
||||
---
|
||||
|
||||
Catalog `SHOW` statements share one pagination suffix:
|
||||
|
||||
```sql
|
||||
SHOW <objects> [IN <scope>] [LIMIT <count> [FROM '<page-token>']];
|
||||
```
|
||||
|
||||
This applies to tables, views, materialized views, functions and function
|
||||
history, secrets, storage integrations, namespaces/schemas, and databases.
|
||||
|
||||
```sql
|
||||
SHOW TABLES LIMIT 100;
|
||||
SHOW TABLES IN team$ml LIMIT 100;
|
||||
SHOW TABLES IN team$ml LIMIT 100 FROM '<page-token>';
|
||||
SHOW TABLES IN analytics.team$ml LIMIT 100;
|
||||
SHOW DATABASES LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
## Namespace scope
|
||||
|
||||
`IN` is optional. Without it, listings include root and every authorized
|
||||
namespace in the current database, regardless of the session's default namespace.
|
||||
With `IN`, objects are listed only in that exact namespace. The scope accepts a
|
||||
schema directly: `IN team$ml` means the absolute namespace path `["team", "ml"]`
|
||||
in the current database. Use `IN public` for root, or `IN other_db.team$ml` to
|
||||
select a different database. Quote schema identifiers containing punctuation or
|
||||
mixed case, for example `IN "Team$ML"`.
|
||||
|
||||
`SHOW SCHEMAS` and `SHOW NAMESPACES` use the same rules: without `IN` they
|
||||
include all namespace paths and the public root; with `IN` they list the
|
||||
selected parent's immediate children. `SHOW DATABASES` remains global and
|
||||
does not accept `IN`.
|
||||
|
||||
SDK APIs that take a namespace send an explicit `IN` clause and continue to
|
||||
list that one namespace.
|
||||
|
||||
## Continuation metadata
|
||||
|
||||
The Arrow result schema carries `lancedb.next_page_token` when more results
|
||||
remain. Pass that opaque value unchanged in `FROM` on the next request.
|
||||
The token identifies the object type, database, namespace scope, and complete
|
||||
position, including namespace path and any numeric revision. It cannot be
|
||||
reused with another object type or scope.
|
||||
|
||||
The token is metadata, not an additional result column or a synthetic row.
|
||||
Namespace-scoped object listings contain `namespace_path` and `name`;
|
||||
databases contain only `name`. Function history retains revision metadata.
|
||||
SDK listing APIs expose this server-issued value as the page's `page_token`.
|
||||
|
||||
Continue until no token is returned. A short or empty page may still have a
|
||||
token. Pages observe the live catalog; continuation does not create a snapshot.
|
||||
|
||||
## Page size and ordering
|
||||
|
||||
The default page limit is 1,000, with a maximum of 10,000. `LIMIT 0` returns
|
||||
no rows. Object catalogs order identifiers by namespace path components and
|
||||
unqualified name; database names are ordered by name. Table page boundaries
|
||||
follow native catalog order, which may differ from logical name order for
|
||||
directory-backed tables; table names are sorted within each returned page. Function history uses
|
||||
numeric revision order within each name. The token preserves that complete
|
||||
position, including when names repeat in different namespaces.
|
||||
|
||||
The old name-comparison `WHERE` and Function `WITH` pagination forms are no
|
||||
longer supported. Use the same `LIMIT ... FROM ...` suffix for every listing.
|
||||
Reference in New Issue
Block a user