mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-21 18:13:57 +00:00
feat(sql): use server-issued catalog listing tokens (#5026)
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM '<page-token>'` and server-issued Arrow continuation metadata. Namespace-specific APIs send an explicit `IN` schema clause, including `IN "public"` for root, so raw SQL can default to all namespaces. ```sql SHOW VIEWS LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>'; SHOW VIEWS IN other_db.team$ml LIMIT 100; ``` SDK clients forward opaque tokens unchanged, preserve empty continuations, and decode identifier columns by name. Removes legacy cursor reconstruction and updates the SQL reference and Python Flight fixture. Public docs cover optional direct schema paths, all-namespace defaults, and the shared pagination syntax across every catalog listing.
This commit is contained in:
@@ -57,15 +57,21 @@ properties and requires system `USAGE`.
|
||||
|
||||
## SHOW DATABASES {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW DATABASES
|
||||
```sql
|
||||
SHOW DATABASES LIMIT 100;
|
||||
SHOW DATABASES LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Returns a sorted `name` column containing the databases the caller has
|
||||
`USAGE` on, including `system` when permitted. `LIKE` and `WHERE` filters
|
||||
are unsupported.
|
||||
Returns one non-null `name` column, sorted by database name and filtered to
|
||||
databases the caller has `USAGE` on, including `system` when permitted.
|
||||
Databases are above namespaces, so this command does not accept `IN` or
|
||||
return a `namespace_path` column.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## ATTACH DATABASE {#attach}
|
||||
|
||||
|
||||
@@ -135,20 +135,33 @@ from it.
|
||||
|
||||
## SHOW FUNCTIONS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW FUNCTIONS
|
||||
SHOW FUNCTIONS HISTORY IN "analytics" WITH (name = 'embed', limit = 100)
|
||||
SHOW FUNCTIONS HISTORY WITH (after_name = 'embed', after_version = '1', limit = 100)
|
||||
```sql
|
||||
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW FUNCTIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`SHOW FUNCTIONS` returns the highest published numeric revision for each name.
|
||||
Add `HISTORY` to list every published revision, as Python's `list_functions()`
|
||||
does. Rows are ordered by name and numeric revision, so `2` comes before `10`.
|
||||
`after_name` and `after_version` must be supplied together for keyset pagination.
|
||||
`limit` bounds
|
||||
the response; `include_definition = false` omits the optional definition JSON.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
`SHOW FUNCTIONS` lists each function once. Add `HISTORY` to list every
|
||||
published revision with its metadata and definition, as `list_functions()`
|
||||
does. History uses the same pagination suffix; numeric revision order is
|
||||
encoded in the token, so revision `2` precedes `10`.
|
||||
|
||||
```sql
|
||||
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW FUNCTIONS HISTORY IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP FUNCTION {#drop}
|
||||
|
||||
|
||||
@@ -70,16 +70,25 @@ automatic execution needs the same server-side switch enabled. Omitting
|
||||
|
||||
## SHOW MATERIALIZED VIEWS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW MATERIALIZED VIEWS
|
||||
```sql
|
||||
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW MATERIALIZED VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
This asks the server for its views, as the SDK's `list_materialized_views` does
|
||||
on a `db://` connection.
|
||||
</Note>
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
The listing contains identifiers; use `DESCRIBE MATERIALIZED VIEW` for its defining query and refresh metadata.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## Refreshing one {#refresh}
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ backtick-quoted identifier with `$` between components, such as
|
||||
one path component, relative to an explicit parent schema or the session's
|
||||
current schema -- it cannot contain `$` or be qualified with a database.
|
||||
|
||||
`SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
|
||||
For single-object commands, `SCHEMA` never accepts `IN`. `NAMESPACE` accepts `IN` followed by a complete
|
||||
schema identifier. Names are lowercased unless quoted.
|
||||
|
||||
## CREATE SCHEMA and CREATE NAMESPACE {#create}
|
||||
@@ -126,27 +126,30 @@ Requires `USAGE` on the database and the namespace.
|
||||
|
||||
## SHOW SCHEMAS and SHOW NAMESPACES {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW SCHEMAS
|
||||
SHOW NAMESPACES
|
||||
SHOW NAMESPACES IN analytics.team
|
||||
```sql
|
||||
SHOW SCHEMAS LIMIT 100;
|
||||
SHOW SCHEMAS LIMIT 100 FROM '<page-token>';
|
||||
SHOW NAMESPACES IN analytics.team LIMIT 100;
|
||||
SHOW NAMESPACES IN analytics.team LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`SHOW SCHEMAS` returns complete paths in the current database, recursively,
|
||||
including `public` when the caller can use it. It does not accept `IN`.
|
||||
Both commands return two non-null string columns: `namespace_path`, the
|
||||
parent path as a compact JSON array, and `name`, the unqualified child name.
|
||||
Results are ordered by parent path components, then name, and omit namespaces
|
||||
the caller cannot use.
|
||||
|
||||
`SHOW NAMESPACES` lists only immediate child names under the session's
|
||||
current schema. `IN` selects a different parent schema, optionally in
|
||||
another database -- use `IN public` for the current database's root, or
|
||||
`IN analytics.public` for another database's root. A missing parent is an
|
||||
error.
|
||||
`SHOW SCHEMAS` and `SHOW NAMESPACES` are equivalent listings. Without `IN`,
|
||||
both list recursively in the current database, include the implicit `public`
|
||||
root, and ignore the session namespace. With `IN`, both list immediate children
|
||||
of the selected parent. A schema path can appear directly: `IN team$features`;
|
||||
a database qualifier is optional. `IN public` selects the current database root,
|
||||
and `IN analytics.public` selects another database's root.
|
||||
|
||||
Both forms return a sorted `name` column and omit namespaces the caller
|
||||
cannot use. Large listings that exceed 10,000 namespaces, or recursive
|
||||
listings deeper than 32 levels, return an error suggesting a narrower
|
||||
listing.
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP SCHEMA and DROP NAMESPACE {#drop}
|
||||
|
||||
@@ -187,7 +190,6 @@ with `IF EXISTS`. Inspect a cascade drop's child jobs with
|
||||
| `CREATE SCHEMA child IN parent` | `CREATE NAMESPACE child IN parent` or `` CREATE SCHEMA `parent$child` `` |
|
||||
| `` DESCRIBE NAMESPACE `parent$child` `` | `` DESCRIBE SCHEMA `parent$child` `` or `DESCRIBE NAMESPACE child IN parent` |
|
||||
| `DROP NAMESPACE analytics.child` | `DROP NAMESPACE child IN analytics.public` |
|
||||
| `SHOW SCHEMAS IN parent` | `SHOW NAMESPACES IN parent` |
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
|
||||
+17
-13
@@ -98,21 +98,25 @@ picked up by every Function still bound to that name.
|
||||
|
||||
## SHOW SECRETS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW SECRETS
|
||||
SHOW SECRETS IN platform
|
||||
SHOW SECRETS IN analytics.`ml$vision`
|
||||
SHOW SECRETS IN public
|
||||
SHOW SECRETS IN platform WHERE name > 'model-api-key' LIMIT 50
|
||||
```sql
|
||||
SHOW SECRETS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW SECRETS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Lists the Secret names in one namespace as a sorted `name` column. Without
|
||||
`IN`, it lists the session's namespace; `IN public` lists the root. `WHERE
|
||||
name > '<cursor>'` pages past the last name already seen, and `LIMIT`
|
||||
bounds the page size; `WHERE` must compare `name` with `>`. Requires
|
||||
`USAGE` on the database and the namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Secret listings contain no values or Secret metadata.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DESCRIBE SECRET {#describe}
|
||||
|
||||
|
||||
@@ -194,16 +194,25 @@ only the reference naming it.
|
||||
|
||||
## SHOW STORAGE INTEGRATIONS {#show}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL icon="database"
|
||||
SHOW STORAGE INTEGRATIONS
|
||||
SHOW STORAGE INTEGRATIONS IN platform
|
||||
LIST STORAGE INTEGRATIONS
|
||||
```sql
|
||||
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`LIST` is an alias for `SHOW` here. Returns one `name` column; `IN`
|
||||
narrows the listing to one namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
`LIST STORAGE INTEGRATIONS` is an exact alias, including pagination.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## DROP STORAGE INTEGRATION {#drop}
|
||||
|
||||
|
||||
@@ -268,3 +268,23 @@ deployment stores its catalog. See
|
||||
Renaming, retyping and dropping columns from the SDK.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## SHOW TABLES {#show}
|
||||
|
||||
```sql
|
||||
SHOW TABLES IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW TABLES IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
+13
-20
@@ -81,29 +81,22 @@ was none to begin with.
|
||||
## Listing {#listing}
|
||||
|
||||
```sql
|
||||
SHOW VIEWS
|
||||
SHOW VIEWS IN "analytics" WHERE name > 'previous_view' LIMIT 100
|
||||
SHOW VIEWS IN analytics.`team$ml` LIMIT 100;
|
||||
SHOW VIEWS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
||||
```
|
||||
|
||||
`SHOW VIEWS` returns ascending `name` values. The `name` cursor is scoped to
|
||||
the selected namespace.
|
||||
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
||||
of parent namespace components (`[]` for root), and `name`, the unqualified
|
||||
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
||||
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
||||
the current database's root. Listings require the relevant database and
|
||||
namespace `USAGE` privileges.
|
||||
|
||||
Object commands retain their small results in shared storage even when general
|
||||
query-result copying is disabled, so asynchronous polling can use any query node.
|
||||
|
||||
Views and tables share a namespace, and both are things you can query. The
|
||||
catalog distinguishes them:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| view names | the views in the database |
|
||||
| table names | the tables |
|
||||
| relation names | everything queryable (both) |
|
||||
|
||||
Use the relation listing when the question is "what can I select from," and the
|
||||
separate listings when the distinction matters: before dropping, for
|
||||
instance, since dropping a view removes only the saved query while dropping a
|
||||
table removes data.
|
||||
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
||||
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
||||
unchanged with the same object type and scope. Stop when no token is returned;
|
||||
a short or empty page can still carry a continuation. See
|
||||
[catalog listing pagination](/sql/pagination).
|
||||
|
||||
## Choosing between a view and a materialized view {#choosing}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user