diff --git a/docs/web/basics/query.mdx b/docs/web/basics/query.mdx index 449c9b498..1447c4489 100644 --- a/docs/web/basics/query.mdx +++ b/docs/web/basics/query.mdx @@ -83,8 +83,10 @@ return results. They are fast enough not to need a handle. See Query-builder searches -- [vector](/search/vector-search), [full-text](/search/full-text-search), [hybrid](/search/hybrid-search) -- return -results directly. They are fast enough not to need a handle, and they have no -SQL form. See [Semantic Search](/search). +results directly. They are fast enough not to need a handle. In SQL on a +deployment, full-text search is the [`fts()`](/sql/functions#fts) table +function and vector search [depends on the deployment](/basics/sql#no-vector-sql); +hybrid search has no SQL form. See [Semantic Search](/search). ## When the work outlives the call {#jobs} diff --git a/docs/web/basics/sdk.mdx b/docs/web/basics/sdk.mdx index 3f8170e11..28d094d09 100644 --- a/docs/web/basics/sdk.mdx +++ b/docs/web/basics/sdk.mdx @@ -4,6 +4,13 @@ sidebarTitle: "SDKs" description: "The language clients — Python, TypeScript, Rust — and what stays true across all of them." icon: code-compare keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"] +availability: + oss: available + enterprise: available + summary: >- + The Python, TypeScript and Rust SDKs work with embedded databases and, + through `db://`, with Enterprise deployments; the Java SDK reaches + deployments only. What each can do differs by language and connection. --- A language SDK is the primary way to work with LanceDB: create and manage @@ -29,14 +36,15 @@ Not every SDK covers the same surface, though: | --- | :---: | :---: | | Create and manage catalog objects | yes | yes | | Insert, merge, update, delete | yes | yes | -| Filter, project, aggregate, join | yes | yes | +| Filter, project | yes | yes | | Vector search | yes | yes | | Full-text search | yes | yes | | Feature engineering, jobs | yes | partly | | Runs against a local database | yes | yes | -TypeScript has no methods for [functions](/management/function). Reach the -gap with SQL instead -- see [SQL](/basics/sql). +Neither SDK aggregates or joins, and TypeScript has no methods for +[functions](/management/function). Reach those with SQL on a deployment +instead -- see [SQL](/basics/sql). ## REST API {#rest} @@ -62,8 +70,8 @@ and release cadence vary. | [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package | | [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings | -For any other language, [SQL](/basics/sql) reaches the same tables without a -LanceDB-specific SDK at all. +For any other language, [SQL](/basics/sql) on a deployment reaches the same +tables without a LanceDB-specific SDK at all. ## Python multiprocessing {#multiprocessing} diff --git a/docs/web/basics/sql.mdx b/docs/web/basics/sql.mdx index 6b74486eb..1a348e304 100644 --- a/docs/web/basics/sql.mdx +++ b/docs/web/basics/sql.mdx @@ -4,11 +4,14 @@ sidebarTitle: "SQL" description: "The same tables, reached with SQL — the universal access path, and where every operation is formally documented." icon: table-list keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"] +availability: + oss: unavailable + enterprise: varies + summary: >- + SQL runs on an Enterprise deployment whose Flight SQL endpoint is enabled + and reachable; embedded connections raise instead. --- -Enterprise SQL runs on a deployment. A local -database has no engine to plan a statement, so `execute_query` raises there. - SQL is a second way to reach the same tables the [SDKs](/basics/sdk) do -- not a different product, a different interface onto one engine. A table written from Python is queried from SQL without an export step, and a table @@ -39,12 +42,17 @@ code speaks, the other is available too. | --- | :---: | :---: | | Create and manage catalog objects | yes | partly | | Insert, merge, update, delete | yes | yes | -| Filter, project, aggregate, join | yes | yes | +| Filter, project | yes | yes | +| Aggregate, join | no | yes | | Vector search | yes | version-dependent | | Full-text search | yes | yes | | Feature engineering, jobs | yes | yes | | Runs against a local database | yes | **no** | +On a deployment, feature engineering must also be enabled by the deployment, +whether it is declared in SQL or through an SDK -- see +[Computed columns](/feature-engineering/add-computed-column). + ## Vector search depends on the deployment {#no-vector-sql} The SDK query builder provides vector search across embedded and deployed @@ -59,9 +67,9 @@ Full-text search is available through the [`fts()`](/sql/functions#fts) table function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. -The reverse gap also exists: a [view](/management/view) is a saved SQL query, -so SQL is the only way to create one -- there's nothing for an SDK method to -call. +A [view](/management/view) is a saved SQL query. On a deployment you can +create one with `CREATE VIEW` or by passing the query to an SDK's +`create_view`; an embedded database has no views. ## SQL needs a deployment {#sql-is-remote} @@ -76,6 +84,12 @@ db = lancedb.connect("./.lancedb") # no SQL db = lancedb.connect("db://my-database") # SQL ``` +A deployment serves SQL only when its Flight SQL endpoint is enabled and +reachable, which is part of the deployment's configuration. The SDK also has to +know where that endpoint is: pass `sql_host_override` when connecting, or +connect through an `http://` `host_override`, from which the SDK derives it +(the next port up, or 10025 when the override names no port). + ## Running SQL from Python {#sql-from-python} ```python diff --git a/docs/web/sql/dml/copy.mdx b/docs/web/sql/dml/copy.mdx index 837b92285..3fe002fe8 100644 --- a/docs/web/sql/dml/copy.mdx +++ b/docs/web/sql/dml/copy.mdx @@ -6,7 +6,11 @@ icon: file-export keywords: ["copy", "export", "parquet", "sql"] --- -Enterprise [SQL runs on a deployment.](/sql) + +Newer deployment versions refuse `COPY` over the SQL endpoint with "COPY is not +supported". On those, export a result by reading it through a Flight SQL client +or an SDK and writing the files from your application. + ```sql COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET diff --git a/docs/web/sql/dql/aggregation.mdx b/docs/web/sql/dql/aggregation.mdx index 2bd8ddd58..82237ff76 100644 --- a/docs/web/sql/dql/aggregation.mdx +++ b/docs/web/sql/dql/aggregation.mdx @@ -112,7 +112,8 @@ estimate -- a 1% sample of a skewed column will miss rare categories entirely. ## What aggregation does not do {#not} -Aggregates run over rows the query already produced. They cannot combine with -[vector search](/search/vector-search) in one statement, because similarity -search has no SQL form. Run the search in an SDK and aggregate its result, or -materialize the search output first. +Aggregates run over rows the query already produced. Whether a statement can +include [vector search](/search/vector-search) +[depends on the deployment](/basics/sql#no-vector-sql); where it cannot, run the +search in an SDK and aggregate its result, or materialize the search output +first. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 2f00b126a..4b92ef38c 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -6,9 +6,9 @@ icon: table-list keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"] --- -Enterprise SQL runs on a deployment. A local database -has no engine to plan a statement, so `execute_query` raises there -- see -[SQL](/basics/sql#sql-is-remote). +Enterprise SQL runs on a deployment that enables its +Flight SQL endpoint. A local database has no SQL endpoint, so `execute_query` +raises there -- see [SQL](/basics/sql#sql-is-remote). This is a reference rather than a guide: one page per construct, syntax first. The guides elsewhere in these docs show the same operations from the SDKs. @@ -87,8 +87,8 @@ const result = await client.query("SELECT * FROM flights WHERE origin = 'SFO'"); The endpoint executes one statement per request and is meant for the queries and DDL/DML pages below. Reach for the SDKs instead for table-management -operations that have no SQL form, such as building a vector index or a merge -insert -- see [What SQL cannot do](#cannot). +operations that have no SQL form, such as a merge insert -- see +[What SQL cannot do](#cannot). ## Reference {#reference} @@ -111,7 +111,7 @@ insert -- see [What SQL cannot do](#cannot). | [Insert](/sql/dml/insert) | `INSERT INTO`, including `INSERT ... SELECT` | | [Update](/sql/dml/update) | `UPDATE ... SET` | | [Delete](/sql/dml/delete) | `DELETE FROM` | -| [Copy](/sql/dml/copy) | `COPY INTO` / `COPY TO` | +| [Copy](/sql/dml/copy) | `COPY` -- refused by newer deployment versions | | [Refresh](/sql/dml/refresh) | filling computed columns and materialized views | ### DQL {#dql} @@ -141,9 +141,10 @@ insert -- see [What SQL cannot do](#cannot). | | | | --- | --- | -| Vector search | no SQL form -- a query-builder operation | +| Vector search | [depends on the deployment](/basics/sql#no-vector-sql); the query builder always has it | | Merge insert | no SQL form -- needs a source relation and a per-match policy | | `information_schema` | not enabled, so `SHOW TABLES` and `SHOW COLUMNS` fail | +| `COPY` | refused by newer deployment versions -- see [Copy](/sql/dml/copy) | | `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement | | Run on a local database | no engine to plan the statement | diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx index 05cad2467..5ffecc9bf 100644 --- a/docs/web/sql/operators.mdx +++ b/docs/web/sql/operators.mdx @@ -44,8 +44,8 @@ IN`, which is the usual place this bites. See Arithmetic over a vector column is **list** arithmetic, not scalar arithmetic — a vector is a `FixedSizeList`. When the operation you want is a -similarity comparison, use [vector search](/search/vector-search); there is no -SQL form for it. +similarity comparison, use [vector search](/search/vector-search); whether SQL +can express it [depends on the deployment](/basics/sql#no-vector-sql). ## Literals {#literals}