diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx index 661911f2e..d03b7ba5c 100644 --- a/docs/web/architecture.mdx +++ b/docs/web/architecture.mdx @@ -48,7 +48,7 @@ server in the picture at all. **What follows from that.** Nothing runs unless you call it — no background compaction, no server-side [jobs](/basics/jobs), no -[SQL endpoint](/basics/apis). Scale is your process's scale: a query gets the +[SQL endpoint](/basics/sql). Scale is your process's scale: a query gets the cores and memory your process has. Access control is whatever your object store enforces, since there is no other gatekeeper. diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx deleted file mode 100644 index de3a07b1e..000000000 --- a/docs/web/basics/apis.mdx +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: "SDKs and SQL" -sidebarTitle: "SDKs & SQL" -description: "The ways to reach LanceDB — the language SDKs, the SQL endpoint, the REST API — and what each can do." -icon: code-compare -keywords: ["python", "typescript", "rust", "java", "sql", "rest", "api reference", "sdk"] ---- - -The same table can be reached several ways: a language SDK, SQL, or the REST -API. They are views onto one engine, not separate products -- a table written -from Python is queried from SQL without an export step. - -They are not interchangeable, though, and the differences are worth knowing -before you pick one. - -## What each one can do {#matrix} - -| | Python | TypeScript | SQL | -| --- | :---: | :---: | :---: | -| Create and manage catalog objects | yes | yes | partly | -| Insert, merge, update, delete | yes | yes | yes | -| Filter, project, aggregate, join | yes | yes | yes | -| Vector search | yes | yes | -- | -| Full-text search | yes | yes | yes | -| Feature engineering, jobs | yes | partly | yes | -| Runs against a local database | yes | yes | **no** | - -Three entries in that table do most of the work of explaining the rest. - -## SQL needs a deployment {#sql-is-remote} - - -SQL is not available on a local database. A connection to a directory or a -bucket has no server to plan and execute a statement, and asking it to raises -`NotImplementedError`. - - -```python -db = lancedb.connect("./.lancedb") # no SQL -db = lancedb.connect("db://my-database") # SQL -``` - -This is the largest behavioural difference between the embedded and deployed -forms, and the reason the SDKs are the portable choice: SDK code moves between -the two by changing the URI, SQL does not. - -Enterprise Queries are served over an -[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) -endpoint, so any Flight SQL client can reach it and results arrive as Arrow -record batches. - -## Vector search has no SQL form {#no-vector-sql} - -Similarity search is a query-builder operation. A query that needs both vector -similarity and SQL analytics runs the search in an SDK and the analytics over -its result. Full-text search *is* available in SQL, as the -[`fts()`](/sql/functions#fts) table function and the -[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. - -## TypeScript has no SQL method {#ts-no-sql} - -The TypeScript SDK does not expose SQL, and has no methods for -[functions](/management/function). Reach the endpoint with a Flight -SQL client instead -- see [connecting a client](/sql#connecting). This is a gap in -the SDK rather than in the endpoint: the same statements work. - -## Running SQL from Python {#sql-from-python} - -```python -reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") -table = reader.read_all() -``` - -`execute_query` returns a blocking Arrow reader once results start arriving; it -does not wait for the whole query. `execute_query_async` returns a handle -instead -- see [Queries](/basics/query). - -## Python multiprocessing {#multiprocessing} - - -Do not use `fork` as the start method with Python's `multiprocessing` module. -Lance is multi-threaded internally, and `fork` with alive threads is unsafe -- -use `spawn` instead. See -[this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555) -for why. - - -## Choosing between them {#choosing} - -Reach for **the SDKs** when you are writing data, managing the catalog, running -vector search, or building an application that should work the same embedded and -deployed. - -Reach for **SQL** when the question is analytical -- joins across tables, -aggregates, window functions -- or when the thing asking is a BI tool, a -notebook, or a person who already thinks in SQL. It is also the only way to -create a [view](/management/view). - -The two compose. A common shape is SDK code to build and maintain tables, and -SQL for the analysis on top. - -## Reference {#reference} - -### Officially supported SDKs {#supported-sdks} - -Python, TypeScript and Rust are supported against both embedded and deployed -LanceDB. - -| Reference | Description | -|:--------------|-------------------| -| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration | -| [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` | -| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance | - -### REST API {#rest} - -Enterprise A deployment implements the -[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable -over HTTP by anything that speaks it -- see the -[REST API reference](/api-reference/rest). - -| Reference | Description | -|:--------------|-------------------| -| [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java | - -### Community SDKs {#community-sdks} - -Contributed by the community rather than supported by LanceDB, so feature parity -and release cadence vary. - -| Reference | Description | -|:--------------|-------------------| -| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK | -| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings | -| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK | -| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package | -| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings | - -## What stays the same {#invariants} - -Whichever you pick: - -- **Types are Arrow types.** A vector column is a `FixedSizeList` - everywhere, not a SQL type mapped onto storage. See - [Functions and types](/sql/functions). -- **Reads are Arrow.** Nothing is serialized into language objects on the way out - unless you convert it. -- **The catalog is one catalog.** A table created from TypeScript is visible to - Python and to SQL immediately -- there is no per-language namespace. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index aa6c9cda5..a17425551 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -191,4 +191,4 @@ one with nothing behind it: `id` is `None`, `status()` reports `"finished"`, `wait()` returns immediately, `cancel()` does nothing, and `events()` raises. This is the difference to expect when moving between embedded and deployed -- -see [SDKs and SQL](/basics/apis). +see [SDKs](/basics/sdk). diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 136457bfd..6552c3735 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -53,7 +53,7 @@ db = lancedb.connect("s3://bucket/path") There is no server, which is what makes it embedded -- and also what bounds it. There is nothing to run work on your behalf, so there are no server-side -[jobs](/basics/jobs), no [SQL endpoint](/basics/apis), and no shared catalog +[jobs](/basics/jobs), no [SQL endpoint](/basics/sql), and no shared catalog beyond what the storage layer itself provides. LanceDB OSS is Apache 2.0 licensed. LanceDB Enterprise is the commercial, @@ -73,7 +73,7 @@ db = lancedb.connect("db://my-database", api_key="...") Implementing the specification rather than inventing an interface is what makes the catalog reachable by anything that speaks it, and what makes [authorization](/governance/authorization), server-side jobs, and -[SQL](/basics/apis) possible: there is now a server to enforce and to run them. +[SQL](/basics/sql) possible: there is now a server to enforce and to run them. The tables are the same Lance tables either way. What changes is who resolves their names and who does the work. diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index 34c84f9ee..0c3647b03 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -94,7 +94,7 @@ not another. | `tableExists` / `namespaceExists` in TypeScript | not present | | Server-side jobs on a local database | run in-process; `events()` raises | -See [SDKs and SQL](/basics/apis) for why the first two hold. +See [SQL](/basics/sql) for why the first two hold. ## Nothing here is a quota {#not-quotas} diff --git a/docs/web/basics/query.mdx b/docs/web/basics/query.mdx index b643d0f35..449c9b498 100644 --- a/docs/web/basics/query.mdx +++ b/docs/web/basics/query.mdx @@ -72,7 +72,7 @@ query.cancel() # stop it for a job. -SQL requires a deployment. See [SDKs and SQL](/basics/apis). +SQL requires a deployment. See [SQL](/basics/sql#sql-is-remote). Query-builder searches -- vector, full-text, hybrid -- are ordinary calls that diff --git a/docs/web/basics/sdk.mdx b/docs/web/basics/sdk.mdx new file mode 100644 index 000000000..3f8170e11 --- /dev/null +++ b/docs/web/basics/sdk.mdx @@ -0,0 +1,99 @@ +--- +title: "SDKs" +sidebarTitle: "SDKs" +description: "The language clients — Python, TypeScript, Rust — and what stays true across all of them." +icon: code-compare +keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"] +--- + +A language SDK is the primary way to work with LanceDB: create and manage +tables, write and query data, build indexes, run feature engineering. The same +table API works whether you're connected to an embedded database or an +Enterprise deployment -- application code moves between the two by changing +the connection URI. + +## Officially supported SDKs {#supported-sdks} + +Python, TypeScript and Rust are supported against both embedded and deployed +LanceDB. + +| Reference | Description | +|:--------------|-------------------| +| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration | +| [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` | +| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance | + +Not every SDK covers the same surface, though: + +| | Python | TypeScript | +| --- | :---: | :---: | +| Create and manage catalog objects | yes | yes | +| Insert, merge, update, delete | yes | yes | +| Filter, project, aggregate, join | yes | yes | +| Vector search | yes | yes | +| Full-text search | yes | yes | +| Feature engineering, jobs | yes | partly | +| Runs against a local database | yes | yes | + +TypeScript has no methods for [functions](/management/function). Reach the +gap with SQL instead -- see [SQL](/basics/sql). + +## REST API {#rest} + +Enterprise A deployment implements the +[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable +over HTTP by anything that speaks it -- see the +[REST API reference](/api-reference/rest). + +| Reference | Description | +|:--------------|-------------------| +| [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java | + +## Community SDKs {#community-sdks} + +Contributed by the community rather than supported by LanceDB, so feature parity +and release cadence vary. + +| Reference | Description | +|:--------------|-------------------| +| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK | +| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings | +| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK | +| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package | +| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings | + +For any other language, [SQL](/basics/sql) reaches the same tables without a +LanceDB-specific SDK at all. + +## Python multiprocessing {#multiprocessing} + + +Do not use `fork` as the start method with Python's `multiprocessing` module. +Lance is multi-threaded internally, and `fork` with alive threads is unsafe -- +use `spawn` instead. See +[this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555) +for why. + + +## What stays the same {#invariants} + +Whichever you pick: + +- **Types are Arrow types.** A vector column is a `FixedSizeList` + everywhere, not a SQL type mapped onto storage. See + [Functions and types](/sql/functions). +- **Reads are Arrow.** Nothing is serialized into language objects on the way out + unless you convert it. +- **The catalog is one catalog.** A table created from TypeScript is visible to + Python and to SQL immediately -- there is no per-language namespace. + +## What to read next {#next} + + + + The same tables, reached by any Flight SQL client -- and the one way in for a language with no SDK. + + + Give a coding agent a current reference to this same API surface. + + diff --git a/docs/web/basics/sql.mdx b/docs/web/basics/sql.mdx new file mode 100644 index 000000000..5b8cff8b9 --- /dev/null +++ b/docs/web/basics/sql.mdx @@ -0,0 +1,119 @@ +--- +title: "SQL" +sidebarTitle: "SQL" +description: "The same tables, reached with SQL — the universal access path, and where every operation is formally documented." +icon: table-list +keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"] +--- + +Enterprise SQL runs on a deployment. A local +database has no engine to plan a statement, so `execute_query` raises there. + +SQL is a second way to reach the same tables the [SDKs](/basics/sdk) do -- +not a different product, a different interface onto one engine. A table +written from Python is queried from SQL without an export step, and a table +created from SQL is visible to every SDK immediately. + +## Any language, via Flight SQL {#flight-sql} + +SQL is served over an +[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) +endpoint, so any Flight SQL client reaches it -- which makes SQL the way in +for a language LanceDB has no SDK for. You don't need a LanceDB-specific +client at all, just a Flight SQL driver for your language. + +The TypeScript SDK itself has no SQL method, for the same reason a Go or C# +application would reach for SQL first: connect a Flight SQL client instead of +waiting on a native binding. See [SDKs](/basics/sdk) for the officially +supported and community clients, and reach for Flight SQL whenever yours +isn't among them. + +## Every SDK operation, in SQL {#sdk-parity} + +Broadly, a public SDK method has a corresponding SQL statement -- creating a +table is `CREATE TABLE`, inserting rows is `INSERT INTO`, building an index is +`CREATE INDEX`. The two describe the same operations, so whichever one your +code speaks, the other is available too. + +| | SDK | SQL | +| --- | :---: | :---: | +| Create and manage catalog objects | yes | partly | +| Insert, merge, update, delete | yes | yes | +| Filter, project, aggregate, join | yes | yes | +| Vector search | yes | -- | +| Full-text search | yes | yes | +| Feature engineering, jobs | yes | yes | +| Runs against a local database | yes | **no** | + +## Vector search has no SQL form {#no-vector-sql} + +The clearest exception to that parity. Similarity search is a query-builder +operation. A query that needs both vector similarity and SQL analytics runs +the search in an SDK and the analytics over its result. Full-text search *is* +available in SQL, as the [`fts()`](/sql/functions#fts) table function and the +[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. + +The reverse gap also exists: a [view](/management/view) is a saved SQL query, +so SQL is the only way to create one -- there's nothing for an SDK method to +call. + +## SQL needs a deployment {#sql-is-remote} + +A connection to a directory or a bucket has no server to plan and execute a +statement, and asking it to raises `NotImplementedError`. This is the largest +behavioral difference between the embedded and deployed forms, and the reason +the SDKs are the portable choice: SDK code moves between the two by changing +the URI, SQL does not. + +```python +db = lancedb.connect("./.lancedb") # no SQL +db = lancedb.connect("db://my-database") # SQL +``` + +## Running SQL from Python {#sql-from-python} + +```python +reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") +table = reader.read_all() +``` + +`execute_query` returns a blocking Arrow reader once results start arriving; it +does not wait for the whole query. `execute_query_async` returns a handle +instead -- see [Queries](/basics/query). + +## Choosing between them {#choosing} + +Reach for **the SDKs** when you are writing data, managing the catalog, running +vector search, or building an application that should work the same embedded and +deployed. + +Reach for **SQL** when the question is analytical -- joins across tables, +aggregates, window functions -- when the thing asking is a BI tool, a +notebook, or a person who already thinks in SQL, or when it's the only client +available for your language. + +The two compose. A common shape is SDK code to build and maintain tables, and +SQL for the analysis on top. + +## The complete reference {#reference} + +This page is the concept; [SQL Reference](/sql) is the specification. It has +one page per statement, syntax first -- and since every SDK operation has a +SQL form, those pages end up defining the precise behavior of the SDK +operations too, not just the SQL dialect. + + + DDL by object, then DML, then DQL -- the detailed technical documentation + that defines the behavior of every operation and SDK API. + + +## What to read next {#next} + + + + The language clients, and what stays true across all of them. + + + Give a coding agent a current reference to this same API surface. + + diff --git a/docs/web/curation/aggregate.mdx b/docs/web/curation/aggregate.mdx index fde064a99..91c32fdd4 100644 --- a/docs/web/curation/aggregate.mdx +++ b/docs/web/curation/aggregate.mdx @@ -7,7 +7,7 @@ keywords: ["aggregate", "group by", "class balance", "count", "profiling", "dist --- Enterprise Aggregation is SQL, and -[SQL needs a deployment](/basics/apis). +[SQL needs a deployment](/basics/sql#sql-is-remote). Before curating anything, find out what you have. Most of the surprises in a training set — a class that is 90% of the rows, a column that is half null, a diff --git a/docs/web/curation/index.mdx b/docs/web/curation/index.mdx index 09f7aa851..0bfc89b2d 100644 --- a/docs/web/curation/index.mdx +++ b/docs/web/curation/index.mdx @@ -16,7 +16,7 @@ reach for it, and links to the [SQL reference](/sql) for the complete syntax. Most curation is analytical -- grouping, joining, ranking -- which the query builders do not do. It is therefore SQL, and SQL needs a deployment. See -[SDKs and SQL](/basics/apis). +[SQL](/basics/sql#sql-is-remote). ## The four questions {#questions} diff --git a/docs/web/curation/join.mdx b/docs/web/curation/join.mdx index a05b93609..b4c198ef9 100644 --- a/docs/web/curation/join.mdx +++ b/docs/web/curation/join.mdx @@ -7,7 +7,7 @@ keywords: ["join", "labels", "metadata", "enrichment", "namespace"] --- Enterprise Joins are SQL, and -[SQL needs a deployment](/basics/apis). +[SQL needs a deployment](/basics/sql#sql-is-remote). Curation usually means combining things that were produced separately — images in one table, human labels in another, model scores in a third. A join is how diff --git a/docs/web/curation/windowing.mdx b/docs/web/curation/windowing.mdx index b8ad4aeca..fbb3cbbf2 100644 --- a/docs/web/curation/windowing.mdx +++ b/docs/web/curation/windowing.mdx @@ -7,7 +7,7 @@ keywords: ["window", "deduplication", "row_number", "rank", "train test split", --- Enterprise Window functions are SQL, and -[SQL needs a deployment](/basics/apis). +[SQL needs a deployment](/basics/sql#sql-is-remote). A window function computes across a set of related rows and returns a value **per row** — nothing is collapsed. That is what makes it the tool for "which of diff --git a/docs/web/docs.json b/docs/web/docs.json index a70073b54..308fdbf5a 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -230,6 +230,10 @@ { "source": "/troubleshooting", "destination": "/deployment/support" + }, + { + "source": "/basics/apis", + "destination": "/basics/sdk" } ], "navigation": { @@ -243,6 +247,8 @@ "index", "quickstart", "architecture", + "basics/sdk", + "basics/sql", "basics/agentic-skills" ] }, @@ -256,8 +262,7 @@ "basics/jobs", "tables/consistency", "basics/guarantees", - "basics/limits", - "basics/apis" + "basics/limits" ] }, { diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 1c3e1c084..36c016e2e 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -40,7 +40,7 @@ storage and does the work itself. The third talks to a **deployment**. | --- | --- | --- | | Where queries run | in your process | on the server | | Table handle | `LanceTable` | `RemoteTable` | -| [SQL](/basics/apis) | -- | yes | +| [SQL](/basics/sql) | -- | yes | | Server-side [jobs](/basics/jobs) | -- | yes | | [Functions](/management/function) | -- | yes | | [Authorization](/governance/authorization) | storage permissions | granted per object | diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index 4e8bb85af..ce8ac9105 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -17,7 +17,7 @@ Operations that can take a while have an `_async` form returning a SQL requires a deployment -- a local database has no engine to plan a statement. -See [Python, TypeScript, and SQL](/basics/apis). +See [SQL](/basics/sql#sql-is-remote). ### Namespaces {#namespaces} diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx index 10e432055..e46101a3e 100644 --- a/docs/web/sql/data-types.mdx +++ b/docs/web/sql/data-types.mdx @@ -75,7 +75,7 @@ SELECT id FROM documents WHERE vector IS NOT NULL Arithmetic over it is **list** arithmetic. There is no SQL similarity operator and no SQL form for vector search — run the search in an SDK and use SQL over -its result. See [SDKs and SQL](/basics/apis#no-vector-sql). +its result. See [SQL](/basics/sql#no-vector-sql). ### JSON {#json} diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 7958ea63e..1057e9a0d 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -8,7 +8,7 @@ keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"] Enterprise SQL runs on a deployment. A local database has no engine to plan a statement, so `execute_query` raises there -- see -[SDKs and SQL](/basics/apis). +[SQL](/basics/sql#sql-is-remote). This is a reference rather than a guide: one page per construct, syntax first. The guides elsewhere in these docs show the same operations from the SDKs.