docs: split SDKs and SQL into two pages, promote to Getting Started

SQL now leads with why you'd reach for it beyond analytics: it's served over
Flight SQL, so it's the way in for any language without a native LanceDB SDK,
and broadly every public SDK method has a corresponding SQL statement. Closes
by pointing at the SQL Reference as the spec that defines the precise
behavior behind both the SQL surface and the SDK operations it mirrors.

Moving both into Getting Started (before Agentic Skills) means a reader knows
what the API surface even is before being told how to hand it to an agent.
This commit is contained in:
Jack Ye
2026-09-28 17:40:52 -05:00
parent 21786845a7
commit c220e63828
17 changed files with 239 additions and 165 deletions
+1 -1
View File
@@ -48,7 +48,7 @@ server in the picture at all.
**What follows from that.** Nothing runs unless you call it — no background
compaction, no server-side [jobs](/basics/jobs), no
[SQL endpoint](/basics/apis). Scale is your process's scale: a query gets the
[SQL endpoint](/basics/sql). Scale is your process's scale: a query gets the
cores and memory your process has. Access control is whatever your object store
enforces, since there is no other gatekeeper.
-149
View File
@@ -1,149 +0,0 @@
---
title: "SDKs and SQL"
sidebarTitle: "SDKs & SQL"
description: "The ways to reach LanceDB — the language SDKs, the SQL endpoint, the REST API — and what each can do."
icon: code-compare
keywords: ["python", "typescript", "rust", "java", "sql", "rest", "api reference", "sdk"]
---
The same table can be reached several ways: a language SDK, SQL, or the REST
API. They are views onto one engine, not separate products -- a table written
from Python is queried from SQL without an export step.
They are not interchangeable, though, and the differences are worth knowing
before you pick one.
## What each one can do {#matrix}
| | Python | TypeScript | SQL |
| --- | :---: | :---: | :---: |
| Create and manage catalog objects | yes | yes | partly |
| Insert, merge, update, delete | yes | yes | yes |
| Filter, project, aggregate, join | yes | yes | yes |
| Vector search | yes | yes | -- |
| Full-text search | yes | yes | yes |
| Feature engineering, jobs | yes | partly | yes |
| Runs against a local database | yes | yes | **no** |
Three entries in that table do most of the work of explaining the rest.
## SQL needs a deployment {#sql-is-remote}
<Warning>
SQL is not available on a local database. A connection to a directory or a
bucket has no server to plan and execute a statement, and asking it to raises
`NotImplementedError`.
</Warning>
```python
db = lancedb.connect("./.lancedb") # no SQL
db = lancedb.connect("db://my-database") # SQL
```
This is the largest behavioural difference between the embedded and deployed
forms, and the reason the SDKs are the portable choice: SDK code moves between
the two by changing the URI, SQL does not.
<Badge color="red">Enterprise</Badge> Queries are served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client can reach it and results arrive as Arrow
record batches.
## Vector search has no SQL form {#no-vector-sql}
Similarity search is a query-builder operation. A query that needs both vector
similarity and SQL analytics runs the search in an SDK and the analytics over
its result. Full-text search *is* available in SQL, as the
[`fts()`](/sql/functions#fts) table function and the
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate.
## TypeScript has no SQL method {#ts-no-sql}
The TypeScript SDK does not expose SQL, and has no methods for
[functions](/management/function). Reach the endpoint with a Flight
SQL client instead -- see [connecting a client](/sql#connecting). This is a gap in
the SDK rather than in the endpoint: the same statements work.
## Running SQL from Python {#sql-from-python}
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
table = reader.read_all()
```
`execute_query` returns a blocking Arrow reader once results start arriving; it
does not wait for the whole query. `execute_query_async` returns a handle
instead -- see [Queries](/basics/query).
## Python multiprocessing {#multiprocessing}
<Warning>
Do not use `fork` as the start method with Python's `multiprocessing` module.
Lance is multi-threaded internally, and `fork` with alive threads is unsafe --
use `spawn` instead. See
[this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555)
for why.
</Warning>
## Choosing between them {#choosing}
Reach for **the SDKs** when you are writing data, managing the catalog, running
vector search, or building an application that should work the same embedded and
deployed.
Reach for **SQL** when the question is analytical -- joins across tables,
aggregates, window functions -- or when the thing asking is a BI tool, a
notebook, or a person who already thinks in SQL. It is also the only way to
create a [view](/management/view).
The two compose. A common shape is SDK code to build and maintain tables, and
SQL for the analysis on top.
## Reference {#reference}
### Officially supported SDKs {#supported-sdks}
Python, TypeScript and Rust are supported against both embedded and deployed
LanceDB.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="python" /> [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration |
| <Icon icon="square-js" /> [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` |
| <Icon icon="rust" /> [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance |
### REST API {#rest}
<Badge color="red">Enterprise</Badge> A deployment implements the
[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable
over HTTP by anything that speaks it -- see the
[REST API reference](/api-reference/rest).
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="java" /> [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java |
### Community SDKs {#community-sdks}
Contributed by the community rather than supported by LanceDB, so feature parity
and release cadence vary.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="golang" /> [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK |
| <Icon icon="gem" /> [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings |
| <Icon icon="swift" /> [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK |
| <Icon icon="R" /> [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package |
| <Icon icon="flutter" /> [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings |
## What stays the same {#invariants}
Whichever you pick:
- **Types are Arrow types.** A vector column is a `FixedSizeList<Float32>`
everywhere, not a SQL type mapped onto storage. See
[Functions and types](/sql/functions).
- **Reads are Arrow.** Nothing is serialized into language objects on the way out
unless you convert it.
- **The catalog is one catalog.** A table created from TypeScript is visible to
Python and to SQL immediately -- there is no per-language namespace.
+1 -1
View File
@@ -191,4 +191,4 @@ one with nothing behind it: `id` is `None`, `status()` reports `"finished"`,
`wait()` returns immediately, `cancel()` does nothing, and `events()` raises.
This is the difference to expect when moving between embedded and deployed --
see [SDKs and SQL](/basics/apis).
see [SDKs](/basics/sdk).
+2 -2
View File
@@ -53,7 +53,7 @@ db = lancedb.connect("s3://bucket/path")
There is no server, which is what makes it embedded -- and also what bounds it.
There is nothing to run work on your behalf, so there are no server-side
[jobs](/basics/jobs), no [SQL endpoint](/basics/apis), and no shared catalog
[jobs](/basics/jobs), no [SQL endpoint](/basics/sql), and no shared catalog
beyond what the storage layer itself provides.
LanceDB OSS is Apache 2.0 licensed. LanceDB Enterprise is the commercial,
@@ -73,7 +73,7 @@ db = lancedb.connect("db://my-database", api_key="...")
Implementing the specification rather than inventing an interface is what makes
the catalog reachable by anything that speaks it, and what makes
[authorization](/governance/authorization), server-side jobs, and
[SQL](/basics/apis) possible: there is now a server to enforce and to run them.
[SQL](/basics/sql) possible: there is now a server to enforce and to run them.
The tables are the same Lance tables either way. What changes is who resolves
their names and who does the work.
+1 -1
View File
@@ -94,7 +94,7 @@ not another.
| `tableExists` / `namespaceExists` in TypeScript | not present |
| Server-side jobs on a local database | run in-process; `events()` raises |
See [SDKs and SQL](/basics/apis) for why the first two hold.
See [SQL](/basics/sql) for why the first two hold.
## Nothing here is a quota {#not-quotas}
+1 -1
View File
@@ -72,7 +72,7 @@ query.cancel() # stop it
for a job.
<Note>
SQL requires a deployment. See [SDKs and SQL](/basics/apis).
SQL requires a deployment. See [SQL](/basics/sql#sql-is-remote).
</Note>
Query-builder searches -- vector, full-text, hybrid -- are ordinary calls that
+99
View File
@@ -0,0 +1,99 @@
---
title: "SDKs"
sidebarTitle: "SDKs"
description: "The language clients — Python, TypeScript, Rust — and what stays true across all of them."
icon: code-compare
keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"]
---
A language SDK is the primary way to work with LanceDB: create and manage
tables, write and query data, build indexes, run feature engineering. The same
table API works whether you're connected to an embedded database or an
Enterprise deployment -- application code moves between the two by changing
the connection URI.
## Officially supported SDKs {#supported-sdks}
Python, TypeScript and Rust are supported against both embedded and deployed
LanceDB.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="python" /> [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration |
| <Icon icon="square-js" /> [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` |
| <Icon icon="rust" /> [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance |
Not every SDK covers the same surface, though:
| | Python | TypeScript |
| --- | :---: | :---: |
| Create and manage catalog objects | yes | yes |
| Insert, merge, update, delete | yes | yes |
| Filter, project, aggregate, join | yes | yes |
| Vector search | yes | yes |
| Full-text search | yes | yes |
| Feature engineering, jobs | yes | partly |
| Runs against a local database | yes | yes |
TypeScript has no methods for [functions](/management/function). Reach the
gap with SQL instead -- see [SQL](/basics/sql).
## REST API {#rest}
<Badge color="red">Enterprise</Badge> A deployment implements the
[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable
over HTTP by anything that speaks it -- see the
[REST API reference](/api-reference/rest).
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="java" /> [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java |
## Community SDKs {#community-sdks}
Contributed by the community rather than supported by LanceDB, so feature parity
and release cadence vary.
| Reference | Description |
|:--------------|-------------------|
| <Icon icon="golang" /> [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK |
| <Icon icon="gem" /> [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings |
| <Icon icon="swift" /> [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK |
| <Icon icon="R" /> [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package |
| <Icon icon="flutter" /> [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings |
For any other language, [SQL](/basics/sql) reaches the same tables without a
LanceDB-specific SDK at all.
## Python multiprocessing {#multiprocessing}
<Warning>
Do not use `fork` as the start method with Python's `multiprocessing` module.
Lance is multi-threaded internally, and `fork` with alive threads is unsafe --
use `spawn` instead. See
[this discussion](https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555)
for why.
</Warning>
## What stays the same {#invariants}
Whichever you pick:
- **Types are Arrow types.** A vector column is a `FixedSizeList<Float32>`
everywhere, not a SQL type mapped onto storage. See
[Functions and types](/sql/functions).
- **Reads are Arrow.** Nothing is serialized into language objects on the way out
unless you convert it.
- **The catalog is one catalog.** A table created from TypeScript is visible to
Python and to SQL immediately -- there is no per-language namespace.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="SQL" icon="table-list" href="/basics/sql">
The same tables, reached by any Flight SQL client -- and the one way in for a language with no SDK.
</Card>
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
Give a coding agent a current reference to this same API surface.
</Card>
</CardGroup>
+119
View File
@@ -0,0 +1,119 @@
---
title: "SQL"
sidebarTitle: "SQL"
description: "The same tables, reached with SQL — the universal access path, and where every operation is formally documented."
icon: table-list
keywords: ["sql", "flight sql", "datafusion", "reference", "dialect"]
---
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local
database has no engine to plan a statement, so `execute_query` raises there.
SQL is a second way to reach the same tables the [SDKs](/basics/sdk) do --
not a different product, a different interface onto one engine. A table
written from Python is queried from SQL without an export step, and a table
created from SQL is visible to every SDK immediately.
## Any language, via Flight SQL {#flight-sql}
SQL is served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it -- which makes SQL the way in
for a language LanceDB has no SDK for. You don't need a LanceDB-specific
client at all, just a Flight SQL driver for your language.
The TypeScript SDK itself has no SQL method, for the same reason a Go or C#
application would reach for SQL first: connect a Flight SQL client instead of
waiting on a native binding. See [SDKs](/basics/sdk) for the officially
supported and community clients, and reach for Flight SQL whenever yours
isn't among them.
## Every SDK operation, in SQL {#sdk-parity}
Broadly, a public SDK method has a corresponding SQL statement -- creating a
table is `CREATE TABLE`, inserting rows is `INSERT INTO`, building an index is
`CREATE INDEX`. The two describe the same operations, so whichever one your
code speaks, the other is available too.
| | SDK | SQL |
| --- | :---: | :---: |
| Create and manage catalog objects | yes | partly |
| Insert, merge, update, delete | yes | yes |
| Filter, project, aggregate, join | yes | yes |
| Vector search | yes | -- |
| Full-text search | yes | yes |
| Feature engineering, jobs | yes | yes |
| Runs against a local database | yes | **no** |
## Vector search has no SQL form {#no-vector-sql}
The clearest exception to that parity. Similarity search is a query-builder
operation. A query that needs both vector similarity and SQL analytics runs
the search in an SDK and the analytics over its result. Full-text search *is*
available in SQL, as the [`fts()`](/sql/functions#fts) table function and the
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate.
The reverse gap also exists: a [view](/management/view) is a saved SQL query,
so SQL is the only way to create one -- there's nothing for an SDK method to
call.
## SQL needs a deployment {#sql-is-remote}
A connection to a directory or a bucket has no server to plan and execute a
statement, and asking it to raises `NotImplementedError`. This is the largest
behavioral difference between the embedded and deployed forms, and the reason
the SDKs are the portable choice: SDK code moves between the two by changing
the URI, SQL does not.
```python
db = lancedb.connect("./.lancedb") # no SQL
db = lancedb.connect("db://my-database") # SQL
```
## Running SQL from Python {#sql-from-python}
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
table = reader.read_all()
```
`execute_query` returns a blocking Arrow reader once results start arriving; it
does not wait for the whole query. `execute_query_async` returns a handle
instead -- see [Queries](/basics/query).
## Choosing between them {#choosing}
Reach for **the SDKs** when you are writing data, managing the catalog, running
vector search, or building an application that should work the same embedded and
deployed.
Reach for **SQL** when the question is analytical -- joins across tables,
aggregates, window functions -- when the thing asking is a BI tool, a
notebook, or a person who already thinks in SQL, or when it's the only client
available for your language.
The two compose. A common shape is SDK code to build and maintain tables, and
SQL for the analysis on top.
## The complete reference {#reference}
This page is the concept; [SQL Reference](/sql) is the specification. It has
one page per statement, syntax first -- and since every SDK operation has a
SQL form, those pages end up defining the precise behavior of the SDK
operations too, not just the SQL dialect.
<Card title="SQL Reference" icon="table-list" href="/sql">
DDL by object, then DML, then DQL -- the detailed technical documentation
that defines the behavior of every operation and SDK API.
</Card>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="SDKs" icon="code-compare" href="/basics/sdk">
The language clients, and what stays true across all of them.
</Card>
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
Give a coding agent a current reference to this same API surface.
</Card>
</CardGroup>
+1 -1
View File
@@ -7,7 +7,7 @@ keywords: ["aggregate", "group by", "class balance", "count", "profiling", "dist
---
<Badge color="red">Enterprise</Badge> Aggregation is SQL, and
[SQL needs a deployment](/basics/apis).
[SQL needs a deployment](/basics/sql#sql-is-remote).
Before curating anything, find out what you have. Most of the surprises in a
training set — a class that is 90% of the rows, a column that is half null, a
+1 -1
View File
@@ -16,7 +16,7 @@ reach for it, and links to the [SQL reference](/sql) for the complete syntax.
<Note>
Most curation is analytical -- grouping, joining, ranking -- which the query
builders do not do. It is therefore SQL, and SQL needs a deployment. See
[SDKs and SQL](/basics/apis).
[SQL](/basics/sql#sql-is-remote).
</Note>
## The four questions {#questions}
+1 -1
View File
@@ -7,7 +7,7 @@ keywords: ["join", "labels", "metadata", "enrichment", "namespace"]
---
<Badge color="red">Enterprise</Badge> Joins are SQL, and
[SQL needs a deployment](/basics/apis).
[SQL needs a deployment](/basics/sql#sql-is-remote).
Curation usually means combining things that were produced separately — images
in one table, human labels in another, model scores in a third. A join is how
+1 -1
View File
@@ -7,7 +7,7 @@ keywords: ["window", "deduplication", "row_number", "rank", "train test split",
---
<Badge color="red">Enterprise</Badge> Window functions are SQL, and
[SQL needs a deployment](/basics/apis).
[SQL needs a deployment](/basics/sql#sql-is-remote).
A window function computes across a set of related rows and returns a value
**per row** — nothing is collapsed. That is what makes it the tool for "which of
+7 -2
View File
@@ -230,6 +230,10 @@
{
"source": "/troubleshooting",
"destination": "/deployment/support"
},
{
"source": "/basics/apis",
"destination": "/basics/sdk"
}
],
"navigation": {
@@ -243,6 +247,8 @@
"index",
"quickstart",
"architecture",
"basics/sdk",
"basics/sql",
"basics/agentic-skills"
]
},
@@ -256,8 +262,7 @@
"basics/jobs",
"tables/consistency",
"basics/guarantees",
"basics/limits",
"basics/apis"
"basics/limits"
]
},
{
+1 -1
View File
@@ -40,7 +40,7 @@ storage and does the work itself. The third talks to a **deployment**.
| --- | --- | --- |
| Where queries run | in your process | on the server |
| Table handle | `LanceTable` | `RemoteTable` |
| [SQL](/basics/apis) | -- | yes |
| [SQL](/basics/sql) | -- | yes |
| Server-side [jobs](/basics/jobs) | -- | yes |
| [Functions](/management/function) | -- | yes |
| [Authorization](/governance/authorization) | storage permissions | granted per object |
+1 -1
View File
@@ -17,7 +17,7 @@ Operations that can take a while have an `_async` form returning a
<Note>
SQL requires a deployment -- a local database has no engine to plan a statement.
See [Python, TypeScript, and SQL](/basics/apis).
See [SQL](/basics/sql#sql-is-remote).
</Note>
### Namespaces {#namespaces}
+1 -1
View File
@@ -75,7 +75,7 @@ SELECT id FROM documents WHERE vector IS NOT NULL
<Warning>
Arithmetic over it is **list** arithmetic. There is no SQL similarity operator
and no SQL form for vector search — run the search in an SDK and use SQL over
its result. See [SDKs and SQL](/basics/apis#no-vector-sql).
its result. See [SQL](/basics/sql#no-vector-sql).
</Warning>
### JSON {#json}
+1 -1
View File
@@ -8,7 +8,7 @@ keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
has no engine to plan a statement, so `execute_query` raises there -- see
[SDKs and SQL](/basics/apis).
[SQL](/basics/sql#sql-is-remote).
This is a reference rather than a guide: one page per construct, syntax first.
The guides elsewhere in these docs show the same operations from the SDKs.