mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-27 14:06:14 +00:00
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
125 lines
5.7 KiB
Plaintext
125 lines
5.7 KiB
Plaintext
---
|
|
title: "SDKs"
|
|
sidebarTitle: "SDKs"
|
|
description: "Choose a LanceDB SDK and find the API reference for your language."
|
|
icon: code-compare
|
|
keywords: ["python", "typescript", "rust", "java", "sdk", "rest", "api reference"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Python, TypeScript and Rust support both LanceDB OSS and LanceDB Enterprise.
|
|
Java provides a client for the Enterprise REST API.
|
|
---
|
|
|
|
Use a LanceDB SDK to create tables, add data, build indexes and run searches
|
|
from your application. Python, TypeScript and Rust work with both the embedded
|
|
OSS library and LanceDB Enterprise.
|
|
|
|
The [quickstart](/quickstart) includes installation instructions and complete
|
|
examples in all three languages. For connection settings, see
|
|
[Connect to LanceDB](/basics/connect).
|
|
|
|
## Official SDKs {#supported-sdks}
|
|
|
|
| SDK | Use it with | API reference |
|
|
| --- | --- | --- |
|
|
| Python | Python applications, notebooks, pandas and PyArrow | [Python reference](https://lancedb.github.io/lancedb/python/python/) |
|
|
| TypeScript | Node.js and TypeScript applications | [TypeScript reference](https://lancedb.github.io/lancedb/js/) |
|
|
| Rust | Rust applications using the native library | [Rust reference](https://docs.rs/lancedb/latest/lancedb/index.html) |
|
|
| Java | Applications using the Enterprise REST API | [Java reference](https://lancedb.github.io/lancedb/java/java/) |
|
|
|
|
Guides show the available language examples together. Use the API reference for
|
|
method signatures and options; SDKs do not expose every feature through the same
|
|
methods. In Rust, enable the `remote` feature to connect to Enterprise.
|
|
|
|
For SQL joins, aggregations and window functions, use
|
|
[SQL](/basics/sql). Python provides a SQL client, and applications in other
|
|
languages can use a Flight SQL client. TypeScript's table query API does not
|
|
provide a SQL execution method.
|
|
|
|
## REST API {#rest}
|
|
|
|
LanceDB Enterprise also exposes an HTTP API for catalog and table operations.
|
|
Use it when integrating a language or tool without a LanceDB SDK. The
|
|
[REST API reference](/api-reference/rest) describes its endpoints and request
|
|
formats. For SQL queries, connect through [Flight SQL](/basics/sql).
|
|
|
|
## Community SDKs {#community-sdks}
|
|
|
|
These clients are maintained by the community. Check each project's supported
|
|
features, installation instructions and releases before choosing one.
|
|
|
|
| Language | Project |
|
|
| --- | --- |
|
|
| Go | [lancedb-go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) |
|
|
| Ruby | [Lancelot](https://github.com/scientist-labs/lancelot) |
|
|
| Swift | [LanceDbSwiftKit](https://github.com/RyanLisse/LanceDbSwiftKit) |
|
|
| R | [lancedb](https://github.com/CathalByrneGit/lancedb) |
|
|
| Flutter | [flutter_lancedb](https://github.com/Alexcn/flutter_lancedb) |
|
|
|
|
## Python multiprocessing {#multiprocessing}
|
|
|
|
<Warning>
|
|
Use `spawn` when starting Python worker processes with `multiprocessing`.
|
|
LanceDB uses threads internally, and forking a process with active threads can
|
|
cause deadlocks. Create a connection inside each worker instead of sharing an
|
|
open connection from the parent process.
|
|
</Warning>
|
|
|
|
For PyTorch worker setup and connection factories, see
|
|
[PyTorch integration](/training/torch).
|
|
|
|
## Moving to LanceDB Enterprise {#moving-to-enterprise}
|
|
|
|
The core table workflow stays familiar: connect, create or open a table, write
|
|
records, and query them. Enterprise runs those operations on remote compute and
|
|
manages background maintenance.
|
|
|
|
- **Configure the connection.** Replace the local path with your database URI,
|
|
and supply the endpoint, region and credentials provided by your administrator.
|
|
[Connect to Enterprise](/basics/connect#enterprise) shows these settings.
|
|
- **Retrieve query results.** Python's remote tables do not support whole-table
|
|
`to_arrow()` or `to_pandas()` calls. Use a query and select the columns and
|
|
number of rows you need, for example `table.search().limit(10).to_arrow()`.
|
|
- **Handle table listings and namespaces.** Remote table listings are paginated.
|
|
Use the namespace and pagination options described in
|
|
[Namespaces](/management/namespace).
|
|
- **Use server-side jobs.** Index builds and feature refreshes can run as
|
|
[jobs](/basics/jobs). Each operation's guide explains how to submit work and
|
|
track completion.
|
|
- **Let Enterprise manage maintenance.** Use
|
|
[background maintenance](/maintenance/background-jobs) rather than calling
|
|
the embedded table's `optimize()` method.
|
|
|
|
{/* Draft for the remote option validation release.
|
|
- **Remove local storage and cache settings.** `db://` connections reject local
|
|
sessions, table cache settings, and per-table storage and write options. Omit
|
|
these options so Enterprise manages storage and caching. Connection-level
|
|
storage options supported by your Enterprise deployment remain available.
|
|
*/}
|
|
|
|
For SQL connection details, see [SQL](/basics/sql). For the operating model and
|
|
deployment options, see [LanceDB Enterprise](/enterprise).
|
|
|
|
## Shared data across languages {#invariants}
|
|
|
|
LanceDB uses Arrow schemas across its SDKs. A fixed-size vector column stores
|
|
vectors of a consistent dimension; the [data types reference](/sql/data-types)
|
|
describes the supported element types and other column types.
|
|
|
|
Applications in different languages can work with the same tables. Connect to
|
|
the same database and namespace, use the appropriate credentials, and account
|
|
for [read consistency](/tables/consistency) when another process writes data.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="SQL" icon="table-list" href="/basics/sql">
|
|
Query Enterprise tables with SQL from Python or a Flight SQL client.
|
|
</Card>
|
|
<Card title="Agentic Skills" icon="robot" href="/basics/agentic-skills">
|
|
Use the LanceDB plugin to help a coding agent work with your data.
|
|
</Card>
|
|
</CardGroup>
|