diff --git a/docs/web/api-reference/index.mdx b/docs/web/api-reference/index.mdx deleted file mode 100644 index 50756023a..000000000 --- a/docs/web/api-reference/index.mdx +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: SDKs and REST API Reference -sidebarTitle: "SDKs" -description: "SDK and REST API reference for LanceDB Enterprise and OSS." - ---- - -For detailed information of the available functions and methods in your preferred language's SDKs, -refer to the API documentation linked below. - -If you're looking for a REST API reference, visit the [REST API](/api-reference/rest) page. - -If you're looking for conceptual and practical namespace guidance before diving into method signatures, see -[Namespaces and Catalog Model](/namespaces) and [Using Namespaces in SDKs](/namespaces/usage). - -## Supported SDKs {#supported-sdks} - -Python, Typescript and Rust SDKs are officially supported by LanceDB. You can use these SDKs to interact with both LanceDB OSS and Enterprise deployments. - -| Reference | Description | -|:--------------|-------------------| -| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration | -| [Typescript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` -| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance | - -## REST API SDKs {#rest-api-sdks} - -Enterprise - -REST API-based SDKs provide a convenient way to interact with LanceDB Enterprise deployments using the Lance REST Namespace API. - -| Reference | Description | -|:--------------|-------------------| -| [Java](https://lancedb.github.io/lancedb/java/java/)| REST API Enterprise SDK in Java | - -## Community-driven SDKs {#community-driven-sdks} - -In addition to the officially supported SDKs, the LanceDB community may contribute SDKs in other languages. -These SDKs may not have the same level of support or feature parity as the official ones supported by LanceDB, but they can be an option -for users working in languages other than those listed above. - -| Reference | Description | -|:--------------|-------------------| -| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK for LanceDB | -| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings for LanceDB | -| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK for LanceDB | -| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package for LanceDB | -| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings for LanceDB | diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx index 3839d8584..4d5e6c6eb 100644 --- a/docs/web/basics/apis.mdx +++ b/docs/web/basics/apis.mdx @@ -1,13 +1,13 @@ --- -title: "Python, TypeScript, and SQL" -sidebarTitle: "Python vs SQL" -description: "Three ways to reach the same data, what each one can do, and where each one runs." +title: "SDKs and SQL" +sidebarTitle: "SDKs & SQL" +description: "The ways to reach LanceDB — the language SDKs, the SQL endpoint, the REST API — and what each can do." icon: code-compare -keywords: ["python", "typescript", "sql", "api", "flight sql", "client"] +keywords: ["python", "typescript", "rust", "java", "sql", "rest", "api reference", "sdk"] --- -The same table can be reached three ways: the Python SDK, the TypeScript SDK, -or SQL. They are views onto one engine, not three products -- a table written +The same table can be reached several ways: a language SDK, SQL, or the REST +API. They are views onto one engine, not separate products -- a table written from Python is queried from SQL without an export step. They are not interchangeable, though, and the differences are worth knowing @@ -18,13 +18,14 @@ before you pick one. | | Python | TypeScript | SQL | | --- | :---: | :---: | :---: | | Create and manage catalog objects | yes | yes | partly | -| Insert, merge, update, delete | yes | yes | -- | +| Insert, merge, update, delete | yes | yes | yes | | Filter, project, aggregate, join | yes | yes | yes | -| Vector and full-text search | yes | yes | yes | -| Feature engineering, jobs | yes | yes | -- | +| Vector search | yes | yes | -- | +| Full-text search | yes | yes | yes | +| Feature engineering, jobs | yes | partly | yes | | Runs against a local database | yes | yes | **no** | -Two entries in that table do most of the work of explaining the rest. +Three entries in that table do most of the work of explaining the rest. ## SQL needs a deployment {#sql-is-remote} @@ -39,18 +40,31 @@ db = lancedb.connect("./.lancedb") # no SQL db = lancedb.connect("db://my-database") # SQL ``` -This is the single largest behavioural difference between the embedded and -deployed forms, and the reason the SDKs are the portable choice: SDK code moves -between the two by changing the URI, SQL does not. +This is the largest behavioural difference between the embedded and deployed +forms, and the reason the SDKs are the portable choice: SDK code moves between +the two by changing the URI, SQL does not. Enterprise Queries are served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) endpoint, so any Flight SQL client can reach it and results arrive as Arrow -record batches -- no conversion to Python objects unless you ask for one. +record batches. -## SQL from each language {#sql-clients} +## Vector search has no SQL form {#no-vector-sql} -From Python, the connection runs SQL directly: +Similarity search is a query-builder operation. A query that needs both vector +similarity and SQL analytics runs the search in an SDK and the analytics over +its result. Full-text search *is* available in SQL, as the +[`fts()`](/sql/functions#fts) table function and the +[`contains_tokens`](/sql/functions#contains-tokens) predicate. + +## TypeScript has no SQL method {#ts-no-sql} + +The TypeScript SDK does not expose SQL, and has no methods for +[functions](/feature-engineering/functions). Reach the endpoint with a Flight +SQL client instead -- see [connecting a client](/search/sql). This is a gap in +the SDK rather than in the endpoint: the same statements work. + +## Running SQL from Python {#sql-from-python} ```python reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") @@ -58,20 +72,14 @@ table = reader.read_all() ``` `execute_query` returns a blocking Arrow reader once results start arriving; it -does not wait for the whole query. `execute_query_async` returns a -[job](/basics/jobs) instead, for a statement you would rather not block on. - - -The TypeScript SDK has no SQL method. Reach the endpoint with a Flight SQL -client instead -- see [connecting a client](/search/sql). This is a gap in the -SDK, not in the endpoint: the same statements work. - +does not wait for the whole query. `execute_query_async` returns a handle +instead -- see [Query and job](/basics/jobs#queries). ## Choosing between them {#choosing} Reach for **the SDKs** when you are writing data, managing the catalog, running -feature engineering, or building an application that should work the same -embedded and deployed. They are the complete surface. +vector search, or building an application that should work the same embedded and +deployed. Reach for **SQL** when the question is analytical -- joins across tables, aggregates, window functions -- or when the thing asking is a BI tool, a @@ -81,6 +89,43 @@ create a [view](/management/view). The two compose. A common shape is SDK code to build and maintain tables, and SQL for the analysis on top. +## Reference {#reference} + +### Officially supported SDKs {#supported-sdks} + +Python, TypeScript and Rust are supported against both embedded and deployed +LanceDB. + +| Reference | Description | +|:--------------|-------------------| +| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas and numpy integration | +| [TypeScript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` | +| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent storage and high performance | + +### REST API {#rest} + +Enterprise A deployment implements the +[Lance REST Namespace](/basics/lance#enterprise), so the catalog is reachable +over HTTP by anything that speaks it -- see the +[REST API reference](/api-reference/rest). + +| Reference | Description | +|:--------------|-------------------| +| [Java](https://lancedb.github.io/lancedb/java/java/) | REST API Enterprise SDK in Java | + +### Community SDKs {#community-sdks} + +Contributed by the community rather than supported by LanceDB, so feature parity +and release cadence vary. + +| Reference | Description | +|:--------------|-------------------| +| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK | +| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings | +| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK | +| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package | +| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings | + ## What stays the same {#invariants} Whichever you pick: @@ -88,7 +133,7 @@ Whichever you pick: - **Types are Arrow types.** A vector column is a `FixedSizeList` everywhere, not a SQL type mapped onto storage. See [Functions and types](/sql/functions). -- **Reads are Arrow.** Nothing is serialized into language objects on the way - out unless you convert it. +- **Reads are Arrow.** Nothing is serialized into language objects on the way out + unless you convert it. - **The catalog is one catalog.** A table created from TypeScript is visible to Python and to SQL immediately -- there is no per-language namespace. diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index 6670a7e7f..f850daadf 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -1,19 +1,24 @@ --- -title: "The catalog" -sidebarTitle: "Catalog" -description: "What lives in a LanceDB catalog, how the objects nest, and how you connect to one." +title: "Lakehouse and catalog" +sidebarTitle: "Lakehouse & Catalog" +description: "What a multimodal lakehouse holds, and how the catalog organizes it." icon: sitemap -keywords: ["catalog", "database", "namespace", "table", "view", "function"] +keywords: ["lakehouse", "catalog", "database", "namespace", "table", "view", "function"] --- -A catalog is what LanceDB knows about. Everything you create lives in one and is -addressed through it. +LanceDB is a **multimodal lakehouse**: data sits in open [Lance](/basics/lance) +tables on object storage, and a catalog gives those tables names, structure, and +the operations that act on them. + +The lakehouse part is why images, embeddings, and the columns you derive from +them live in one table rather than in a warehouse plus a vector store plus a +feature store. The catalog part is what this page is about. ## The objects {#objects} | Object | What it is | | --- | --- | -| Database | The root. A connection opens exactly one. | +| [Database](/management/database) | The root. A connection opens exactly one. | | [Namespace](/namespaces) | A named grouping inside a database. Can nest. | | [Table](/tables) | Rows and columns, versioned. What most operations act on. | | [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. | @@ -33,83 +38,22 @@ A namespace can contain further namespaces, so the hierarchy goes as deep as you need. A table is reached by its path through it. This page is about what the objects *are*. For the operations on them -- -creating, listing, dropping -- see [Management](/management). +connecting, creating, listing, dropping -- see [Management](/management). -## Connecting {#connecting} +## A table is more than rows {#tables} -```python -import lancedb +A Lance table carries things a warehouse table does not, and most of the rest of +these docs is about one of them: -db = lancedb.connect("./.lancedb") # a local directory -db = lancedb.connect("s3://bucket/path") # object storage -db = lancedb.connect("db://my-database", api_key="...") # a deployment -``` +| | | +| --- | --- | +| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/computed-columns) from other columns | +| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) | +| **Indexes** | vector, full-text, and scalar, built over the columns | +| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) | -The URI decides what kind of database you get, and that decision reaches further -than it looks. A local or object-store URI opens the data directly from your -process. A `db://` URI talks to a deployment, which is what makes server-side -execution -- [functions](/feature-engineering/functions), remote refreshes, -[SQL](/basics/apis) -- available, and what makes a few local-only features -unavailable. - -`connect_async` returns an `AsyncConnection` with the same surface as -coroutines. - -### What a table handle actually is {#table-handles} - -The URI also decides what `open_table` hands back, which is worth knowing -because the two behave the same but fail differently: - -- **`LanceTable`** -- direct access to storage (a local path, `file://`, `s3://` - and similar). Your process resolves and reads the table itself. -- **`RemoteTable`** -- catalog-backed access through a deployment (`db://`). - Resolution and execution happen on the server. - -![](/static/assets/images/overview/understanding-tables.png) - -Both expose the same table API -- create, open, mutate, query. What differs is -where the work happens, and therefore which operations exist at all. - -### Storage options {#storage-options} - -`storage_options` configures the underlying store -- credentials, endpoints, and -per-table defaults for tables created through this connection: - -```python -db = lancedb.connect( - "s3://bucket/path", - storage_options={"new_table_enable_stable_row_ids": "true"}, -) -``` - - -`new_table_enable_stable_row_ids` is worth setting deliberately. Stable row ids -cannot be enabled after a table exists, and a -[materialized view](/feature-engineering/materialized-views) requires its source -to have them. A table created without them cannot later become a view's source -without being recreated. - - -See [Storage](/storage) for the full set. - -### Read consistency {#consistency} - -`read_consistency_interval` controls how often a connection checks for versions -committed by other writers. The default reads a snapshot taken when the table was -opened, so a long-lived connection will not see other processes' writes until it -reopens. - -Set it when several processes write the same table and readers must observe those -writes: - -```python -from datetime import timedelta - -db = lancedb.connect(uri, read_consistency_interval=timedelta(seconds=5)) -``` - -The cost is a metadata check on that interval. See -[Consistency](/tables/consistency). +That is why "add a feature column" is a normal operation here rather than a +migration: it writes one new column and leaves the expensive data alone. ## Views and materialized views are different objects {#views} @@ -131,4 +75,4 @@ latency, or when the query is cheap. performance decision -- see [Maintenance](/indexing). **Jobs** are not stored data at all; they are handles to work the server is -doing. See [Jobs](/basics/jobs). +doing. See [Query and job](/basics/jobs). diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index 501d1f5fe..db4be33d0 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -1,39 +1,108 @@ --- -title: "Jobs" -sidebarTitle: "Jobs" -description: "The handle every long-running operation returns, and how to wait on, inspect, or cancel one." +title: "Query and job" +sidebarTitle: "Query & Job" +description: "Sync and async in LanceDB: two different distinctions that use the same word." icon: hourglass-half -keywords: ["job", "async", "background", "create index", "refresh column"] +keywords: ["job", "async", "sync", "query", "background", "await"] --- +"Async" means two unrelated things in LanceDB, and conflating them is the most +common source of confusion about how work gets done. + +| | What it is | What it changes | +| --- | --- | --- | +| **Async client** | `connect_async`, `await` | whether *your thread* blocks | +| **Async execution** | `_async` methods returning a [job](#jobs) | whether *the work* outlives the call | + +They are independent. A synchronous Python call can start a job that runs for an +hour; an `await` can wait for something that finishes instantly. + +## Async client: whether you block {#async-client} + +A concern of your language, not of LanceDB. The work is identical either way. + + +```python Python icon="python" +db = lancedb.connect("db://my-db", api_key="...") # blocking +tbl = db.open_table("documents") + +db = await lancedb.connect_async("db://my-db", api_key="...") # coroutines +tbl = await db.open_table("documents") +``` + +```typescript TypeScript icon="square-js" +// TypeScript is async throughout; there is no blocking form. +const db = await lancedb.connect("db://my-db", { apiKey: "..." }); +const tbl = await db.openTable("documents"); +``` + + +Python offers both. `AsyncConnection` exposes the same surface as coroutines, so +choosing between them is about the program you are writing -- a service wants +the async form, a script does not care. + +## Queries: reading before the query finishes {#queries} + +A query is not a job, but it has the same shape of choice: wait for everything, +or take a handle and start reading. + + +```python Python icon="python" +reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 1000") +table = reader.read_all() + +query = db.execute_query_async("SELECT id, text FROM my_docs") +``` + +```sql SQL +SELECT id, text FROM my_docs LIMIT 1000 +``` + + +`execute_query` returns a blocking Arrow reader **once results start arriving** +-- it does not wait for the whole query to finish. Results stream as record +batches, so a large result set does not have to fit in memory before you see +its first row. + +`execute_query_async` returns the query handle instead, for a statement you +would rather not sit on at all. `describe_query` reports on one by id. + + +SQL requires a deployment. See [SDKs and SQL](/basics/apis). + + +Query-builder searches -- vector, full-text, hybrid -- are ordinary calls that +return results. They are fast enough not to need a handle. See +[Query](/search). + +## Jobs: work that outlives the call {#jobs} + Some operations take longer than a request should. Building an index over a large table, refreshing a computed column, registering a function -- these run on the server after the call that started them has returned. -Each is submitted the same way and returns the same handle. Learn the handle -once and every long-running operation behaves alike. +Each is submitted the same way and returns the same handle. -## The `_async` convention {#convention} +### The `_async` convention {#convention} -An operation that can run in the background has two forms. The plain name -blocks until the work is done and returns its result. The `_async` / -`Async` name submits the work and returns a **job** immediately. +An operation that can run in the background has two forms. The plain name blocks +until the work is done. The `_async` / `Async` name submits the work and returns +a **job** immediately. -| Blocking | Returns a job | What runs | +| Blocking | Returns a job | SQL | | --- | --- | --- | -| `create_index` | `create_index_async` | index build | -| `refresh_column` | `refresh_column_async` | recompute a [computed column](/feature-engineering/computed-columns) | -| `create_function` | `create_function_async` | register a [function](/feature-engineering/functions) | -| `drop_table` | `drop_table_async` | drop a table | -| `execute_query` | `execute_query_async` | a [SQL](/basics/apis) query | +| `create_index` | `create_index_async` | `CREATE INDEX` | +| `refresh_column` | `refresh_column_async` | `REFRESH COLUMN` | +| `create_function` | `create_function_async` | `CREATE FUNCTION` | +| `drop_table` | `drop_table_async` | `DROP TABLE` | The blocking form is not a different operation -- it submits the same job and -waits. Reach for the `_async` form when you want to do something else -meanwhile, run several at once, or survive the client going away. +waits. Reach for the `_async` form when you want to do something else meanwhile, +run several at once, or survive the client going away. -## Submitting and waiting {#waiting} +### Submitting and waiting {#waiting} ```python Python icon="python" @@ -73,15 +142,15 @@ result = table.refresh_column_async("embedding").wait(timeout=timedelta(minutes= still running; the handle just gave up watching. -The TypeScript `wait()` resolves to `void`. Read the outcome from `result` -after it resolves. +The TypeScript `wait()` resolves to `void`. Read the outcome from `result` after +it resolves. -## Inspecting a job {#inspecting} +### Inspecting a job {#inspecting} -Submitting an operation returns only an id, so the detail fields are empty -until the handle fetches the record. `refresh()` does that; `status()` does it -as a side effect. +Submitting an operation returns only an id, so the detail fields are empty until +the handle fetches the record. `refresh()` does that; `status()` does it as a +side effect. ```python Python icon="python" @@ -110,7 +179,7 @@ job.failure; // why it failed, when it did Printing the handle renders every field it knows, which is usually faster than reading them one at a time. -## Events {#events} +### Progress, from events {#events} `result` only tells you anything once the job has finished. Events are written as it runs, and they outlive the workers that produced them -- so a job that is @@ -124,6 +193,10 @@ events = job.events(limit=5000, filter="state = 'claim_complete'") ```typescript TypeScript icon="square-js" const events = await job.events({ limit: 5000, filter: "state = 'claim_complete'" }); ``` + +```sql SQL +SHOW ERRORS ON my_table FOR JOB 'job-id' LIMIT 100 +``` A distributed job records a `claim` / `claim_complete` pair per unit of work, @@ -142,7 +215,7 @@ silently lose its tail unless you pass `limit`. Events are recorded server-side, so this raises for an operation that ran in-process. -## Finding and cancelling jobs {#managing} +### Finding and cancelling jobs {#managing} The connection lists and reopens jobs, so a handle lost to a restarted process is recoverable from its id. @@ -155,9 +228,13 @@ db.cancel_job(job_id) # ask it to stop ``` ```typescript TypeScript icon="square-js" -await db.listJobs(); // every job the server knows about -const job = await db.openJob(jobId); // reattach to one -await db.cancelJob(jobId); // ask it to stop +await db.listJobs(); +const job = await db.openJob(jobId); +await db.cancelJob(jobId); +``` + +```sql SQL +SHOW JOBS ``` @@ -172,10 +249,10 @@ parsing it or storing it to resume the job later is not supported. ## In-process jobs {#in-process} -A local connection has no server to run anything, so its operations run in your -process. They still return a handle, so the same code works against both, but +A local database has no server to run anything, so its operations run in your +process. They still return a handle, so the same code works against both -- but one with nothing behind it: `id` is `None`, `status()` reports `"finished"`, `wait()` returns immediately, `cancel()` does nothing, and `events()` raises. -This is the difference to expect when moving between a local database and a -deployment -- see [Python, TypeScript, and SQL](/basics/apis). +This is the difference to expect when moving between embedded and deployed -- +see [SDKs and SQL](/basics/apis). diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index b38f63f29..448aaf2fc 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -1,6 +1,6 @@ --- -title: "What about Lance?" -sidebarTitle: "What about Lance" +title: "Lance format" +sidebarTitle: "Lance Format" description: "Lance is the open format underneath. LanceDB is what implements a lakehouse on top of it." icon: "/static/assets/logo/lance-logo-gray.svg" keywords: ["lance", "format", "open standard", "lakehouse", "namespace", "catalog"] diff --git a/docs/web/docs.json b/docs/web/docs.json index bbe60eb53..9c9fc3626 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -58,6 +58,10 @@ { "source": "/sql", "destination": "/basics/apis" + }, + { + "source": "/api-reference", + "destination": "/basics/apis" } ], "navigation": { @@ -75,17 +79,25 @@ { "group": "Basics", "pages": [ + "basics/lance", "basics/catalog", - "basics/apis", "basics/jobs", - "basics/agentic-skills", - "basics/lance" + "basics/apis", + { + "group": "REST API", + "expanded": true, + "pages": [ + "api-reference/rest/index" + ] + }, + "basics/agentic-skills" ] }, { "group": "Management", "pages": [ "management/index", + "management/database", { "group": "Namespace", "pages": [ @@ -334,24 +346,6 @@ } ] }, - { - "tab": "API Reference", - "groups": [ - { - "group": "API Reference", - "pages": [ - "api-reference/index", - { - "group": "REST API", - "expanded": true, - "pages": [ - "api-reference/rest/index" - ] - } - ] - } - ] - }, { "tab": "Agentic Skills", "groups": [ diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx new file mode 100644 index 000000000..c1d3e9096 --- /dev/null +++ b/docs/web/management/database.mdx @@ -0,0 +1,194 @@ +--- +title: "Database" +sidebarTitle: "Database" +description: "Connecting to LanceDB: the URI forms, their options, and what each one changes." +icon: database +keywords: ["connect", "database", "uri", "api key", "storage options", "consistency"] +--- + +A database is the root of a [catalog](/basics/catalog). A connection opens +exactly one, and every other object is reached through it. + +Connecting is the one decision that reaches furthest: the URI you pass decides +where queries execute, which features exist, and what a table handle actually +is. + +## The three URI forms {#uris} + + +```python Python icon="python" +import lancedb + +db = lancedb.connect("./.lancedb") # local directory +db = lancedb.connect("s3://bucket/path") # object storage +db = lancedb.connect("db://my-database", api_key="...") # a deployment +``` + +```typescript TypeScript icon="square-js" +import * as lancedb from "@lancedb/lancedb"; + +const db = await lancedb.connect("./.lancedb"); +const db = await lancedb.connect("s3://bucket/path"); +const db = await lancedb.connect("db://my-database", { apiKey: "..." }); +``` + + +The first two are **embedded**: your process opens Lance tables directly from +storage and does the work itself. The third talks to a **deployment**. + +| | Embedded | Deployment | +| --- | --- | --- | +| Where queries run | in your process | on the server | +| Table handle | `LanceTable` | `RemoteTable` | +| [SQL](/basics/apis) | -- | yes | +| Server-side [jobs](/basics/jobs) | -- | yes | +| [Functions](/feature-engineering/functions) | -- | yes | +| [Authorization](/governance/authorization) | storage permissions | granted per object | + +![](/static/assets/images/overview/understanding-tables.png) + +Both handles expose the same table API -- create, open, mutate, query -- so +application code moves between them by changing the URI. What differs is where +resolution and execution happen, and therefore which operations exist at all. + +## Connecting to a deployment {#deployment} + +Enterprise + + +```python Python icon="python" +db = lancedb.connect( + "db://my-database", + api_key="...", + region="us-east-1", +) +``` + +```typescript TypeScript icon="square-js" +const db = await lancedb.connect("db://my-database", { + apiKey: "...", + region: "us-east-1", +}); +``` + + +| Option | Meaning | +| --- | --- | +| `api_key` / `apiKey` | authenticates the connection -- see [Authentication](/enterprise/authentication) | +| `region` | which region hosts the database. Defaults to `us-east-1` | +| `host_override` / `hostOverride` | point at a specific endpoint instead of the regional one | +| `sql_host_override` | the same, for the [SQL endpoint](/search/sql). Python only | +| `client_config` / `clientConfig` | retries, timeouts, and connection pooling | + +An API key belongs in the environment, not in source. `region` is easy to get +wrong silently -- the default is `us-east-1` whether or not that is where your +database lives, and a wrong region fails as a connection error rather than a +helpful one. + +## Storage options {#storage-options} + +`storage_options` configures the underlying object store -- credentials, +endpoints, and defaults for tables created through this connection: + + +```python Python icon="python" +db = lancedb.connect( + "s3://bucket/path", + storage_options={"new_table_enable_stable_row_ids": "true"}, +) +``` + +```typescript TypeScript icon="square-js" +const db = await lancedb.connect("s3://bucket/path", { + storageOptions: { new_table_enable_stable_row_ids: "true" }, +}); +``` + + + +`new_table_enable_stable_row_ids` is worth setting deliberately. Stable row ids +cannot be enabled after a table exists, and a +[materialized view](/feature-engineering/materialized-views) requires its source +to have them. A table created without them cannot later become a view's source +without being recreated. + + +See [Storage](/storage) for the full set, including per-cloud credentials. + +## Read consistency {#consistency} + +`read_consistency_interval` controls how often a connection checks for versions +committed by other writers. + + +```python Python icon="python" +from datetime import timedelta + +db = lancedb.connect(uri, read_consistency_interval=timedelta(seconds=5)) +``` + +```typescript TypeScript icon="square-js" +const db = await lancedb.connect(uri, { readConsistencyInterval: 5 }); +``` + + +The default reads a snapshot taken when the table was opened, so a long-lived +connection **will not see other processes' writes** until it reopens the table. +That is the right default for a batch job and the wrong one for a service. + +Set it when several processes write the same table and readers must observe +those writes. The cost is a metadata check on that interval. See +[Consistency](/tables/consistency). + +## Database-level operations {#operations} + +| | Python | TypeScript | SQL | +| --- | --- | --- | --- | +| list tables | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` | +| the URI it opened | `uri` | `display()` | -- | +| drop every table | `drop_all_tables` | `dropAllTables` | -- | +| drop the database | `drop_database` | -- | -- | +| close | -- | `close`, `isOpen` | -- | + + +```python Python icon="python" +db.table_names() +tbl = db.open_table("documents") +db.drop_table("documents") +``` + +```typescript TypeScript icon="square-js" +await db.tableNames(); +const tbl = await db.openTable("documents"); +await db.dropTable("documents"); +``` + +```sql SQL +SHOW TABLES; +DROP TABLE documents; +``` + + +Each takes a `namespace_path` to act inside a [namespace](/namespaces) rather +than at the root. + + +`drop_database()` removes every table in the database. There is no confirmation +and no undo -- table [versioning](/tables/versioning) does not survive the table +being dropped. + + +## Async connections {#async} + +`connect_async` returns an `AsyncConnection` with the same surface as +coroutines. The TypeScript SDK is async throughout, so its `connect` already is +one. + +```python +db = await lancedb.connect_async("db://my-database", api_key="...") +tbl = await db.open_table("documents") +``` + +This is a different axis from the `_async` operations that return a +[job](/basics/jobs): `connect_async` is about not blocking your event loop, +while `create_index_async` is about work that outlives the call. diff --git a/docs/web/namespaces/usage.mdx b/docs/web/namespaces/usage.mdx index 5c9096cd0..7e69e600c 100644 --- a/docs/web/namespaces/usage.mdx +++ b/docs/web/namespaces/usage.mdx @@ -144,6 +144,6 @@ serves more than one database, LanceDB can also use headers such as `x-lancedb-d ## Related references {#related-references} -- [Client SDK API references](/api-reference) +- [Client SDK API references](/basics/apis#reference) - [REST API Reference](/api-reference/rest) - [Namespaces and the Catalog Model](/namespaces) diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index a60a347e0..7811eafc2 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -382,6 +382,38 @@ Next, you can query a compact view of the new feature: The same workflow is used for data preparation tasks when adding derived features, cached model signals, review scores, or dataset quality indicators. +### Declare now, compute later {#computed-columns} + +The call above computes every row as part of the call, which is fine for three +rows. On a table large enough that you would rather choose *when* the work +happens, declare the column instead and fill it separately: + +```python +# Commits the column, empty. Nothing is computed yet. +characters.add_columns(computed={"power_score": "(stats.magic + stats.strength) / 2.0"}) + +# Fills it, and commits again. +characters.refresh_column("power_score") +# RefreshColumnResult(rows_filled=3, version=4) +``` + +A refresh only gives values to rows that have none, so appending more characters +and refreshing again fills just the new ones -- it does not recompute the rows +already done. That is what makes a backfill safe to retry and cheap to repeat as +data arrives. + +On a deployment the same refresh runs server-side, on a cluster, as a +[job](/basics/jobs) you can walk away from: + +```python +job = characters.refresh_column_async("power_score") +job.wait() +``` + +See [Computed columns](/feature-engineering/computed-columns) for expressions, +and [Functions](/feature-engineering/functions) for filling a column with Python +-- an embedding model, a captioner, anything you can register. + ## 7\. Store multimodal data {#7-store-multimodal-data} Multimodal data is a first-class citizen in LanceDB. Binary data (image, audio, video, etc.) is @@ -424,6 +456,44 @@ These snippets load the local image file and store the bytes in an `image` colum For more examples, see the [multimodal data](/tables/multimodal) section. +## 8\. Train on the result {#8-train} + +The point of the previous steps is this one. The table you searched, filtered, +and added features to is directly usable as a PyTorch +[Dataset](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) +-- there is no export step, and no second copy of the data to keep in sync. + +```python +import torch +from lancedb.util import tbl_to_tensor + +dataloader = torch.utils.data.DataLoader( + characters, batch_size=1024, shuffle=True, collate_fn=tbl_to_tensor +) + +for batch in dataloader: + ... # train +``` + +Because the loader reads from the table, the feature columns you just +materialized are read straight off storage rather than recomputed every epoch. +That is the whole argument for doing feature engineering in the lakehouse: an +expensive transform runs once, is versioned with the data, and every subsequent +epoch is a read. + +Projection matters here. Selecting only the columns a step needs means the +images are never read for an epoch that only wants embeddings: + +```python +from lancedb.permutation import Permutation + +permutation = Permutation.identity(characters) +dataloader = torch.utils.data.DataLoader(permutation) +``` + +See [Training](/training) for shuffling, multi-worker loading, and reading from +a remote table. + ## Code {#code} See the full code for these examples (including helper functions) in the @@ -432,13 +502,13 @@ See the full code for these examples (including helper functions) in the ## Next steps {#next-steps} -You've learned how to install LanceDB, connect, create one table for AI data, retrieve related -examples, curate with metadata, add a derived feature, and represent multimodal records. These same -primitives apply across the AI data lifecycle, from data preparation and feature engineering to -retrieval, evaluation, and training. +You've gone from a table, through search and curation, to derived features and a +training loop -- in one table, with no export step between any two stages. That +arc is the argument for a multimodal lakehouse, and every stage of it scales +past this toy example without changing shape. -Continue to the table and search guides to build on this example with schema options, appends, -updates, versioning, indexing, full-text search, hybrid search, and reranking. +Continue to the guides to build on it with schema options, versioning, indexing, +full-text and hybrid search, reranking, and server-side feature engineering.