chore: merge main to pick up Tokio advisory fix

2026-10-25 21:22:29 +00:00 · 2026-10-23 21:48:31 +02:00
parent 3d4f93e431 e74a7c15b5
commit 0ba5e9637f
135 changed files with 14115 additions and 1011 deletions
@@ -653,94 +653,48 @@ See [Listing](Listing.md) for pagination state and error behavior.
 ### listNamespaces()

 ```ts
-abstract listNamespaces(namespacePath?, options?): Promise<ListNamespacesResponse>
+abstract listNamespaces(namespacePath?, options?): Listing<string>
 ```

-List the immediate child namespaces under the given parent.
-
-Results may be paginated. To retrieve subsequent pages, pass the
-`pageToken` returned by a previous call.
+Lazily list immediate child namespaces, 100 per page by default.

 #### Parameters

 * **namespacePath?**: `string`[]
-    The parent namespace path. Defaults
-    to the root namespace if omitted.

-* **options?**: `Partial`&lt;[`ListNamespacesOptions`](../interfaces/ListNamespacesOptions.md)&gt;
-    Pagination options
-    (`pageToken`, `limit`).
+* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)

 #### Returns

-`Promise`&lt;[`ListNamespacesResponse`](../interfaces/ListNamespacesResponse.md)&gt;
-
-Child namespace names and
-  an optional token for fetching the next page.
+[`Listing`](Listing.md)&lt;`string`&gt;

 ***

 ### listTables()

-#### listTables(options)
-
 ```ts
-abstract listTables(options?): Promise<ListTablesResponse>
+abstract listTables(namespacePath?, options?): Listing<string>
 ```

-List a page of the tables in this database.
+Lazily list tables in a namespace, requesting 100 per page by default.

-To retrieve the tables after the page, pass the `pageToken` the response
-carries back in. A page can be shorter than `limit` without being the last
-one, so walk until a response carries no page token:
-
-```ts
-const names = [];
-let pageToken = undefined;
-do {
-  const page = await conn.listTables({ pageToken, limit: 100 });
-  names.push(...page.tables);
-  pageToken = page.pageToken;
-} while (pageToken);
-```
-
-##### Parameters
-
-* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
-    Pagination options
-    (`pageToken`, `limit`).
-
-##### Returns
-
-`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
-
-A page of table names and an
-  optional token for the tables after it.
-
-#### listTables(namespacePath, options)
-
-```ts
-abstract listTables(namespacePath?, options?): Promise<ListTablesResponse>
-```
-
-List a page of the tables in this database.
-
-##### Parameters
+#### Parameters

 * **namespacePath?**: `string`[]
-    The namespace path to list tables from
-    (defaults to root namespace)

-* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
-    Pagination options
-    (`pageToken`, `limit`).
+* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)

-##### Returns
+#### Returns

-`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
+[`Listing`](Listing.md)&lt;`string`&gt;

-A page of table names and an
-  optional token for the tables after it.
+#### Example
+
+```ts
+for await (const name of conn.listTables([], { pageLimit: 20 })) {
+  console.log(name);
+}
+```

 ***

@@ -1298,7 +1298,11 @@ Specify exactly one of `column` or `indexName`.

 Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
 the client process from index metadata. For remote tables, this means the
-same tokenizer model files must also exist locally.
+same tokenizer model files must also exist locally (under
+`LANCE_LANGUAGE_MODEL_HOME`) and must match the files the server uses. A
+mismatch cannot be detected: the returned tokens may differ from what the
+server's index produces. If the model files are missing, an error naming
+the tokenizer is raised.

 #### Parameters

@@ -108,10 +108,6 @@
 - [JobEventsOptions](interfaces/JobEventsOptions.md)
 - [JobFailureInfo](interfaces/JobFailureInfo.md)
 - [JobInfo](interfaces/JobInfo.md)
- [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
- [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
- [ListTablesOptions](interfaces/ListTablesOptions.md)
- [ListTablesResponse](interfaces/ListTablesResponse.md)
 - [ListingOptions](interfaces/ListingOptions.md)
 - [MaterializedViewDefinition](interfaces/MaterializedViewDefinition.md)
 - [MemtableStats](interfaces/MemtableStats.md)
@@ -65,7 +65,7 @@ then no error will be raised.
 ### mode

 ```ts
-mode: "overwrite" | "create";
+mode: "overwrite" | "create" | "exist_ok";
 ```

 The mode to use when creating the table.
@@ -76,6 +76,9 @@ happen.  Any provided data will be ignored.

 If this is set to "overwrite" then any existing table will be replaced.

+If this is set to "exist_ok" then an existing table is opened and any
+provided data is ignored.
+
 ***

 ### schema?
@@ -1,27 +0,0 @@
-[**@lancedb/lancedb**](../README.md) • **Docs**
-
-***
-
-[@lancedb/lancedb](../globals.md) / ListNamespacesOptions
-
-# Interface: ListNamespacesOptions
-
-## Properties
-
-### limit?
-
-```ts
-optional limit: number;
-```
-
-An optional limit to the number of results to return.
-
-***
-
-### pageToken?
-
-```ts
-optional pageToken: string;
-```
-
-Token from a previous response for pagination.
@@ -1,23 +0,0 @@
-[**@lancedb/lancedb**](../README.md) • **Docs**
-
-***
-
-[@lancedb/lancedb](../globals.md) / ListNamespacesResponse
-
-# Interface: ListNamespacesResponse
-
-## Properties
-
-### namespaces
-
-```ts
-namespaces: string[];
-```
-
-***
-
-### pageToken?
-
-```ts
-optional pageToken: string;
-```
@@ -1,35 +0,0 @@
-[**@lancedb/lancedb**](../README.md) • **Docs**
-
-***
-
-[@lancedb/lancedb](../globals.md) / ListTablesOptions
-
-# Interface: ListTablesOptions
-
-## Properties
-
-### limit?
-
-```ts
-optional limit: number;
-```
-
-An upper bound on how many tables to return.
-Defaults to 100. Zero returns an empty page without a continuation token.
-
-A page may hold fewer than this and still not be the last one, so keep
-going while the response carries a page token rather than while pages are
-full.
-
-***
-
-### pageToken?
-
-```ts
-optional pageToken: string;
-```
-
-Token from a previous response, to resume listing where it left off.
-
-The token is opaque: it carries whatever the database needs to resume, and
-callers should not construct or interpret one.
@@ -1,23 +0,0 @@
-[**@lancedb/lancedb**](../README.md) • **Docs**
-
-***
-
-[@lancedb/lancedb](../globals.md) / ListTablesResponse
-
-# Interface: ListTablesResponse
-
-## Properties
-
-### pageToken?
-
-```ts
-optional pageToken: string;
-```
-
-***
-
-### tables
-
-```ts
-tables: string[];
-```
@@ -20,12 +20,14 @@ How to return blob v2 columns. Applies to `toArrow()` and `toArray()`.

 - `"descriptions"` (default): each blob is a descriptor. Read the bytes
  with [Table.fetchBlobs](../classes/Table.md#fetchblobs) or [Table.fetchBlobFiles](../classes/Table.md#fetchblobfiles).
- `"bytes"`: each top-level blob column holds the blob bytes as
-  `LargeBinary`. The query also reads `_rowid` to fetch the bytes, and
+- `"bytes"`: each blob column, including blobs nested in structs, holds
+  the bytes as `LargeBinary`. The query also reads `_rowid` to fetch them, and
  leaves it out of the result unless [QueryBase.withRowId](../classes/QueryBase.md#withrowid) was
  called. All the bytes are held in memory.

-Blobs nested in a struct or a list keep their descriptors.
+Selecting a parent struct or directly aliasing a blob or struct also returns
+bytes. Blobs inside lists keep their descriptors. Arbitrary projection
+expressions are not materialized as bytes.

 The rows and the bytes come from the same table version. If the table
 changes while the query runs, the query runs again, up to three times.
@@ -8,7 +8,7 @@

 ## Deprecated

-Use [ListTablesOptions](ListTablesOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
+Use [ListingOptions](ListingOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
 instead.

 ## Properties
@@ -26,14 +26,17 @@ is also an [asynchronous API client](#connections-asynchronous).

 ::: lancedb.db.DBConnection

-Resource listings (`list_secrets`, `list_views`, `list_jobs`, `list_functions`, and
-`list_materialized_views`) return lazy iterators. Use `list(db.list_views())` to
-collect synchronous results, or `[name async for name in db.list_views()]` with
-an asynchronous connection. `page_limit` controls each request, and `page_token`
-resumes a listing. Drain the iterator's cached results before saving its token.
+Resource listings return iterators. Use `list(db.list_tables())` to collect all
+table names. `db.list_tables().tables` is a list of the first page; `limit`
+controls its page size and `page_token` resumes at a saved token. The same
+patterns apply to `list_namespaces()`. Other resource listings use `page_limit`.

 ::: lancedb.listing.Listing

+::: lancedb.listing.TableListing
+
+::: lancedb.listing.NamespaceListing
+
 ::: lancedb.Session

 ## Catalogs (Synchronous)
@@ -631,6 +634,16 @@ can be used to create, list, or open tables.

 ::: lancedb.listing.AsyncListing

+`async for name in db.list_tables()` loads pages as needed. To inspect the
+first page, use `page = await db.list_tables()` and then `page.tables`. Reading
+the page list does not consume the iterator. `page.page_token` is the token for
+the next request; drain cached results before using it to resume in a new
+listing. The same behavior applies to namespaces.
+
+::: lancedb.listing.AsyncTableListing
+
+::: lancedb.listing.AsyncNamespaceListing
+
 ## Namespaces (Asynchronous)

 ::: lancedb.connect_namespace_async
@@ -4,6 +4,12 @@ import { afterEach, describe, expect, jest, test } from "@jest/globals";
 import * as lancedb from "@lancedb/lancedb";
 import { withTempDirectory } from "./util.ts";

+async function collectNames(items: AsyncIterable<string>): Promise<string[]> {
+  const names: string[] = [];
+  for await (const name of items) names.push(name);
+  return names;
+}
+
 test("connect to a local database", async () => {
  await withTempDirectory(async (tempDir) => {
    const cwd = process.cwd();
@@ -13,12 +19,12 @@ test("connect to a local database", async () => {
      const db = await lancedb.connect("ex_lancedb");
      // --8<-- [end:connect]
      await db.createTable("probe", [{ id: 1 }]);
-      expect((await db.listTables()).tables).toEqual(["probe"]);
+      expect(await collectNames(db.listTables())).toEqual(["probe"]);
    } finally {
      process.chdir(cwd);
    }
    const reopened = await lancedb.connect(`${tempDir}/ex_lancedb`);
-    expect((await reopened.listTables()).tables).toEqual(["probe"]);
+    expect(await collectNames(reopened.listTables())).toEqual(["probe"]);
  });
 });

@@ -109,8 +115,8 @@ async function namespaceTableOpsExample() {
  );

  // Verify
-  console.log((await db.listNamespaces()).namespaces); // ["prod"]
-  console.log((await db.listNamespaces(["prod"])).namespaces); // ["recommendations", "search"]
+  console.log(await collectNames(db.listNamespaces())); // ["prod"]
+  console.log(await collectNames(db.listNamespaces(["prod"]))); // ["recommendations", "search"]
  console.log(await db.tableNames(["prod", "search"])); // ["user"]
  console.log(await db.tableNames(["prod", "recommendations"])); // ["user"]
  // --8<-- [end:namespace_table_ops]
@@ -124,7 +130,7 @@ async function namespaceAdminOpsExample() {
  await db.createNamespace(["prod"]);
  await db.createNamespace(["prod", "search"]);

-  const childNamespaces = (await db.listNamespaces(["prod"])).namespaces;
+  const childNamespaces = await collectNames(db.listNamespaces(["prod"]));
  console.log(`Child namespaces under ${JSON.stringify(namespace)}:`, childNamespaces);
  // Child namespaces under ["prod","search"]: [ 'search' ]

@@ -20,12 +20,7 @@ const connections =

 async function tableNames(db: lancedb.Connection) {
  const names: string[] = [];
-  let pageToken: string | undefined;
-  do {
-    const page = await db.listTables({ pageToken });
-    names.push(...page.tables);
-    pageToken = page.pageToken;
-  } while (pageToken);
+  for await (const name of db.listTables()) names.push(name);
  return names;
 }

@@ -350,6 +350,7 @@
            "pages": [
              "enterprise/index",
              "enterprise/architecture",
+              "enterprise/distributed-search",
              "enterprise/deployment/index",
              "deployment/storage",
              "deployment/monitoring",
@@ -18,3 +18,6 @@ run on machines that can be replaced without any of them owning the data.

 [Architecture](/architecture) covers this alongside the embedded form, which is
 the comparison most readers want first.
+
+See [Distributed search](/enterprise/distributed-search) for how Enterprise
+distributes index execution and table-data reads.
@@ -0,0 +1,33 @@
+---
+title: "Distributed search"
+sidebarTitle: "Distributed search"
+description: "How LanceDB Enterprise distributes index execution and table-data reads."
+icon: "network-wired"
+availability:
+  oss: unavailable
+  enterprise: available
+  summary: >-
+    LanceDB Enterprise distributes index execution and table-data reads across
+    plan executors. Embedded LanceDB executes searches in the application process.
+---
+
+Distributed search spreads index execution, table-data reads, and caching across
+multiple serving machines in LanceDB Enterprise. Query nodes plan queries and
+combine results. Plan executors cache index structures and table data and execute
+the remote work. See [Enterprise architecture](/enterprise/architecture).
+
+## Embedded search {#embedded-search}
+
+LanceDB OSS runs inside your application process. It searches indexes and reads
+table data using that process's resources, including its index cache. Reading a
+table from object storage does not by itself distribute query execution across
+machines.
+
+## Search interfaces {#search-interfaces}
+
+The shared search guides describe the query APIs and index settings:
+
+- [Vector search](/search/vector-search) and [vector indexes](/indexing/vector-index).
+- [Full-text search](/search/full-text-search) and [FTS indexes](/indexing/fts-index).
+- [Filtering](/search/filtering) and [scalar indexes](/indexing/scalar-index).
+- [Query plans](/search/explain) for inspecting execution and runtime measurements.
@@ -384,7 +384,7 @@ The inner workings of RaBitQ quantization are rather mathematically dense. It ge
 applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube, 
 where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497)

-##### Using RaBitQ
+##### Using RaBitQ {#using-rabitq}
 Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`. 
 Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`.

@@ -10,6 +10,9 @@ Generate embeddings via the [ollama](https://github.com/ollama/ollama-python) py
 - [Ollama docs on embeddings](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings)
 - [Ollama blog on embeddings](https://ollama.com/blog/embedding-models)

+For direct calls, `compute_source_embeddings` accepts a list of strings or a
+one-dimensional NumPy string array. Both inputs are sent to Ollama in one batch.
+
 | Parameter              | Type                       | Default Value            | Description                                                                                                                                    |
 |------------------------|----------------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
 | `name`                 | `str`                      | `nomic-embed-text`       | The name of the model.                                                                                                                         |
@@ -45,6 +45,8 @@ Supported parameters (to be passed in `create` method) are:
 | `output_dimension` | `int` | `None` | Output embedding dimension. Only supported by `voyage-multimodal-3.5`. Valid options: 256, 512, 1024 (default), 2048. |


+For text models, `compute_source_embeddings()` accepts a single string, a list of strings, a one-dimensional NumPy array, or a PyArrow `Array` or `ChunkedArray`. Batch inputs retain their order.
+
 Usage Example:

 <CodeGroup>
@@ -55,10 +57,15 @@ Usage Example:

 ### Multimodal Example {#multimodal-example}

-The `voyage-multimodal-3.5` model can embed text alongside images. You can use image URLs, file paths, or PIL Image objects:
+The `voyage-multimodal-3.5` model can embed text alongside images. You can use image URLs, local `pathlib.Path` objects, raw image bytes, or PIL Image objects.
+
+For source embeddings, PIL images and images loaded from bytes or local paths are
+converted to RGB and encoded as JPEG. This supports RGBA, grayscale with alpha,
+and palette images; transparency is discarded. Composite transparent images onto
+your chosen background before embedding if the background color matters.

 <CodeGroup>
  <CodeBlock filename="Python" language="Python" icon="python">
    {PyEmbeddingVoyageaiMultimodal}
  </CodeBlock>
-</CodeGroup>
+</CodeGroup>
@@ -58,6 +58,33 @@ uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
 ```
 </CodeGroup>

+If you also need the optional `lance` Python API, install the `pylance` extra and
+include the Lance preview index:
+
+<CodeGroup>
+```bash Python (pip) icon="terminal"
+pip install --pre \
+  --extra-index-url https://pypi.fury.io/lancedb/ \
+  --extra-index-url https://pypi.fury.io/lance-format/ \
+  "lancedb[pylance]"
+```
+
+```bash Python (uv) icon="terminal"
+uv pip install --prerelease allow \
+  --index https://pypi.fury.io/lancedb/ \
+  --index https://pypi.fury.io/lance-format/ \
+  --index-strategy unsafe-best-match \
+  "lancedb[pylance]"
+```
+</CodeGroup>
+
+The LanceDB preview index contains old `pylance` releases; current Lance previews
+are published to the `lance-format` index. By default, uv only considers versions
+from the first index containing a package, which can select an old `pylance` or
+backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
+indexes, like pip; use it only with indexes you trust. Install the extra rather
+than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
+
 <Note>
 Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
 for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
@@ -235,7 +235,7 @@ The table below summarizes the behavior of `_distance` in search results based o
 See [vector indexes](/indexing/vector-index) for index configuration. The
 following sections cover query-time settings.

-### Tuning `approx_mode`
+### Tuning `approx_mode` {#tuning-approx-mode}

 Use `approx_mode` when you want to adjust the speed/recall trade-off for approximate vector search at query time. This setting currently applies only to RQ-quantized indexes, such as `IVF_RQ`; other index types ignore it.