mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-27 08:28:28 +00:00
Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 011def461c | |||
| ed6be12ad6 | |||
| ac2b689cdb | |||
| 4fc8114871 |
@@ -92,6 +92,8 @@ Python bindings changes:
|
|||||||
* Should use `LOOP.run()` to call the corresponding `AsyncTable` method.
|
* Should use `LOOP.run()` to call the corresponding `AsyncTable` method.
|
||||||
6. Add concrete sync method to `RemoteTable` class in `python/python/lancedb/remote/table.py`.
|
6. Add concrete sync method to `RemoteTable` class in `python/python/lancedb/remote/table.py`.
|
||||||
7. Add unit test in `python/tests/test_table.py`.
|
7. Add unit test in `python/tests/test_table.py`.
|
||||||
|
8. If you added a new public class or module-level function (not just a method on an
|
||||||
|
existing class), expose it in the API reference. See "Python API reference" below.
|
||||||
|
|
||||||
TypeScript bindings changes:
|
TypeScript bindings changes:
|
||||||
|
|
||||||
@@ -103,6 +105,33 @@ TypeScript bindings changes:
|
|||||||
5. Add test in `nodejs/__test__/table.test.ts`.
|
5. Add test in `nodejs/__test__/table.test.ts`.
|
||||||
6. Run `npm run docs` to generate TypeScript documentation.
|
6. Run `npm run docs` to generate TypeScript documentation.
|
||||||
|
|
||||||
|
## Python API reference
|
||||||
|
|
||||||
|
`docs/src/python/python.md` is the entire Python API reference. It is maintained by
|
||||||
|
hand, and anything not listed there is not rendered at all, so new public classes and
|
||||||
|
module-level functions have to be added explicitly. How depends on the module:
|
||||||
|
|
||||||
|
* `lancedb.index`, `lancedb.embeddings`, `lancedb.remote`, and `lancedb.rerankers` are
|
||||||
|
rendered by a single directive each, driven by the module's `__all__`. Add the new
|
||||||
|
name to `__all__` and it appears; forget, and it is silently omitted.
|
||||||
|
* Everything else (`lancedb`, `lancedb.table`, `lancedb.query`, `lancedb.db`, ...) is
|
||||||
|
listed symbol by symbol. Add a `::: lancedb.<module>.<Name>` line to the matching
|
||||||
|
section, and remember that the page separates synchronous and asynchronous APIs.
|
||||||
|
|
||||||
|
Deliberately undocumented: concrete implementations reached through an abstract base
|
||||||
|
(`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already
|
||||||
|
covered by `inherited_members`, and internal helpers.
|
||||||
|
|
||||||
|
Cross-references in docstrings use mkdocstrings syntax, `[text][lancedb.table.Table]`.
|
||||||
|
Plain relative links such as `[Table](Table)` do not resolve. To check your work:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
pip install -r docs/requirements.txt
|
||||||
|
cd docs && PYTHONPATH=. mkdocs build
|
||||||
|
```
|
||||||
|
|
||||||
|
The docs site only builds on pushes to `main`, so this is not covered by PR CI.
|
||||||
|
|
||||||
## Review Guidelines
|
## Review Guidelines
|
||||||
|
|
||||||
Please consider the following when reviewing code contributions.
|
Please consider the following when reviewing code contributions.
|
||||||
|
|||||||
@@ -51,6 +51,11 @@ plugins:
|
|||||||
paths: [../python/python]
|
paths: [../python/python]
|
||||||
options:
|
options:
|
||||||
docstring_style: numpy
|
docstring_style: numpy
|
||||||
|
docstring_options:
|
||||||
|
# Attributes documented in a `Parameters` section, and pydantic
|
||||||
|
# dataclasses whose `__init__` griffe cannot see statically, both
|
||||||
|
# trip this check. It reports nothing actionable here.
|
||||||
|
warn_unknown_params: false
|
||||||
heading_level: 3
|
heading_level: 3
|
||||||
show_signature_annotations: true
|
show_signature_annotations: true
|
||||||
show_root_heading: true
|
show_root_heading: true
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# Contributing to LanceDB Typescript
|
# Contributing to LanceDB Typescript
|
||||||
|
|
||||||
This document outlines the process for contributing to LanceDB Typescript.
|
This document outlines the process for contributing to LanceDB Typescript.
|
||||||
For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md).
|
For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md).
|
||||||
|
|
||||||
## Project layout
|
## Project layout
|
||||||
|
|
||||||
|
|||||||
+114
-49
@@ -26,6 +26,18 @@ is also an [asynchronous API client](#connections-asynchronous).
|
|||||||
|
|
||||||
::: lancedb.db.DBConnection
|
::: lancedb.db.DBConnection
|
||||||
|
|
||||||
|
::: lancedb.Session
|
||||||
|
|
||||||
|
## Namespaces (Synchronous)
|
||||||
|
|
||||||
|
A namespace-backed connection resolves tables through a
|
||||||
|
[Lance namespace](https://lancedb.github.io/lance-namespace/) service instead of
|
||||||
|
listing a storage directory.
|
||||||
|
|
||||||
|
::: lancedb.connect_namespace
|
||||||
|
|
||||||
|
::: lancedb.namespace.LanceNamespaceDBConnection
|
||||||
|
|
||||||
## Tables (Synchronous)
|
## Tables (Synchronous)
|
||||||
|
|
||||||
::: lancedb.table.Table
|
::: lancedb.table.Table
|
||||||
@@ -34,8 +46,12 @@ is also an [asynchronous API client](#connections-asynchronous).
|
|||||||
|
|
||||||
::: lancedb.table.FragmentSummaryStats
|
::: lancedb.table.FragmentSummaryStats
|
||||||
|
|
||||||
|
::: lancedb.table.TableStatistics
|
||||||
|
|
||||||
::: lancedb.table.Tags
|
::: lancedb.table.Tags
|
||||||
|
|
||||||
|
::: lancedb.table.Branches
|
||||||
|
|
||||||
## Expressions
|
## Expressions
|
||||||
|
|
||||||
Type-safe expression builder for filters and projections. Use these instead
|
Type-safe expression builder for filters and projections. Use these instead
|
||||||
@@ -62,29 +78,46 @@ of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and
|
|||||||
|
|
||||||
::: lancedb.query.LanceHybridQueryBuilder
|
::: lancedb.query.LanceHybridQueryBuilder
|
||||||
|
|
||||||
|
::: lancedb.query.LanceEmptyQueryBuilder
|
||||||
|
|
||||||
|
::: lancedb.query.LanceTakeQueryBuilder
|
||||||
|
|
||||||
|
## Full text queries
|
||||||
|
|
||||||
|
Structured full text queries can be passed to
|
||||||
|
[Table.search][lancedb.table.Table.search] or
|
||||||
|
[AsyncTable.search][lancedb.table.AsyncTable.search] in place of a query string,
|
||||||
|
and combined with [BooleanQuery][lancedb.query.BooleanQuery].
|
||||||
|
|
||||||
|
::: lancedb.query.FullTextQuery
|
||||||
|
|
||||||
|
::: lancedb.query.MatchQuery
|
||||||
|
|
||||||
|
::: lancedb.query.PhraseQuery
|
||||||
|
|
||||||
|
::: lancedb.query.BoostQuery
|
||||||
|
|
||||||
|
::: lancedb.query.MultiMatchQuery
|
||||||
|
|
||||||
|
::: lancedb.query.BooleanQuery
|
||||||
|
|
||||||
|
::: lancedb.query.FullTextOperator
|
||||||
|
|
||||||
|
::: lancedb.query.Occur
|
||||||
|
|
||||||
## Embeddings
|
## Embeddings
|
||||||
|
|
||||||
::: lancedb.embeddings.registry.EmbeddingFunctionRegistry
|
::: lancedb.embeddings
|
||||||
|
options:
|
||||||
::: lancedb.embeddings.base.EmbeddingFunctionConfig
|
show_root_heading: false
|
||||||
|
show_root_toc_entry: false
|
||||||
::: lancedb.embeddings.base.EmbeddingFunction
|
|
||||||
|
|
||||||
::: lancedb.embeddings.base.TextEmbeddingFunction
|
|
||||||
|
|
||||||
::: lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings
|
|
||||||
|
|
||||||
::: lancedb.embeddings.openai.OpenAIEmbeddings
|
|
||||||
|
|
||||||
::: lancedb.embeddings.open_clip.OpenClipEmbeddings
|
|
||||||
|
|
||||||
## Remote configuration
|
## Remote configuration
|
||||||
|
|
||||||
::: lancedb.remote.ClientConfig
|
::: lancedb.remote
|
||||||
|
options:
|
||||||
::: lancedb.remote.TimeoutConfig
|
show_root_heading: false
|
||||||
|
show_root_toc_entry: false
|
||||||
::: lancedb.remote.RetryConfig
|
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -122,7 +155,22 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
|
|||||||
custom_stop_words=["acme"]))
|
custom_stop_words=["acme"]))
|
||||||
```
|
```
|
||||||
|
|
||||||
::: lancedb.index.FTS
|
::: lancedb.tokenize
|
||||||
|
|
||||||
|
::: lancedb.FtsToken
|
||||||
|
|
||||||
|
## Blobs
|
||||||
|
|
||||||
|
Blob columns store large binary values out of line so they can be read lazily
|
||||||
|
instead of being materialized with the rest of the row.
|
||||||
|
|
||||||
|
::: lancedb.blob
|
||||||
|
|
||||||
|
::: lancedb.BlobType
|
||||||
|
|
||||||
|
::: lancedb._blob.BlobFile
|
||||||
|
options:
|
||||||
|
show_root_full_path: false
|
||||||
|
|
||||||
## Utilities
|
## Utilities
|
||||||
|
|
||||||
@@ -130,6 +178,14 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
|
|||||||
|
|
||||||
::: lancedb.merge.LanceMergeInsertBuilder
|
::: lancedb.merge.LanceMergeInsertBuilder
|
||||||
|
|
||||||
|
::: lancedb.otel.instrument_lancedb_metrics
|
||||||
|
|
||||||
|
## Exceptions
|
||||||
|
|
||||||
|
::: lancedb.exceptions.MissingValueError
|
||||||
|
|
||||||
|
::: lancedb.exceptions.MissingColumnError
|
||||||
|
|
||||||
## Integrations
|
## Integrations
|
||||||
|
|
||||||
## Pydantic
|
## Pydantic
|
||||||
@@ -138,19 +194,30 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
|
|||||||
|
|
||||||
::: lancedb.pydantic.vector
|
::: lancedb.pydantic.vector
|
||||||
|
|
||||||
|
::: lancedb.pydantic.Vector
|
||||||
|
|
||||||
|
::: lancedb.pydantic.MultiVector
|
||||||
|
|
||||||
::: lancedb.pydantic.LanceModel
|
::: lancedb.pydantic.LanceModel
|
||||||
|
|
||||||
|
## PyTorch
|
||||||
|
|
||||||
|
::: lancedb.streaming.StreamingDataset
|
||||||
|
|
||||||
|
::: lancedb.permutation.permutation_builder
|
||||||
|
|
||||||
|
::: lancedb.permutation.PermutationBuilder
|
||||||
|
|
||||||
|
::: lancedb.permutation.Permutation
|
||||||
|
|
||||||
|
::: lancedb.permutation.Transforms
|
||||||
|
|
||||||
## Reranking
|
## Reranking
|
||||||
|
|
||||||
::: lancedb.rerankers.linear_combination.LinearCombinationReranker
|
::: lancedb.rerankers
|
||||||
|
options:
|
||||||
::: lancedb.rerankers.cohere.CohereReranker
|
show_root_heading: false
|
||||||
|
show_root_toc_entry: false
|
||||||
::: lancedb.rerankers.colbert.ColbertReranker
|
|
||||||
|
|
||||||
::: lancedb.rerankers.cross_encoder.CrossEncoderReranker
|
|
||||||
|
|
||||||
::: lancedb.rerankers.openai.OpenaiReranker
|
|
||||||
|
|
||||||
## Connections (Asynchronous)
|
## Connections (Asynchronous)
|
||||||
|
|
||||||
@@ -161,6 +228,12 @@ can be used to create, list, or open tables.
|
|||||||
|
|
||||||
::: lancedb.db.AsyncConnection
|
::: lancedb.db.AsyncConnection
|
||||||
|
|
||||||
|
## Namespaces (Asynchronous)
|
||||||
|
|
||||||
|
::: lancedb.connect_namespace_async
|
||||||
|
|
||||||
|
::: lancedb.namespace.AsyncLanceNamespaceDBConnection
|
||||||
|
|
||||||
## Tables (Asynchronous)
|
## Tables (Asynchronous)
|
||||||
|
|
||||||
Table hold your actual data as a collection of records / rows.
|
Table hold your actual data as a collection of records / rows.
|
||||||
@@ -169,32 +242,20 @@ Table hold your actual data as a collection of records / rows.
|
|||||||
|
|
||||||
::: lancedb.table.AsyncTags
|
::: lancedb.table.AsyncTags
|
||||||
|
|
||||||
|
::: lancedb.table.AsyncBranches
|
||||||
|
|
||||||
## Indices (Asynchronous)
|
## Indices (Asynchronous)
|
||||||
|
|
||||||
Indices can be created on a table to speed up queries. This section
|
Indices can be created on a table to speed up queries. This section
|
||||||
lists the indices that LanceDb supports.
|
lists the indices that LanceDb supports.
|
||||||
|
|
||||||
::: lancedb.index.BTree
|
::: lancedb.index
|
||||||
|
options:
|
||||||
::: lancedb.index.Bitmap
|
show_root_heading: false
|
||||||
|
show_root_toc_entry: false
|
||||||
::: lancedb.index.LabelList
|
# `lang_mapping` is defined in the module rather than imported, so it is
|
||||||
|
# picked up despite not being in `__all__`. It is an internal lookup table.
|
||||||
::: lancedb.index.FTS
|
filters: ["!^_", "!^lang_mapping$"]
|
||||||
|
|
||||||
::: lancedb.index.IvfPq
|
|
||||||
|
|
||||||
::: lancedb.index.HnswPq
|
|
||||||
|
|
||||||
::: lancedb.index.HnswSq
|
|
||||||
|
|
||||||
::: lancedb.index.IvfFlat
|
|
||||||
|
|
||||||
::: lancedb.index.IvfSq
|
|
||||||
|
|
||||||
::: lancedb.index.IvfRq
|
|
||||||
|
|
||||||
::: lancedb.index.HnswFlat
|
|
||||||
|
|
||||||
::: lancedb.table.IndexStatistics
|
::: lancedb.table.IndexStatistics
|
||||||
|
|
||||||
@@ -222,3 +283,7 @@ rows nearest to a query vector and can be created with the
|
|||||||
::: lancedb.query.AsyncHybridQuery
|
::: lancedb.query.AsyncHybridQuery
|
||||||
options:
|
options:
|
||||||
inherited_members: true
|
inherited_members: true
|
||||||
|
|
||||||
|
::: lancedb.query.AsyncTakeQuery
|
||||||
|
options:
|
||||||
|
inherited_members: true
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# Contributing to LanceDB Typescript
|
# Contributing to LanceDB Typescript
|
||||||
|
|
||||||
This document outlines the process for contributing to LanceDB Typescript.
|
This document outlines the process for contributing to LanceDB Typescript.
|
||||||
For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md).
|
For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md).
|
||||||
|
|
||||||
## Project layout
|
## Project layout
|
||||||
|
|
||||||
|
|||||||
@@ -359,7 +359,7 @@ class DBConnection(EnforceOverrides):
|
|||||||
|
|
||||||
Data is converted to Arrow before being written to disk. For maximum
|
Data is converted to Arrow before being written to disk. For maximum
|
||||||
control over how data is saved, either provide the PyArrow schema to
|
control over how data is saved, either provide the PyArrow schema to
|
||||||
convert to or else provide a [PyArrow Table](pyarrow.Table) directly.
|
convert to or else provide a [PyArrow Table][pyarrow.Table] directly.
|
||||||
|
|
||||||
>>> import pyarrow as pa
|
>>> import pyarrow as pa
|
||||||
>>> custom_schema = pa.schema([
|
>>> custom_schema = pa.schema([
|
||||||
@@ -1529,7 +1529,7 @@ class AsyncConnection(object):
|
|||||||
|
|
||||||
Data is converted to Arrow before being written to disk. For maximum
|
Data is converted to Arrow before being written to disk. For maximum
|
||||||
control over how data is saved, either provide the PyArrow schema to
|
control over how data is saved, either provide the PyArrow schema to
|
||||||
convert to or else provide a [PyArrow Table](pyarrow.Table) directly.
|
convert to or else provide a [PyArrow Table][pyarrow.Table] directly.
|
||||||
|
|
||||||
>>> import pyarrow as pa
|
>>> import pyarrow as pa
|
||||||
>>> custom_schema = pa.schema([
|
>>> custom_schema = pa.schema([
|
||||||
|
|||||||
@@ -21,3 +21,32 @@ from .watsonx import WatsonxEmbeddings
|
|||||||
from .voyageai import VoyageAIEmbeddingFunction
|
from .voyageai import VoyageAIEmbeddingFunction
|
||||||
from .colpali import ColPaliEmbeddings
|
from .colpali import ColPaliEmbeddings
|
||||||
from .siglip import SigLipEmbeddings
|
from .siglip import SigLipEmbeddings
|
||||||
|
|
||||||
|
# The API reference renders this package with a single mkdocstrings directive,
|
||||||
|
# which only picks up names listed here. New embedding functions must be added
|
||||||
|
# to both the imports above and this list, or they will silently go undocumented.
|
||||||
|
__all__ = [
|
||||||
|
"EmbeddingFunction",
|
||||||
|
"EmbeddingFunctionConfig",
|
||||||
|
"TextEmbeddingFunction",
|
||||||
|
"EmbeddingFunctionRegistry",
|
||||||
|
"get_registry",
|
||||||
|
"register",
|
||||||
|
"SentenceTransformerEmbeddings",
|
||||||
|
"OpenAIEmbeddings",
|
||||||
|
"OpenClipEmbeddings",
|
||||||
|
"BedRockText",
|
||||||
|
"CohereEmbeddingFunction",
|
||||||
|
"GeminiText",
|
||||||
|
"GteEmbeddings",
|
||||||
|
"InstructorEmbeddingFunction",
|
||||||
|
"JinaEmbeddings",
|
||||||
|
"OllamaEmbeddings",
|
||||||
|
"TransformersEmbeddingFunction",
|
||||||
|
"ColbertEmbeddings",
|
||||||
|
"VoyageAIEmbeddingFunction",
|
||||||
|
"WatsonxEmbeddings",
|
||||||
|
"ColPaliEmbeddings",
|
||||||
|
"ImageBindEmbeddings",
|
||||||
|
"SigLipEmbeddings",
|
||||||
|
]
|
||||||
|
|||||||
@@ -21,20 +21,20 @@ class BedRockText(TextEmbeddingFunction):
|
|||||||
"""
|
"""
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "amazon.titan-embed-text-v1"
|
name : str, default "amazon.titan-embed-text-v1"
|
||||||
The model ID of the bedrock model to use. Supported models for are:
|
The model ID of the bedrock model to use. Supported models for are:
|
||||||
- amazon.titan-embed-text-v1
|
- amazon.titan-embed-text-v1
|
||||||
- cohere.embed-english-v3
|
- cohere.embed-english-v3
|
||||||
- cohere.embed-multilingual-v3
|
- cohere.embed-multilingual-v3
|
||||||
region: str, default "us-east-1"
|
region : str, default "us-east-1"
|
||||||
Optional name of the AWS Region in which the service should be called.
|
Optional name of the AWS Region in which the service should be called.
|
||||||
profile_name: str, default None
|
profile_name : str, default None
|
||||||
Optional name of the AWS profile to use for calling the Bedrock service.
|
Optional name of the AWS profile to use for calling the Bedrock service.
|
||||||
If not specified, the default profile will be used.
|
If not specified, the default profile will be used.
|
||||||
assumed_role: str, default None
|
assumed_role : str, default None
|
||||||
Optional ARN of an AWS IAM role to assume for calling the Bedrock service.
|
Optional ARN of an AWS IAM role to assume for calling the Bedrock service.
|
||||||
If not specified, the current active credentials will be used.
|
If not specified, the current active credentials will be used.
|
||||||
role_session_name: str, default "lancedb-embeddings"
|
role_session_name : str, default "lancedb-embeddings"
|
||||||
Optional name of the AWS IAM role session to use for calling the Bedrock
|
Optional name of the AWS IAM role session to use for calling the Bedrock
|
||||||
service. If not specified, "lancedb-embeddings" name will be used.
|
service. If not specified, "lancedb-embeddings" name will be used.
|
||||||
|
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ class CohereEmbeddingFunction(TextEmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "embed-multilingual-v2.0"
|
name : str, default "embed-multilingual-v2.0"
|
||||||
The name of the model to use. List of acceptable models:
|
The name of the model to use. List of acceptable models:
|
||||||
|
|
||||||
* embed-english-v3.0
|
* embed-english-v3.0
|
||||||
@@ -33,12 +33,14 @@ class CohereEmbeddingFunction(TextEmbeddingFunction):
|
|||||||
* embed-english-light-v2.0
|
* embed-english-light-v2.0
|
||||||
* embed-multilingual-v2.0
|
* embed-multilingual-v2.0
|
||||||
|
|
||||||
source_input_type: str, default "search_document"
|
source_input_type : str, default "search_document"
|
||||||
The input type for the source column in the database
|
The input type for the source column in the database
|
||||||
|
|
||||||
query_input_type: str, default "search_query"
|
query_input_type : str, default "search_query"
|
||||||
The input type for the query column in the database
|
The input type for the query column in the database
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
Cohere supports following input types:
|
Cohere supports following input types:
|
||||||
|
|
||||||
| Input Type | Description |
|
| Input Type | Description |
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ class ColPaliEmbeddings(EmbeddingFunction):
|
|||||||
The token pooling strategy to use, by default "hierarchical".
|
The token pooling strategy to use, by default "hierarchical".
|
||||||
- "hierarchical": Progressively pools tokens to reduce sequence length.
|
- "hierarchical": Progressively pools tokens to reduce sequence length.
|
||||||
- "lambda": A simpler pooling that uses a custom `pooling_func`.
|
- "lambda": A simpler pooling that uses a custom `pooling_func`.
|
||||||
pooling_func: typing.Callable, optional
|
pooling_func : typing.Callable, optional
|
||||||
A function to use for pooling when `pooling_strategy` is "lambda".
|
A function to use for pooling when `pooling_strategy` is "lambda".
|
||||||
pool_factor : int
|
pool_factor : int
|
||||||
Factor to reduce sequence length if token pooling is enabled (default 2).
|
Factor to reduce sequence length if token pooling is enabled (default 2).
|
||||||
@@ -52,7 +52,7 @@ class ColPaliEmbeddings(EmbeddingFunction):
|
|||||||
Quantization configuration for the model. (default None, bitsandbytes needed)
|
Quantization configuration for the model. (default None, bitsandbytes needed)
|
||||||
batch_size : int
|
batch_size : int
|
||||||
Batch size for processing inputs (default 2).
|
Batch size for processing inputs (default 2).
|
||||||
offload_folder: str, optional
|
offload_folder : str, optional
|
||||||
Folder to offload model weights if using CPU offloading (default None). This is
|
Folder to offload model weights if using CPU offloading (default None). This is
|
||||||
useful for large models that do not fit in memory.
|
useful for large models that do not fit in memory.
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -48,16 +48,16 @@ class GeminiText(TextEmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "gemini-embedding-001"
|
name : str, default "gemini-embedding-001"
|
||||||
The name of the model to use. Supported models include:
|
The name of the model to use. Supported models include:
|
||||||
- "gemini-embedding-001" (768 dimensions)
|
- "gemini-embedding-001" (768 dimensions)
|
||||||
|
|
||||||
Note: The legacy "models/embedding-001" format is also supported but
|
Note: The legacy "models/embedding-001" format is also supported but
|
||||||
"gemini-embedding-001" is recommended.
|
"gemini-embedding-001" is recommended.
|
||||||
|
|
||||||
query_task_type: str, default "retrieval_query"
|
query_task_type : str, default "retrieval_query"
|
||||||
Sets the task type for the queries.
|
Sets the task type for the queries.
|
||||||
source_task_type: str, default "retrieval_document"
|
source_task_type : str, default "retrieval_document"
|
||||||
Sets the task type for ingestion.
|
Sets the task type for ingestion.
|
||||||
|
|
||||||
Examples
|
Examples
|
||||||
|
|||||||
@@ -26,13 +26,13 @@ class GteEmbeddings(TextEmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "thenlper/gte-large"
|
name : str, default "thenlper/gte-large"
|
||||||
The name of the model to use.
|
The name of the model to use.
|
||||||
device: str, default "cpu"
|
device : str, default "cpu"
|
||||||
Sets the device type for the model.
|
Sets the device type for the model.
|
||||||
normalize: str, default "True"
|
normalize : str, default "True"
|
||||||
Controls normalize param in encode function for the transformer.
|
Controls normalize param in encode function for the transformer.
|
||||||
mlx: bool, default False
|
mlx : bool, default False
|
||||||
Controls which model to use. False for gte-large,True for the mlx version.
|
Controls which model to use. False for gte-large,True for the mlx version.
|
||||||
|
|
||||||
Examples
|
Examples
|
||||||
|
|||||||
@@ -35,23 +35,23 @@ class InstructorEmbeddingFunction(TextEmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str
|
name : str
|
||||||
The name of the model to use. Available models are listed at
|
The name of the model to use. Available models are listed at
|
||||||
https://github.com/xlang-ai/instructor-embedding#model-list;
|
https://github.com/xlang-ai/instructor-embedding#model-list;
|
||||||
The default model is hkunlp/instructor-base
|
The default model is hkunlp/instructor-base
|
||||||
batch_size: int, default 32
|
batch_size : int, default 32
|
||||||
The batch size to use when generating embeddings
|
The batch size to use when generating embeddings
|
||||||
device: str, default "cpu"
|
device : str, default "cpu"
|
||||||
The device to use when generating embeddings
|
The device to use when generating embeddings
|
||||||
show_progress_bar: bool, default True
|
show_progress_bar : bool, default True
|
||||||
Whether to show a progress bar when generating embeddings
|
Whether to show a progress bar when generating embeddings
|
||||||
normalize_embeddings: bool, default True
|
normalize_embeddings : bool, default True
|
||||||
Whether to normalize the embeddings
|
Whether to normalize the embeddings
|
||||||
quantize: bool, default False
|
quantize : bool, default False
|
||||||
Whether to quantize the model
|
Whether to quantize the model
|
||||||
source_instruction: str, default "represent the document for retrieval"
|
source_instruction : str, default "represent the document for retrieval"
|
||||||
The instruction for the source column
|
The instruction for the source column
|
||||||
query_instruction: str, default "represent the document for retrieving the most
|
query_instruction : str, default "represent the document for retrieving the most
|
||||||
similar documents"
|
similar documents"
|
||||||
The instruction for the query
|
The instruction for the query
|
||||||
|
|
||||||
|
|||||||
@@ -40,10 +40,10 @@ class JinaEmbeddings(EmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "jina-clip-v1". Note that some models support both image
|
name : str, default "jina-clip-v1". Note that some models support both image
|
||||||
and text embeddings and some just text embedding
|
and text embeddings and some just text embedding
|
||||||
|
|
||||||
api_key: str, default None
|
api_key : str, default None
|
||||||
The api key to access Jina API. If you pass None, you can set JINA_API_KEY
|
The api key to access Jina API. If you pass None, you can set JINA_API_KEY
|
||||||
environment variable
|
environment variable
|
||||||
|
|
||||||
|
|||||||
@@ -21,13 +21,13 @@ class SentenceTransformerEmbeddings(TextEmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str, default "all-MiniLM-L6-v2"
|
name : str, default "all-MiniLM-L6-v2"
|
||||||
The name of the model to use.
|
The name of the model to use.
|
||||||
device: str, default "cpu"
|
device : str, default "cpu"
|
||||||
The device to use for the model
|
The device to use for the model
|
||||||
normalize: bool, default True
|
normalize : bool, default True
|
||||||
Whether to normalize the embeddings
|
Whether to normalize the embeddings
|
||||||
trust_remote_code: bool, default True
|
trust_remote_code : bool, default True
|
||||||
Whether to trust the remote code
|
Whether to trust the remote code
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|||||||
@@ -167,7 +167,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction):
|
|||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
name: str
|
name : str
|
||||||
The name of the model to use. List of acceptable models:
|
The name of the model to use. List of acceptable models:
|
||||||
|
|
||||||
* voyage-4 (1024 dims, general-purpose and multilingual retrieval)
|
* voyage-4 (1024 dims, general-purpose and multilingual retrieval)
|
||||||
@@ -185,7 +185,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction):
|
|||||||
* voyage-law-2
|
* voyage-law-2
|
||||||
* voyage-code-2
|
* voyage-code-2
|
||||||
|
|
||||||
output_dimension: int, optional
|
output_dimension : int, optional
|
||||||
The output dimension for models that support flexible dimensions.
|
The output dimension for models that support flexible dimensions.
|
||||||
Currently only voyage-multimodal-3.5 supports this feature.
|
Currently only voyage-multimodal-3.5 supports this feature.
|
||||||
Valid options: 256, 512, 1024 (default), 2048.
|
Valid options: 256, 512, 1024 (default), 2048.
|
||||||
|
|||||||
@@ -219,7 +219,7 @@ class HnswPq:
|
|||||||
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
||||||
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
||||||
|
|
||||||
num_partitions, default sqrt(num_rows)
|
num_partitions: int, default sqrt(num_rows)
|
||||||
|
|
||||||
The number of IVF partitions to create.
|
The number of IVF partitions to create.
|
||||||
|
|
||||||
@@ -228,7 +228,7 @@ class HnswPq:
|
|||||||
will require too much memory. Each partition becomes its own HNSW graph, so
|
will require too much memory. Each partition becomes its own HNSW graph, so
|
||||||
setting this value higher reduces the peak memory use of training.
|
setting this value higher reduces the peak memory use of training.
|
||||||
|
|
||||||
num_sub_vectors, default is vector dimension / 16
|
num_sub_vectors: int, default is vector dimension / 16
|
||||||
|
|
||||||
Number of sub-vectors of PQ.
|
Number of sub-vectors of PQ.
|
||||||
|
|
||||||
@@ -244,13 +244,13 @@ class HnswPq:
|
|||||||
If the dimension is not visible by 8 then we use 1 subvector. This is not
|
If the dimension is not visible by 8 then we use 1 subvector. This is not
|
||||||
ideal and will likely result in poor performance.
|
ideal and will likely result in poor performance.
|
||||||
|
|
||||||
num_bits: int, default 8
|
num_bits: int, default 8
|
||||||
Number of bits to encode each sub-vector.
|
Number of bits to encode each sub-vector.
|
||||||
|
|
||||||
This value controls how much the sub-vectors are compressed. The more bits
|
This value controls how much the sub-vectors are compressed. The more bits
|
||||||
the more accurate the index but the slower search. Only 4 and 8 are supported.
|
the more accurate the index but the slower search. Only 4 and 8 are supported.
|
||||||
|
|
||||||
max_iterations, default 50
|
max_iterations: int, default 50
|
||||||
|
|
||||||
Max iterations to train kmeans.
|
Max iterations to train kmeans.
|
||||||
|
|
||||||
@@ -263,7 +263,7 @@ class HnswPq:
|
|||||||
those cases it is unlikely that setting this larger will lead to the index
|
those cases it is unlikely that setting this larger will lead to the index
|
||||||
converging anyways.
|
converging anyways.
|
||||||
|
|
||||||
sample_rate, default 256
|
sample_rate: int, default 256
|
||||||
|
|
||||||
The rate used to calculate the number of training vectors for kmeans.
|
The rate used to calculate the number of training vectors for kmeans.
|
||||||
|
|
||||||
@@ -279,14 +279,14 @@ class HnswPq:
|
|||||||
Increasing this value might improve the quality of the index but in
|
Increasing this value might improve the quality of the index but in
|
||||||
most cases the default should be sufficient.
|
most cases the default should be sufficient.
|
||||||
|
|
||||||
m, default 20
|
m: int, default 20
|
||||||
|
|
||||||
The number of neighbors to select for each vector in the HNSW graph.
|
The number of neighbors to select for each vector in the HNSW graph.
|
||||||
|
|
||||||
This value controls the tradeoff between search speed and accuracy.
|
This value controls the tradeoff between search speed and accuracy.
|
||||||
The higher the value the more accurate the search but the slower it will be.
|
The higher the value the more accurate the search but the slower it will be.
|
||||||
|
|
||||||
ef_construction, default 300
|
ef_construction: int, default 300
|
||||||
|
|
||||||
The number of candidates to evaluate during the construction of the HNSW graph.
|
The number of candidates to evaluate during the construction of the HNSW graph.
|
||||||
|
|
||||||
@@ -297,7 +297,7 @@ class HnswPq:
|
|||||||
This value should be set to a value that is not less than `ef` in the
|
This value should be set to a value that is not less than `ef` in the
|
||||||
search phase.
|
search phase.
|
||||||
|
|
||||||
target_partition_size, default is 1,048,576
|
target_partition_size: int, default is 1,048,576
|
||||||
|
|
||||||
The target size of each partition.
|
The target size of each partition.
|
||||||
|
|
||||||
@@ -351,7 +351,7 @@ class HnswSq:
|
|||||||
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
||||||
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
||||||
|
|
||||||
num_partitions, default sqrt(num_rows)
|
num_partitions: int, default sqrt(num_rows)
|
||||||
|
|
||||||
The number of IVF partitions to create.
|
The number of IVF partitions to create.
|
||||||
|
|
||||||
@@ -360,7 +360,7 @@ class HnswSq:
|
|||||||
will require too much memory. Each partition becomes its own HNSW graph, so
|
will require too much memory. Each partition becomes its own HNSW graph, so
|
||||||
setting this value higher reduces the peak memory use of training.
|
setting this value higher reduces the peak memory use of training.
|
||||||
|
|
||||||
max_iterations, default 50
|
max_iterations: int, default 50
|
||||||
|
|
||||||
Max iterations to train kmeans.
|
Max iterations to train kmeans.
|
||||||
|
|
||||||
@@ -373,7 +373,7 @@ class HnswSq:
|
|||||||
In those cases it is unlikely that setting this larger will lead to
|
In those cases it is unlikely that setting this larger will lead to
|
||||||
the index converging anyways.
|
the index converging anyways.
|
||||||
|
|
||||||
sample_rate, default 256
|
sample_rate: int, default 256
|
||||||
|
|
||||||
The rate used to calculate the number of training vectors for kmeans.
|
The rate used to calculate the number of training vectors for kmeans.
|
||||||
|
|
||||||
@@ -389,14 +389,14 @@ class HnswSq:
|
|||||||
Increasing this value might improve the quality of the index but in
|
Increasing this value might improve the quality of the index but in
|
||||||
most cases the default should be sufficient.
|
most cases the default should be sufficient.
|
||||||
|
|
||||||
m, default 20
|
m: int, default 20
|
||||||
|
|
||||||
The number of neighbors to select for each vector in the HNSW graph.
|
The number of neighbors to select for each vector in the HNSW graph.
|
||||||
|
|
||||||
This value controls the tradeoff between search speed and accuracy.
|
This value controls the tradeoff between search speed and accuracy.
|
||||||
The higher the value the more accurate the search but the slower it will be.
|
The higher the value the more accurate the search but the slower it will be.
|
||||||
|
|
||||||
ef_construction, default 300
|
ef_construction: int, default 300
|
||||||
|
|
||||||
The number of candidates to evaluate during the construction of the HNSW graph.
|
The number of candidates to evaluate during the construction of the HNSW graph.
|
||||||
|
|
||||||
@@ -407,7 +407,7 @@ class HnswSq:
|
|||||||
This value should be set to a value that is not less than `ef` in the search
|
This value should be set to a value that is not less than `ef` in the search
|
||||||
phase.
|
phase.
|
||||||
|
|
||||||
target_partition_size, default is 1,048,576
|
target_partition_size: int, default is 1,048,576
|
||||||
|
|
||||||
The target size of each partition.
|
The target size of each partition.
|
||||||
|
|
||||||
@@ -460,7 +460,7 @@ class HnswFlat:
|
|||||||
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
|
||||||
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
l2 norm is 1), then dot distance is equivalent to the cosine distance.
|
||||||
|
|
||||||
num_partitions, default sqrt(num_rows)
|
num_partitions: int, default sqrt(num_rows)
|
||||||
|
|
||||||
The number of IVF partitions to create.
|
The number of IVF partitions to create.
|
||||||
|
|
||||||
@@ -470,18 +470,18 @@ class HnswFlat:
|
|||||||
graph, so setting this value higher reduces the peak memory use of
|
graph, so setting this value higher reduces the peak memory use of
|
||||||
training.
|
training.
|
||||||
|
|
||||||
max_iterations, default 50
|
max_iterations: int, default 50
|
||||||
|
|
||||||
Max iterations to train kmeans.
|
Max iterations to train kmeans.
|
||||||
|
|
||||||
When training an IVF index we use kmeans to calculate the partitions.
|
When training an IVF index we use kmeans to calculate the partitions.
|
||||||
This parameter controls how many iterations of kmeans to run.
|
This parameter controls how many iterations of kmeans to run.
|
||||||
|
|
||||||
sample_rate, default 256
|
sample_rate: int, default 256
|
||||||
|
|
||||||
The rate used to calculate the number of training vectors for kmeans.
|
The rate used to calculate the number of training vectors for kmeans.
|
||||||
|
|
||||||
m, default 20
|
m: int, default 20
|
||||||
|
|
||||||
The number of neighbors to select for each vector in the HNSW graph.
|
The number of neighbors to select for each vector in the HNSW graph.
|
||||||
|
|
||||||
@@ -489,7 +489,7 @@ class HnswFlat:
|
|||||||
The higher the value the more accurate the search but the slower it
|
The higher the value the more accurate the search but the slower it
|
||||||
will be.
|
will be.
|
||||||
|
|
||||||
ef_construction, default 300
|
ef_construction: int, default 300
|
||||||
|
|
||||||
The number of candidates to evaluate during the construction of the HNSW
|
The number of candidates to evaluate during the construction of the HNSW
|
||||||
graph.
|
graph.
|
||||||
@@ -501,7 +501,7 @@ class HnswFlat:
|
|||||||
than 500. This value should be set to a value that is not less than `ef`
|
than 500. This value should be set to a value that is not less than `ef`
|
||||||
in the search phase.
|
in the search phase.
|
||||||
|
|
||||||
target_partition_size, default is 1,048,576
|
target_partition_size: int, default is 1,048,576
|
||||||
|
|
||||||
The target size of each partition.
|
The target size of each partition.
|
||||||
"""
|
"""
|
||||||
@@ -605,7 +605,7 @@ class IvfFlat:
|
|||||||
|
|
||||||
The default value is 256.
|
The default value is 256.
|
||||||
|
|
||||||
target_partition_size, default is 8192
|
target_partition_size: int, default is 8192
|
||||||
|
|
||||||
The target size of each partition.
|
The target size of each partition.
|
||||||
|
|
||||||
@@ -769,7 +769,7 @@ class IvfPq:
|
|||||||
|
|
||||||
The default value is 256.
|
The default value is 256.
|
||||||
|
|
||||||
target_partition_size, default is 8192
|
target_partition_size: int, default is 8192
|
||||||
|
|
||||||
The target size of each partition.
|
The target size of each partition.
|
||||||
|
|
||||||
@@ -830,7 +830,7 @@ class IvfRq:
|
|||||||
sample_rate: int, default 256
|
sample_rate: int, default 256
|
||||||
Controls the number of training vectors: sample_rate * num_partitions.
|
Controls the number of training vectors: sample_rate * num_partitions.
|
||||||
|
|
||||||
target_partition_size, default is 8192
|
target_partition_size: int, default is 8192
|
||||||
Target size of each partition.
|
Target size of each partition.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -845,6 +845,9 @@ class IvfRq:
|
|||||||
accelerator: Optional[str] = None
|
accelerator: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
# The API reference renders this module with a single mkdocstrings directive,
|
||||||
|
# which only picks up names listed here. New public names must be added to this
|
||||||
|
# list, or they will silently go undocumented.
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"BTree",
|
"BTree",
|
||||||
"IvfPq",
|
"IvfPq",
|
||||||
|
|||||||
@@ -438,7 +438,8 @@ class Permutation:
|
|||||||
_reader: Optional[PermutationReader] = None,
|
_reader: Optional[PermutationReader] = None,
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
Internal constructor. Use [from_tables](#from_tables) instead.
|
Internal constructor. Use
|
||||||
|
[from_tables][lancedb.permutation.Permutation.from_tables] instead.
|
||||||
"""
|
"""
|
||||||
assert base_table is not None, "base_table is required"
|
assert base_table is not None, "base_table is required"
|
||||||
assert selection is not None, "selection is required"
|
assert selection is not None, "selection is required"
|
||||||
@@ -985,8 +986,9 @@ class Permutation:
|
|||||||
types. Conversion of strings, lists, and structs will require creating python
|
types. Conversion of strings, lists, and structs will require creating python
|
||||||
objects and this is not zero-copy.
|
objects and this is not zero-copy.
|
||||||
|
|
||||||
For custom formatting, use [with_transform](#with_transform) which overrides
|
For custom formatting, use
|
||||||
this method.
|
[with_transform][lancedb.permutation.Permutation.with_transform] which
|
||||||
|
overrides this method.
|
||||||
"""
|
"""
|
||||||
assert format is not None, "format is required"
|
assert format is not None, "format is required"
|
||||||
if format == "python":
|
if format == "python":
|
||||||
@@ -1061,7 +1063,8 @@ class Permutation:
|
|||||||
Note: this method returns a new permutation and does not modify `self`
|
Note: this method returns a new permutation and does not modify `self`
|
||||||
It is provided for compatibility with the huggingface Dataset API.
|
It is provided for compatibility with the huggingface Dataset API.
|
||||||
|
|
||||||
Use [with_skip](#with_skip) instead to avoid confusion.
|
Use [with_skip][lancedb.permutation.Permutation.with_skip] instead to
|
||||||
|
avoid confusion.
|
||||||
"""
|
"""
|
||||||
return self.with_skip(skip)
|
return self.with_skip(skip)
|
||||||
|
|
||||||
@@ -1084,7 +1087,8 @@ class Permutation:
|
|||||||
Note: this method returns a new permutation and does not modify `self`
|
Note: this method returns a new permutation and does not modify `self`
|
||||||
It is provided for compatibility with the huggingface Dataset API.
|
It is provided for compatibility with the huggingface Dataset API.
|
||||||
|
|
||||||
Use [with_take](#with_take) instead to avoid confusion.
|
Use [with_take][lancedb.permutation.Permutation.with_take] instead to
|
||||||
|
avoid confusion.
|
||||||
"""
|
"""
|
||||||
return self.with_take(limit)
|
return self.with_take(limit)
|
||||||
|
|
||||||
@@ -1107,7 +1111,8 @@ class Permutation:
|
|||||||
Note: this method returns a new permutation and does not modify `self`
|
Note: this method returns a new permutation and does not modify `self`
|
||||||
It is provided for compatibility with the huggingface Dataset API.
|
It is provided for compatibility with the huggingface Dataset API.
|
||||||
|
|
||||||
Use [with_repeat](#with_repeat) instead to avoid confusion.
|
Use [with_repeat][lancedb.permutation.Permutation.with_repeat] instead
|
||||||
|
to avoid confusion.
|
||||||
"""
|
"""
|
||||||
return self.with_repeat(times)
|
return self.with_repeat(times)
|
||||||
|
|
||||||
|
|||||||
@@ -651,7 +651,8 @@ class Query(pydantic.BaseModel):
|
|||||||
distance_type : Optional[str]
|
distance_type : Optional[str]
|
||||||
the distance type to use for vector search
|
the distance type to use for vector search
|
||||||
|
|
||||||
This can be l2 (default), cosine and dot. See [metric definitions][search] for
|
This can be l2 (default), cosine and dot. See
|
||||||
|
[metric definitions](https://lancedb.com/docs/search/vector-search/) for
|
||||||
more details.
|
more details.
|
||||||
|
|
||||||
If this is not a vector search this will be None.
|
If this is not a vector search this will be None.
|
||||||
@@ -664,8 +665,9 @@ class Query(pydantic.BaseModel):
|
|||||||
|
|
||||||
- A higher number makes search more accurate but also slower.
|
- A higher number makes search more accurate but also slower.
|
||||||
|
|
||||||
- See discussion in [Querying an ANN Index][querying-an-ann-index] for
|
- See discussion in
|
||||||
tuning advice.
|
[Querying an ANN Index](https://lancedb.com/docs/indexing/)
|
||||||
|
for tuning advice.
|
||||||
|
|
||||||
Will be None if this is not a vector search.
|
Will be None if this is not a vector search.
|
||||||
refine_factor : Optional[int]
|
refine_factor : Optional[int]
|
||||||
@@ -673,8 +675,9 @@ class Query(pydantic.BaseModel):
|
|||||||
|
|
||||||
- A higher number makes search more accurate but also slower.
|
- A higher number makes search more accurate but also slower.
|
||||||
|
|
||||||
- See discussion in [Querying an ANN Index][querying-an-ann-index] for
|
- See discussion in
|
||||||
tuning advice.
|
[Querying an ANN Index](https://lancedb.com/docs/indexing/)
|
||||||
|
for tuning advice.
|
||||||
|
|
||||||
Will be None if this is not a vector search.
|
Will be None if this is not a vector search.
|
||||||
lower_bound : Optional[float]
|
lower_bound : Optional[float]
|
||||||
@@ -1651,8 +1654,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
|
|||||||
Higher values will yield better recall (more likely to find vectors if
|
Higher values will yield better recall (more likely to find vectors if
|
||||||
they exist) at the expense of latency.
|
they exist) at the expense of latency.
|
||||||
|
|
||||||
See discussion in [Querying an ANN Index][querying-an-ann-index] for
|
See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/)
|
||||||
tuning advice.
|
for tuning advice.
|
||||||
|
|
||||||
This method sets both the minimum and maximum number of probes to the same
|
This method sets both the minimum and maximum number of probes to the same
|
||||||
value. See `minimum_nprobes` and `maximum_nprobes` for more fine-grained
|
value. See `minimum_nprobes` and `maximum_nprobes` for more fine-grained
|
||||||
@@ -1752,8 +1755,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
|
|||||||
As an example, a refine factor of 2 will sample 2x as many vectors as
|
As an example, a refine factor of 2 will sample 2x as many vectors as
|
||||||
requested, re-ranks them, and returns the top half most relevant results.
|
requested, re-ranks them, and returns the top half most relevant results.
|
||||||
|
|
||||||
See discussion in [Querying an ANN Index][querying-an-ann-index] for
|
See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/)
|
||||||
tuning advice.
|
for tuning advice.
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
@@ -3379,8 +3382,9 @@ class AsyncQuery(AsyncStandardQuery):
|
|||||||
are various ANN search parameters that will let you fine tune your recall
|
are various ANN search parameters that will let you fine tune your recall
|
||||||
accuracy vs search latency.
|
accuracy vs search latency.
|
||||||
|
|
||||||
Vector searches always have a [limit][]. If `limit` has not been called then
|
Vector searches always have a
|
||||||
a default `limit` of 10 will be used.
|
[limit][lancedb.query.AsyncVectorQuery.limit]. If `limit` has not been
|
||||||
|
called then a default `limit` of 10 will be used.
|
||||||
|
|
||||||
Typically, a single vector is passed in as the query. However, you can also
|
Typically, a single vector is passed in as the query. However, you can also
|
||||||
pass in multiple vectors. When multiple vectors are passed in, if the vector
|
pass in multiple vectors. When multiple vectors are passed in, if the vector
|
||||||
@@ -3511,8 +3515,9 @@ class AsyncFTSQuery(AsyncStandardQuery):
|
|||||||
are various ANN search parameters that will let you fine tune your recall
|
are various ANN search parameters that will let you fine tune your recall
|
||||||
accuracy vs search latency.
|
accuracy vs search latency.
|
||||||
|
|
||||||
Hybrid searches always have a [limit][]. If `limit` has not been called then
|
Hybrid searches always have a
|
||||||
a default `limit` of 10 will be used.
|
[limit][lancedb.query.AsyncHybridQuery.limit]. If `limit` has not been
|
||||||
|
called then a default `limit` of 10 will be used.
|
||||||
|
|
||||||
Typically, a single vector is passed in as the query. However, you can also
|
Typically, a single vector is passed in as the query. However, you can also
|
||||||
pass in multiple vectors. This can be useful if you want to find the nearest
|
pass in multiple vectors. This can be useful if you want to find the nearest
|
||||||
|
|||||||
@@ -11,6 +11,9 @@ from lancedb import __version__
|
|||||||
from .header import HeaderProvider
|
from .header import HeaderProvider
|
||||||
from .oauth import OAuthConfig, OAuthFlowType
|
from .oauth import OAuthConfig, OAuthFlowType
|
||||||
|
|
||||||
|
# The API reference renders this module with a single mkdocstrings directive,
|
||||||
|
# which only picks up names listed here. New public names must be added to this
|
||||||
|
# list, or they will silently go undocumented.
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"TimeoutConfig",
|
"TimeoutConfig",
|
||||||
"RetryConfig",
|
"RetryConfig",
|
||||||
|
|||||||
@@ -53,9 +53,9 @@ class RetryError(LanceDBClientError):
|
|||||||
"""An error that occurs when the client has exceeded the maximum number of retries.
|
"""An error that occurs when the client has exceeded the maximum number of retries.
|
||||||
|
|
||||||
The retry strategy can be adjusted by setting the
|
The retry strategy can be adjusted by setting the
|
||||||
[retry_config](lancedb.remote.ClientConfig.retry_config) in the client
|
[retry_config][lancedb.remote.ClientConfig.retry_config] in the client
|
||||||
configuration. This is passed in the `client_config` argument of
|
configuration. This is passed in the `client_config` argument of
|
||||||
[connect](lancedb.connect) and [connect_async](lancedb.connect_async).
|
[connect][lancedb.connect] and [connect_async][lancedb.connect_async].
|
||||||
|
|
||||||
The __cause__ attribute of this exception will be the last exception that
|
The __cause__ attribute of this exception will be the last exception that
|
||||||
caused the retry to fail. It will be an
|
caused the retry to fail. It will be an
|
||||||
|
|||||||
@@ -580,8 +580,9 @@ class RemoteTable(Table):
|
|||||||
progress: Optional[Union[bool, Callable, Any]] = None,
|
progress: Optional[Union[bool, Callable, Any]] = None,
|
||||||
write_parallelism: Optional[int] = None,
|
write_parallelism: Optional[int] = None,
|
||||||
) -> AddResult:
|
) -> AddResult:
|
||||||
"""Add more data to the [Table](Table). It has the same API signature as
|
"""Add more data to the [Table][lancedb.table.Table].
|
||||||
the OSS version.
|
|
||||||
|
It has the same API signature as the OSS version.
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
@@ -641,7 +642,8 @@ class RemoteTable(Table):
|
|||||||
fast_search: bool = False,
|
fast_search: bool = False,
|
||||||
) -> LanceVectorQueryBuilder:
|
) -> LanceVectorQueryBuilder:
|
||||||
"""Create a search query to find the nearest neighbors
|
"""Create a search query to find the nearest neighbors
|
||||||
of the given query vector. We currently support [vector search][search]
|
of the given query vector. We currently support
|
||||||
|
[vector search](https://lancedb.com/docs/search/vector-search/)
|
||||||
|
|
||||||
All query options are defined in
|
All query options are defined in
|
||||||
[LanceVectorQueryBuilder][lancedb.query.LanceVectorQueryBuilder].
|
[LanceVectorQueryBuilder][lancedb.query.LanceVectorQueryBuilder].
|
||||||
|
|||||||
@@ -14,6 +14,9 @@ from .answerdotai import AnswerdotaiRerankers
|
|||||||
from .voyageai import VoyageAIReranker
|
from .voyageai import VoyageAIReranker
|
||||||
from .watsonx import WatsonxReranker
|
from .watsonx import WatsonxReranker
|
||||||
|
|
||||||
|
# The API reference renders this module with a single mkdocstrings directive,
|
||||||
|
# which only picks up names listed here. New public names must be added to this
|
||||||
|
# list, or they will silently go undocumented.
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"Reranker",
|
"Reranker",
|
||||||
"CrossEncoderReranker",
|
"CrossEncoderReranker",
|
||||||
|
|||||||
@@ -1211,7 +1211,7 @@ class Table(ABC):
|
|||||||
progress: Optional[Union[bool, Callable, Any]] = None,
|
progress: Optional[Union[bool, Callable, Any]] = None,
|
||||||
write_parallelism: Optional[int] = None,
|
write_parallelism: Optional[int] = None,
|
||||||
) -> AddResult:
|
) -> AddResult:
|
||||||
"""Add more data to the [Table](Table).
|
"""Add more data to the [Table][lancedb.table.Table].
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
@@ -1343,8 +1343,8 @@ class Table(ABC):
|
|||||||
fts_columns: Optional[Union[str, List[str]]] = None,
|
fts_columns: Optional[Union[str, List[str]]] = None,
|
||||||
) -> LanceQueryBuilder:
|
) -> LanceQueryBuilder:
|
||||||
"""Create a search query to find the nearest neighbors
|
"""Create a search query to find the nearest neighbors
|
||||||
of the given query vector. We currently support [vector search][search]
|
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
|
||||||
and [full-text search][experimental-full-text-search].
|
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
|
||||||
|
|
||||||
All query options are defined in
|
All query options are defined in
|
||||||
[LanceQueryBuilder][lancedb.query.LanceQueryBuilder].
|
[LanceQueryBuilder][lancedb.query.LanceQueryBuilder].
|
||||||
@@ -1778,7 +1778,7 @@ class Table(ABC):
|
|||||||
for faster reads.
|
for faster reads.
|
||||||
|
|
||||||
Arguments are passed onto Lance's
|
Arguments are passed onto Lance's
|
||||||
[compact_files][lance.dataset.DatasetOptimizer.compact_files].
|
`lance.dataset.DatasetOptimizer.compact_files`.
|
||||||
For most cases, the default should be fine.
|
For most cases, the default should be fine.
|
||||||
|
|
||||||
See Also
|
See Also
|
||||||
@@ -1832,6 +1832,8 @@ class Table(ABC):
|
|||||||
retrain: bool, default False
|
retrain: bool, default False
|
||||||
This parameter is no longer used and is deprecated.
|
This parameter is no longer used and is deprecated.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
The frequency an application should call optimize is based on the frequency of
|
The frequency an application should call optimize is based on the frequency of
|
||||||
data modifications. If data is frequently added, deleted, or updated then
|
data modifications. If data is frequently added, deleted, or updated then
|
||||||
optimize should be run frequently. A good rule of thumb is to run optimize if
|
optimize should be run frequently. A good rule of thumb is to run optimize if
|
||||||
@@ -1986,15 +1988,14 @@ class Table(ABC):
|
|||||||
change permanent you can use the `[Self::restore]` method.
|
change permanent you can use the `[Self::restore]` method.
|
||||||
|
|
||||||
Any operation that modifies the table will fail while the table is in a checked
|
Any operation that modifies the table will fail while the table is in a checked
|
||||||
out state.
|
out state. To return the table to a normal state use
|
||||||
|
`[Self::checkout_latest]`.
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
version: int | str,
|
version: int | str,
|
||||||
The version to check out. A version number (`int`) or a tag
|
The version to check out. A version number (`int`) or a tag
|
||||||
(`str`) can be provided.
|
(`str`) can be provided.
|
||||||
|
|
||||||
To return the table to a normal state use `[Self::checkout_latest]`
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
@@ -3387,8 +3388,8 @@ class LanceTable(Table):
|
|||||||
fts_columns: Optional[Union[str, List[str]]] = None,
|
fts_columns: Optional[Union[str, List[str]]] = None,
|
||||||
) -> LanceQueryBuilder:
|
) -> LanceQueryBuilder:
|
||||||
"""Create a search query to find the nearest neighbors
|
"""Create a search query to find the nearest neighbors
|
||||||
of the given query vector. We currently support [vector search][search]
|
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
|
||||||
and [full-text search][search].
|
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
|
||||||
|
|
||||||
Examples
|
Examples
|
||||||
--------
|
--------
|
||||||
@@ -3418,8 +3419,9 @@ class LanceTable(Table):
|
|||||||
- *default None*.
|
- *default None*.
|
||||||
Acceptable types are: list, np.ndarray, PIL.Image.Image
|
Acceptable types are: list, np.ndarray, PIL.Image.Image
|
||||||
|
|
||||||
- If None then the select/[where][sql]/limit clauses are applied
|
- If None then the
|
||||||
to filter the table
|
select/[where][lancedb.query.LanceQueryBuilder.where]/limit clauses
|
||||||
|
are applied to filter the table
|
||||||
vector_column_name: str, optional
|
vector_column_name: str, optional
|
||||||
The name of the vector column to search.
|
The name of the vector column to search.
|
||||||
|
|
||||||
@@ -3813,6 +3815,8 @@ class LanceTable(Table):
|
|||||||
retrain: bool, default False
|
retrain: bool, default False
|
||||||
This parameter is no longer used and is deprecated.
|
This parameter is no longer used and is deprecated.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
The frequency an application should call optimize is based on the frequency of
|
The frequency an application should call optimize is based on the frequency of
|
||||||
data modifications. If data is frequently added, deleted, or updated then
|
data modifications. If data is frequently added, deleted, or updated then
|
||||||
optimize should be run frequently. A good rule of thumb is to run optimize if
|
optimize should be run frequently. A good rule of thumb is to run optimize if
|
||||||
@@ -4691,7 +4695,7 @@ class AsyncTable:
|
|||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
**kwargs
|
**kwargs
|
||||||
Forwarded to [`lance.dataset`][lance.dataset].
|
Forwarded to `lance.dataset`.
|
||||||
|
|
||||||
Returns
|
Returns
|
||||||
-------
|
-------
|
||||||
@@ -5010,7 +5014,7 @@ class AsyncTable:
|
|||||||
progress: Optional[Union[bool, Callable, Any]] = None,
|
progress: Optional[Union[bool, Callable, Any]] = None,
|
||||||
write_parallelism: Optional[int] = None,
|
write_parallelism: Optional[int] = None,
|
||||||
) -> AddResult:
|
) -> AddResult:
|
||||||
"""Add more data to the [Table](Table).
|
"""Add more data to the [AsyncTable][lancedb.table.AsyncTable].
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
@@ -5212,8 +5216,8 @@ class AsyncTable:
|
|||||||
fts_columns: Optional[Union[str, List[str]]] = None,
|
fts_columns: Optional[Union[str, List[str]]] = None,
|
||||||
) -> Union[AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]:
|
) -> Union[AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]:
|
||||||
"""Create a search query to find the nearest neighbors
|
"""Create a search query to find the nearest neighbors
|
||||||
of the given query vector. We currently support [vector search][search]
|
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
|
||||||
and [full-text search][experimental-full-text-search].
|
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
|
||||||
|
|
||||||
All query options are defined in [AsyncQuery][lancedb.query.AsyncQuery].
|
All query options are defined in [AsyncQuery][lancedb.query.AsyncQuery].
|
||||||
|
|
||||||
@@ -5774,15 +5778,14 @@ class AsyncTable:
|
|||||||
change permanent you can use the `[Self::restore]` method.
|
change permanent you can use the `[Self::restore]` method.
|
||||||
|
|
||||||
Any operation that modifies the table will fail while the table is in a checked
|
Any operation that modifies the table will fail while the table is in a checked
|
||||||
out state.
|
out state. To return the table to a normal state use
|
||||||
|
`[Self::checkout_latest]`.
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
version: int | str,
|
version: int | str,
|
||||||
The version to check out. A version number (`int`) or a tag
|
The version to check out. A version number (`int`) or a tag
|
||||||
(`str`) can be provided.
|
(`str`) can be provided.
|
||||||
|
|
||||||
To return the table to a normal state use `[Self::checkout_latest]`
|
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
await self._inner.checkout(version)
|
await self._inner.checkout(version)
|
||||||
@@ -5966,6 +5969,8 @@ class AsyncTable:
|
|||||||
retrain: bool, default False
|
retrain: bool, default False
|
||||||
This parameter is no longer used and is deprecated.
|
This parameter is no longer used and is deprecated.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
The frequency an application should call optimize is based on the frequency of
|
The frequency an application should call optimize is based on the frequency of
|
||||||
data modifications. If data is frequently added, deleted, or updated then
|
data modifications. If data is frequently added, deleted, or updated then
|
||||||
optimize should be run frequently. A good rule of thumb is to run optimize if
|
optimize should be run frequently. A good rule of thumb is to run optimize if
|
||||||
@@ -6346,6 +6351,8 @@ class Branches:
|
|||||||
dry_run: bool, default False
|
dry_run: bool, default False
|
||||||
When True, only preview. When False, attempt the merge.
|
When True, only preview. When False, attempt the merge.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
A rejected merge returns ``status="rejected"`` instead of raising.
|
A rejected merge returns ``status="rejected"`` instead of raising.
|
||||||
"""
|
"""
|
||||||
return LOOP.run(self._table.branches.merge(from_branch, dry_run))
|
return LOOP.run(self._table.branches.merge(from_branch, dry_run))
|
||||||
|
|||||||
Reference in New Issue
Block a user