mirror of
https://github.com/lancedb/lancedb.git
synced 2025-12-27 23:12:58 +00:00
7ac5f74c80
BREAKING CHANGE: embedding function implementations in Node need to now call `resolveVariables()` in their constructors and should **not** implement `toJSON()`. This tries to address the handling of secrets. In Node, they are currently lost. In Python, they are currently leaked into the table schema metadata. This PR introduces an in-memory variable store on the function registry. It also allows embedding function definitions to label certain config values as "sensitive", and the preprocessing logic will raise an error if users try to pass in hard-coded values. Closes #2110 Closes #521 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>
263 lines
5.3 KiB
Markdown
263 lines
5.3 KiB
Markdown
[**@lancedb/lancedb**](../../../README.md) • **Docs**
|
|
|
|
***
|
|
|
|
[@lancedb/lancedb](../../../globals.md) / [embedding](../README.md) / TextEmbeddingFunction
|
|
|
|
# Class: `abstract` TextEmbeddingFunction<M>
|
|
|
|
an abstract class for implementing embedding functions that take text as input
|
|
|
|
## Extends
|
|
|
|
- [`EmbeddingFunction`](EmbeddingFunction.md)<`string`, `M`>
|
|
|
|
## Type Parameters
|
|
|
|
• **M** *extends* [`FunctionOptions`](../interfaces/FunctionOptions.md) = [`FunctionOptions`](../interfaces/FunctionOptions.md)
|
|
|
|
## Constructors
|
|
|
|
### new TextEmbeddingFunction()
|
|
|
|
```ts
|
|
new TextEmbeddingFunction<M>(): TextEmbeddingFunction<M>
|
|
```
|
|
|
|
#### Returns
|
|
|
|
[`TextEmbeddingFunction`](TextEmbeddingFunction.md)<`M`>
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`constructor`](EmbeddingFunction.md#constructors)
|
|
|
|
## Methods
|
|
|
|
### computeQueryEmbeddings()
|
|
|
|
```ts
|
|
computeQueryEmbeddings(data): Promise<number[] | Float32Array | Float64Array>
|
|
```
|
|
|
|
Compute the embeddings for a single query
|
|
|
|
#### Parameters
|
|
|
|
* **data**: `string`
|
|
|
|
#### Returns
|
|
|
|
`Promise`<`number`[] \| `Float32Array` \| `Float64Array`>
|
|
|
|
#### Overrides
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`computeQueryEmbeddings`](EmbeddingFunction.md#computequeryembeddings)
|
|
|
|
***
|
|
|
|
### computeSourceEmbeddings()
|
|
|
|
```ts
|
|
computeSourceEmbeddings(data): Promise<number[][] | Float32Array[] | Float64Array[]>
|
|
```
|
|
|
|
Creates a vector representation for the given values.
|
|
|
|
#### Parameters
|
|
|
|
* **data**: `string`[]
|
|
|
|
#### Returns
|
|
|
|
`Promise`<`number`[][] \| `Float32Array`[] \| `Float64Array`[]>
|
|
|
|
#### Overrides
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`computeSourceEmbeddings`](EmbeddingFunction.md#computesourceembeddings)
|
|
|
|
***
|
|
|
|
### embeddingDataType()
|
|
|
|
```ts
|
|
embeddingDataType(): Float<Floats>
|
|
```
|
|
|
|
The datatype of the embeddings
|
|
|
|
#### Returns
|
|
|
|
`Float`<`Floats`>
|
|
|
|
#### Overrides
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`embeddingDataType`](EmbeddingFunction.md#embeddingdatatype)
|
|
|
|
***
|
|
|
|
### generateEmbeddings()
|
|
|
|
```ts
|
|
abstract generateEmbeddings(texts, ...args): Promise<number[][] | Float32Array[] | Float64Array[]>
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
* **texts**: `string`[]
|
|
|
|
* ...**args**: `any`[]
|
|
|
|
#### Returns
|
|
|
|
`Promise`<`number`[][] \| `Float32Array`[] \| `Float64Array`[]>
|
|
|
|
***
|
|
|
|
### getSensitiveKeys()
|
|
|
|
```ts
|
|
protected getSensitiveKeys(): string[]
|
|
```
|
|
|
|
Provide a list of keys in the function options that should be treated as
|
|
sensitive. If users pass raw values for these keys, they will be rejected.
|
|
|
|
#### Returns
|
|
|
|
`string`[]
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`getSensitiveKeys`](EmbeddingFunction.md#getsensitivekeys)
|
|
|
|
***
|
|
|
|
### init()?
|
|
|
|
```ts
|
|
optional init(): Promise<void>
|
|
```
|
|
|
|
Optionally load any resources needed for the embedding function.
|
|
|
|
This method is called after the embedding function has been initialized
|
|
but before any embeddings are computed. It is useful for loading local models
|
|
or other resources that are needed for the embedding function to work.
|
|
|
|
#### Returns
|
|
|
|
`Promise`<`void`>
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`init`](EmbeddingFunction.md#init)
|
|
|
|
***
|
|
|
|
### ndims()
|
|
|
|
```ts
|
|
ndims(): undefined | number
|
|
```
|
|
|
|
The number of dimensions of the embeddings
|
|
|
|
#### Returns
|
|
|
|
`undefined` \| `number`
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`ndims`](EmbeddingFunction.md#ndims)
|
|
|
|
***
|
|
|
|
### resolveVariables()
|
|
|
|
```ts
|
|
protected resolveVariables(config): Partial<M>
|
|
```
|
|
|
|
Apply variables to the config.
|
|
|
|
#### Parameters
|
|
|
|
* **config**: `Partial`<`M`>
|
|
|
|
#### Returns
|
|
|
|
`Partial`<`M`>
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`resolveVariables`](EmbeddingFunction.md#resolvevariables)
|
|
|
|
***
|
|
|
|
### sourceField()
|
|
|
|
```ts
|
|
sourceField(): [DataType<Type, any>, Map<string, EmbeddingFunction<any, FunctionOptions>>]
|
|
```
|
|
|
|
sourceField is used in combination with `LanceSchema` to provide a declarative data model
|
|
|
|
#### Returns
|
|
|
|
[`DataType`<`Type`, `any`>, `Map`<`string`, [`EmbeddingFunction`](EmbeddingFunction.md)<`any`, [`FunctionOptions`](../interfaces/FunctionOptions.md)>>]
|
|
|
|
#### See
|
|
|
|
[LanceSchema](../functions/LanceSchema.md)
|
|
|
|
#### Overrides
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`sourceField`](EmbeddingFunction.md#sourcefield)
|
|
|
|
***
|
|
|
|
### toJSON()
|
|
|
|
```ts
|
|
toJSON(): Record<string, any>
|
|
```
|
|
|
|
Get the original arguments to the constructor, to serialize them so they
|
|
can be used to recreate the embedding function later.
|
|
|
|
#### Returns
|
|
|
|
`Record`<`string`, `any`>
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`toJSON`](EmbeddingFunction.md#tojson)
|
|
|
|
***
|
|
|
|
### vectorField()
|
|
|
|
```ts
|
|
vectorField(optionsOrDatatype?): [DataType<Type, any>, Map<string, EmbeddingFunction<any, FunctionOptions>>]
|
|
```
|
|
|
|
vectorField is used in combination with `LanceSchema` to provide a declarative data model
|
|
|
|
#### Parameters
|
|
|
|
* **optionsOrDatatype?**: `DataType`<`Type`, `any`> \| `Partial`<[`FieldOptions`](../interfaces/FieldOptions.md)<`DataType`<`Type`, `any`>>>
|
|
The options for the field
|
|
|
|
#### Returns
|
|
|
|
[`DataType`<`Type`, `any`>, `Map`<`string`, [`EmbeddingFunction`](EmbeddingFunction.md)<`any`, [`FunctionOptions`](../interfaces/FunctionOptions.md)>>]
|
|
|
|
#### See
|
|
|
|
[LanceSchema](../functions/LanceSchema.md)
|
|
|
|
#### Inherited from
|
|
|
|
[`EmbeddingFunction`](EmbeddingFunction.md).[`vectorField`](EmbeddingFunction.md#vectorfield)
|