7ac5f74c80
BREAKING CHANGE: embedding function implementations in Node need to now call `resolveVariables()` in their constructors and should **not** implement `toJSON()`. This tries to address the handling of secrets. In Node, they are currently lost. In Python, they are currently leaked into the table schema metadata. This PR introduces an in-memory variable store on the function registry. It also allows embedding function definitions to label certain config values as "sensitive", and the preprocessing logic will raise an error if users try to pass in hard-coded values. Closes #2110 Closes #521 --------- Co-authored-by: Weston Pace <weston.pace@gmail.com>
3.5 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / embedding / EmbeddingFunctionRegistry
Class: EmbeddingFunctionRegistry
This is a singleton class used to register embedding functions and fetch them by name. It also handles serializing and deserializing. You can implement your own embedding function by subclassing EmbeddingFunction or TextEmbeddingFunction and registering it with the registry
Constructors
new EmbeddingFunctionRegistry()
new EmbeddingFunctionRegistry(): EmbeddingFunctionRegistry
Returns
Methods
functionToMetadata()
functionToMetadata(conf): Record<string, any>
Parameters
- conf:
EmbeddingFunctionConfig
Returns
Record<string, any>
get()
get<T>(name): undefined | EmbeddingFunctionCreate<T>
Fetch an embedding function by name
Type Parameters
• T extends EmbeddingFunction<unknown, FunctionOptions>
Parameters
- name:
stringThe name of the function
Returns
undefined | EmbeddingFunctionCreate<T>
getTableMetadata()
getTableMetadata(functions): Map<string, string>
Parameters
- functions:
EmbeddingFunctionConfig[]
Returns
Map<string, string>
getVar()
getVar(name): undefined | string
Get a variable.
Parameters
- name:
string
Returns
undefined | string
See
length()
length(): number
Get the number of registered functions
Returns
number
register()
register<T>(this, alias?): (ctor) => any
Register an embedding function
Type Parameters
• T extends EmbeddingFunctionConstructor<EmbeddingFunction<any, FunctionOptions>> = EmbeddingFunctionConstructor<EmbeddingFunction<any, FunctionOptions>>
Parameters
-
alias?:
string
Returns
Function
Parameters
- ctor:
T
Returns
any
Throws
Error if the function is already registered
reset()
reset(this): void
reset the registry to the initial state
Parameters
Returns
void
setVar()
setVar(name, value): void
Set a variable. These can be accessed in the embedding function
configuration using the syntax $var:variable_name. If they are not
set, an error will be thrown letting you know which key is unset. If you
want to supply a default value, you can add an additional part in the
configuration like so: $var:variable_name:default_value. Default values
can be used for runtime configurations that are not sensitive, such as
whether to use a GPU for inference.
The name must not contain colons. The default value can contain colons.
Parameters
-
name:
string -
value:
string
Returns
void