lancedb/docs/src/embeddings/embedding_functions.md

Representing multi-modal data as vector embeddings is becoming a standard practice. Embedding functions can themselves be thought of as key part of the data processing pipeline that each request has to be passed through. The assumption here is: after initial setup, these components and the underlying methodology are not expected to change for a particular project.

For this purpose, LanceDB introduces an embedding functions API, that allow you simply set up once, during the configuration stage of your project. After this, the table remembers it, effectively making the embedding functions disappear in the background so you don't have to worry about manually passing callables, and instead, simply focus on the rest of your data engineering pipeline.

!!! warning Using the implicit embeddings management approach means that you can forget about the manually passing around embedding functions in your code, as long as you don't intend to change it at a later time. If your embedding function changes, you'll have to re-configure your table with the new embedding function and regenerate the embeddings.

1. Define the embedding function

We have some pre-defined embedding functions in the global registry, with more coming soon. Here's let's an implementation of CLIP as example.

registry = EmbeddingFunctionRegistry.get_instance()
clip = registry.get("open-clip").create()

You can also define your own embedding function by implementing the EmbeddingFunction abstract base interface. It subclasses Pydantic Model which can be utilized to write complex schemas simply as we'll see next!

2. Define the data model or schema

The embedding function defined above abstracts away all the details about the models and dimensions required to define the schema. You can simply set a field as source or vector column. Here's how:

class Pets(LanceModel):
    vector: Vector(clip.ndims) = clip.VectorField()
    image_uri: str = clip.SourceField()

VectorField tells LanceDB to use the clip embedding function to generate query embeddings for the vector column and SourceField ensures that when adding data, we automatically use the specified embedding function to encode image_uri.

3. Create LanceDB table

Now that we have chosen/defined our embedding function and the schema, we can create the table:

db = lancedb.connect("~/lancedb")
table = db.create_table("pets", schema=Pets)

That's it! We've provided all the information needed to embed the source and query inputs. We can now forget about the model and dimension details and start to build our VectorDB pipeline.

4. Ingest lots of data and query your table

Any new or incoming data can just be added and it'll be vectorized automatically.

table.add([{"image_uri": u} for u in uris])

Our OpenCLIP query embedding function supports querying via both text and images:

result = table.search("dog")

Let's query an image:

p = Path("path/to/images/samoyed_100.jpg")
query_image = Image.open(p)
table.search(query_image)

---

Rate limit Handling

EmbeddingFunction class wraps the calls for source and query embedding generation inside a rate limit handler that retries the requests with exponential backoff after successive failures. By default, the maximum retires is set to 7. You can tune it by setting it to a different number, or disable it by setting it to 0.

An example of how to do this is shown below:

clip = registry.get("open-clip").create() # Defaults to 7 max retries
clip = registry.get("open-clip").create(max_retries=10) # Increase max retries to 10
clip = registry.get("open-clip").create(max_retries=0) # Retries disabled

!!! note Embedding functions can also fail due to other errors that have nothing to do with rate limits. This is why the error is also logged.

Some fun with Pydantic

LanceDB is integrated with Pydantic, which was used in the example above to define the schema in Python. It's also used behind the scenes by the embedding function API to ingest useful information as table metadata.

You can also use the integration for adding utility operations in the schema. For example, in our multi-modal example, you can search images using text or another image. Let's define a utility function to plot the image.

class Pets(LanceModel):
    vector: Vector(clip.ndims) = clip.VectorField()
    image_uri: str = clip.SourceField()

    @property
    def image(self):
        return Image.open(self.image_uri)

Now, you can covert your search results to a Pydantic model and use this property.

rs = table.search(query_image).limit(3).to_pydantic(Pets)
rs[2].image

Now that you have the basic idea about implicit management via embedding functions, let's dive deeper into a custom API that you can use to implement your own embedding functions.