mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
Jack YeandBrendan Clement
47b7babc08
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
72 lines
3.4 KiB
Plaintext
72 lines
3.4 KiB
Plaintext
---
|
|
title: "Multimodal Agent"
|
|
sidebarTitle: "Multimodal Agent"
|
|
description: "Build an AI agent that understands both text and images to help users find recipes using LanceDB and PydanticAI"
|
|
---
|
|
|
|
|
|
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
|
|
|
|
Ever wanted to combine the power of text and images in a single AI agent? In this tutorial,
|
|
you'll build an agent that can understand both text and images to help users discover recipes that are relevant to them. The approach shown combines LanceDB's multimodal capabilities with [Pydantic AI](https://ai.pydantic.dev/) for the agentic workflow.
|
|
|
|
## Key Technologies {#key-technologies}
|
|
|
|
- **LanceDB**: Embedded retrieval library and multimodal lakehouse for efficient storage and retrieval
|
|
- **PydanticAI**: Modern AI agent framework with type safety
|
|
- **Sentence Transformers**: Text embeddings for semantic search
|
|
- **CLIP**: Vision-language model for image understanding
|
|
- **Streamlit**: Interactive web application framework
|
|
|
|
## Tutorial Overview {#tutorial-overview}
|
|
|
|
### Option 1: Notebook {#option-1-notebook}
|
|
The notebook shows how to work through the steps and prepare a small sample recipe dataset, generate both text and image
|
|
embeddings, store everything efficiently in LanceDB, and then build a PydanticAI agent with custom tools to
|
|
query it. You'll finish by testing the agent against a few example questions to see the full multimodal flow
|
|
end to end.
|
|
|
|
<Card title="Notebook" href="https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing">
|
|
This simple tutorial provides a step-by-step workflow with a small demo dataset of 4 examples.
|
|
No local setup required - just click and start learning about multimodal agents.
|
|
[](https://colab.research.google.com/drive/1pxavAGoXa-KSh_4HxNpvP2AjHPcIRpbq?usp=sharing)
|
|
</Card>
|
|
|
|
### Option 2: Demo Application (Local Setup) {#option-2-demo-application-local-setup}
|
|
The demo application is the full codebase: you'll download and process a real recipe dataset with thousands
|
|
of items, run a Streamlit chat interface that supports image upload, and follow a structure that includes
|
|
production-minded touches like error handling, logging, and monitoring. Everything you need to deploy is
|
|
included.
|
|
|
|
<Card icon="download" title="Download Tutorial Files" href="https://github.com/lancedb/vectordb-recipes/tree/main/applications/multimodal-recipe-agent">
|
|
Download the files for the full demo application here.
|
|
</Card>
|
|
|
|
### Dataset Information {#dataset-information}
|
|
- **Source**: [Kaggle Recipe Dataset](https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images)
|
|
- **Size**: Thousands of recipes with images
|
|
- **Format**: CSV file with recipe data and image references
|
|
|
|
### Setup {#setup}
|
|
|
|
<CodeGroup>
|
|
```bash bash icon="code"
|
|
# 1. Extract the downloaded files to a folder
|
|
# 2. Navigate to the folder in terminal
|
|
cd multimodal-recipe-agent
|
|
|
|
# 3. Install dependencies with uv
|
|
uv sync
|
|
|
|
# 4. Download the Kaggle dataset
|
|
# Visit: https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images
|
|
# Extract recipes.csv to the data/ folder
|
|
|
|
# 5. Import the dataset
|
|
uv run python import.py
|
|
|
|
# 6. Run the complete Streamlit chat application
|
|
uv run streamlit run app.py
|
|
```
|
|
</CodeGroup>
|