Files
lancedb/docs/web/integrations/ai/genkit.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

82 lines
3.2 KiB
Plaintext

---
title: "GenKit"
sidebarTitle: "GenKit"
---
import {
TsFrameworksGenkitCustomIndexer,
TsFrameworksGenkitCustomRetriever,
TsFrameworksGenkitUsage,
} from '/snippets/integrations.mdx';
### genkitx-lancedb {#genkitx-lancedb}
Genkit is an open-source framework for building end-to-end AI and RAG pipelines with a clean, TypeScript-first
developer experience. The genkitx-lancedb plugin lets you use LanceDB as a high-performance vector store
inside your Genkit flows, so you can index, search, and retrieve data efficiently as part of your AI
applications.
### Installation {#installation}
<CodeGroup>
```bash Bash icon="terminal"
pnpm install genkitx-lancedb
```
</CodeGroup>
### Usage {#usage}
Adding LanceDB plugin to your genkit instance.
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsFrameworksGenkitUsage}
</CodeBlock>
You can run this app with the following command:
<CodeGroup>
```bash Bash icon="terminal"
genkit start -- tsx --watch src/index.ts
```
</CodeGroup>
This'll add LanceDB as a retriever and indexer to the genkit instance. You can see it in the GUI view
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/e752f7f4-785b-4797-a11e-72ab06a531b7" />
### Testing retrieval on a sample table {#testing-retrieval-on-a-sample-table}
To see the raw retrieval results, open the LanceDB retriever in the Genkit Developer UI,
enter a query and run it:
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/b8d356ed-8421-4790-8fc0-d6af563b9657" />
The retriever returns 5 results from the LanceDB table. Each result looks something like this:
<img width="1417" alt="Screenshot 2025-05-11 at 7 21 18 PM" src="https://github.com/user-attachments/assets/77429525-36e2-4da6-a694-e58c1cf9eb83" />
## Creating a custom RAG flow {#creating-a-custom-rag-flow}
Now that we've seen how you can use LanceDB in a Genkit pipeline, let's refine the flow and create a RAG. A RAG flow will consist of an index and a retriever with its outputs postprocessed and fed into an LLM for final response
### Creating custom indexer flows {#creating-custom-indexer-flows}
You can also create custom indexer flows, using more options and features provided by LanceDB.
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsFrameworksGenkitCustomIndexer}
</CodeBlock>
<img width="1316" alt="Screenshot 2025-05-11 at 8 35 56 PM" src="https://github.com/user-attachments/assets/e2a20ce4-d1d0-4fa2-9a84-f2cc26e3a29f" />
In your console, you can see the logs
<img width="511" alt="Screenshot 2025-05-11 at 7 19 14 PM" src="https://github.com/user-attachments/assets/243f26c5-ed38-40b6-b661-002f40f0423a" />
### Creating custom retriever flows {#creating-custom-retriever-flows}
You can also create custom retriever flows, using more options and features provided by LanceDB.
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsFrameworksGenkitCustomRetriever}
</CodeBlock>
Now using our retrieval flow, we can ask a question about the ingested PDF
<img width="1306" alt="Screenshot 2025-05-11 at 7 18 45 PM" src="https://github.com/user-attachments/assets/86c66b13-7c12-4d5f-9d81-ae36bfb1c346" />