mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-18 16:43:59 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
82 lines
3.2 KiB
Plaintext
82 lines
3.2 KiB
Plaintext
---
|
|
title: "GenKit"
|
|
sidebarTitle: "GenKit"
|
|
|
|
---
|
|
|
|
import {
|
|
TsFrameworksGenkitCustomIndexer,
|
|
TsFrameworksGenkitCustomRetriever,
|
|
TsFrameworksGenkitUsage,
|
|
} from '/snippets/integrations.mdx';
|
|
|
|
### genkitx-lancedb {#genkitx-lancedb}
|
|
Genkit is an open-source framework for building end-to-end AI and RAG pipelines with a clean, TypeScript-first
|
|
developer experience. The genkitx-lancedb plugin lets you use LanceDB as a high-performance vector store
|
|
inside your Genkit flows, so you can index, search, and retrieve data efficiently as part of your AI
|
|
applications.
|
|
|
|
### Installation {#installation}
|
|
<CodeGroup>
|
|
```bash Bash icon="terminal"
|
|
pnpm install genkitx-lancedb
|
|
```
|
|
</CodeGroup>
|
|
|
|
### Usage {#usage}
|
|
|
|
Adding LanceDB plugin to your genkit instance.
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsFrameworksGenkitUsage}
|
|
</CodeBlock>
|
|
|
|
You can run this app with the following command:
|
|
<CodeGroup>
|
|
```bash Bash icon="terminal"
|
|
genkit start -- tsx --watch src/index.ts
|
|
```
|
|
</CodeGroup>
|
|
|
|
This'll add LanceDB as a retriever and indexer to the genkit instance. You can see it in the GUI view
|
|
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/e752f7f4-785b-4797-a11e-72ab06a531b7" />
|
|
|
|
### Testing retrieval on a sample table {#testing-retrieval-on-a-sample-table}
|
|
|
|
To see the raw retrieval results, open the LanceDB retriever in the Genkit Developer UI,
|
|
enter a query and run it:
|
|
|
|
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/b8d356ed-8421-4790-8fc0-d6af563b9657" />
|
|
|
|
The retriever returns 5 results from the LanceDB table. Each result looks something like this:
|
|
|
|
<img width="1417" alt="Screenshot 2025-05-11 at 7 21 18 PM" src="https://github.com/user-attachments/assets/77429525-36e2-4da6-a694-e58c1cf9eb83" />
|
|
|
|
|
|
|
|
## Creating a custom RAG flow {#creating-a-custom-rag-flow}
|
|
|
|
Now that we've seen how you can use LanceDB in a Genkit pipeline, let's refine the flow and create a RAG. A RAG flow will consist of an index and a retriever with its outputs postprocessed and fed into an LLM for final response
|
|
|
|
### Creating custom indexer flows {#creating-custom-indexer-flows}
|
|
You can also create custom indexer flows, using more options and features provided by LanceDB.
|
|
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsFrameworksGenkitCustomIndexer}
|
|
</CodeBlock>
|
|
|
|
<img width="1316" alt="Screenshot 2025-05-11 at 8 35 56 PM" src="https://github.com/user-attachments/assets/e2a20ce4-d1d0-4fa2-9a84-f2cc26e3a29f" />
|
|
|
|
In your console, you can see the logs
|
|
|
|
<img width="511" alt="Screenshot 2025-05-11 at 7 19 14 PM" src="https://github.com/user-attachments/assets/243f26c5-ed38-40b6-b661-002f40f0423a" />
|
|
|
|
### Creating custom retriever flows {#creating-custom-retriever-flows}
|
|
You can also create custom retriever flows, using more options and features provided by LanceDB.
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsFrameworksGenkitCustomRetriever}
|
|
</CodeBlock>
|
|
Now using our retrieval flow, we can ask a question about the ingested PDF
|
|
<img width="1306" alt="Screenshot 2025-05-11 at 7 18 45 PM" src="https://github.com/user-attachments/assets/86c66b13-7c12-4d5f-9d81-ae36bfb1c346" />
|
|
|