mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
Jack Ye
b21fb2a4dc
The pages describing this code lived in lancedb/docs, a repository with ten stars that neither contributing guide mentioned. A user who hit a wrong page while reading the source had to find a separate, near-invisible repository to fix it. These 102 pages now sit beside the code, and a documentation fix is a pull request in the repository the reader already had open. `docs/web` is a complete Mintlify site: `cd docs/web && mint dev` serves it with no other checkout. Enterprise pages stay private and are merged in when the published site is assembled, so links into them do not resolve here — every open-source page does. `docs/web-tests` holds the tests the code examples are extracted from. They ran against a published wheel before; here they can run against this repository's own SDK, so an example that stops compiling fails the pull request that broke it rather than being noticed later. Both directories sit beside `docs/src` and are invisible to it: mkdocs reads `docs_dir: src`, so the SDK reference builds and deploys exactly as before.
82 lines
3.7 KiB
Plaintext
82 lines
3.7 KiB
Plaintext
---
|
|
title: "Meta Llama Synthetic Data Kit"
|
|
sidebarTitle: "Synthetic Data Kit"
|
|
description: "Use Meta Llama's Synthetic Data Kit with LanceDB to generate high-quality synthetic datasets for LLM fine-tuning and training."
|
|
|
|
---
|
|
|
|
|
|
|
|
[Synthetic Data Kit](https://github.com/meta-llama/synthetic-data-kit) is a tool from Meta LLAMA that helps you generate high-quality synthetic datasets for fine-tuning large language models (LLMs). It simplifies the process of preparing data for fine-tuning by providing a command-line interface (CLI) with a modular four-command flow.
|
|
|
|
One of the key features of the `synthetic-data-kit` is its use of the Lance format for storing and ingesting datasets. This allows for efficient storage and retrieval of data, which is crucial when working with large datasets.
|
|
|
|
### Key Features: {#key-features}
|
|
|
|
* **Data Ingestion:** The toolkit can ingest various file formats, including PDF, HTML, YouTube transcripts, DOCX, PPT, and TXT.
|
|
* **Fine-tuning Format Creation:** It can create different fine-tuning formats, such as question-answer (QA) pairs, QA pairs with Chain-of-Thought (CoT), and summarization formats.
|
|
* **Data Curation:** The tool uses Llama as a judge to curate high-quality examples, ensuring the quality of the generated dataset.
|
|
* **Flexible Saving Options:** You can save the generated datasets in various formats compatible with your fine-tuning workflow, including Hugging Face, JSONL, and JSON.
|
|
|
|
### How it Works: {#how-it-works}
|
|
|
|
The synthetic-data-kit follows a simple four-step process:
|
|
|
|
1. **Ingest:** Import your input files into the toolkit. The data is stored in the Lance format for efficient processing.
|
|
2. **Create:** Generate diverse fine-tuning datasets, such as reasoning, summarization, and QA pairs, from the ingested documents.
|
|
3. **Curate:** Use Llama to filter and select high-quality examples from the generated dataset.
|
|
4. **Save-as:** Export the curated dataset in your preferred format.
|
|
|
|
### Usage {#usage}
|
|
|
|
The `synthetic-data-kit` uses Lance format to store and manage the data that you ingest. The workflow is a series of commands that build on each other, starting with the `ingest` command.
|
|
|
|
Here is an example of the end-to-end workflow:
|
|
|
|
1. **Ingest Data into a LanceDB dataset**
|
|
|
|
This command takes a directory of source files and creates a LanceDB dataset from them.
|
|
|
|
```bash
|
|
synthetic-data-kit ingest docs/report.pdf --multimodal
|
|
# This will create a Lance dataset at data/parsed/report.lance
|
|
# with 'text' and 'image' columns.
|
|
|
|
#Generate multimodal-qa pairs from the ingested data
|
|
synthetic-data-kit create data/parsed/report.lance --type multimodal-qa
|
|
```
|
|
|
|
2. **Create fine-tuning data**
|
|
|
|
This command uses the LanceDB dataset created in the previous step to generate synthetic data in the desired format.
|
|
|
|
```bash
|
|
synthetic_data create data/parsed/report.lance
|
|
```
|
|
|
|
3. **Curate the data**
|
|
|
|
This step uses a language model to curate the generated data and ensure its quality.
|
|
|
|
```bash
|
|
synthetic_data curate report.json
|
|
```
|
|
|
|
4. **Save the final dataset**
|
|
|
|
Finally, save the curated data to a file in the desired format.
|
|
|
|
```bash
|
|
synthetic_data save-as report.json --save_path ./my_finetuning_data.jsonl
|
|
```
|
|
|
|
This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable.
|
|
|
|
### Getting Started: {#getting-started}
|
|
|
|
To get started with the synthetic-data-kit, you can clone the [GitHub Repository](https://github.com/meta-llama/synthetic-data-kit) and install the necessary dependencies.
|
|
|
|
> **Note:** You will also need access to a Llama model, either running locally or via a hosted API.
|
|
|
|
|