mirror of
https://github.com/lancedb/lancedb.git
synced 2026-01-07 20:32:59 +00:00
119b928a52
This PR makes incremental changes to the documentation. * Closes #697 * Closes #698 ## Chores - [x] Add dark mode - [x] Fix headers in navbar - [x] Add `extra.css` to customize navbar styles - [x] Customize fonts for prose/code blocks, navbar and admonitions - [x] Inspect all admonition boxes (remove redundant dropdowns) and improve clarity and readability - [x] Ensure that all images in the docs have white background (not transparent) to be viewable in dark mode - [x] Improve code formatting in code blocks to make them consistent with autoformatters (eslint/ruff) - [x] Add bolder weight to h1 headers - [x] Add diagram showing the difference between embedded (OSS) and serverless (Cloud) - [x] Fix [Creating an empty table](https://lancedb.github.io/lancedb/guides/tables/#creating-empty-table) section: right now, the subheaders are not clickable. - [x] In critical data ingestion methods like `table.add` (among others), the type signature often does not match the actual code - [x] Proof-read each documentation section and rewrite as necessary to provide more context, use cases, and explanations so it reads less like reference documentation. This is especially important for CRUD and search sections since those are so central to the user experience. ## Restructure/new content - [x] The section for [Adding data](https://lancedb.github.io/lancedb/guides/tables/#adding-to-a-table) only shows examples for pandas and iterables. We should include pydantic models, arrow tables, etc. - [x] Add conceptual tutorial for IVF-PQ index - [x] Clearly separate vector search, FTS and filtering sections so that these are easier to find - [x] Add docs on refine factor to explain its importance for recall. Closes #716 - [x] Add an FAQ page showing answers to commonly asked questions about LanceDB. Closes #746 - [x] Add simple polars example to the integrations section. Closes #756 and closes #153 - [ ] Add basic docs for the Rust API (more detailed API docs can come later). Closes #781 - [x] Add a section on the various storage options on local vs. cloud (S3, EBS, EFS, local disk, etc.) and the tradeoffs involved. Closes #782 - [x] Revamp filtering docs: add pre-filtering examples and redo headers and update content for SQL filters. Closes #783 and closes #784. - [x] Add docs for data management: compaction, cleaning up old versions and incremental indexing. Closes #785 - [ ] Add a benchmark section that also discusses some best practices. Closes #787 --------- Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com> Co-authored-by: Will Jones <willjones127@gmail.com>
1190 lines
37 KiB
Plaintext
1190 lines
37 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c0de1e6a-61f7-4f99-a2fd-1461902ab36a",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Versioning & Reproducibility\n",
|
||
"\n",
|
||
"Reproducibility is critical for AI. For code, it's easy to keep track of changes using Github or Gitlab.\n",
|
||
"For data, it's not as easy. Most of the time, we're manually writing complicated data tracking code, wrestling with an external tool, and dealing with expensive duplicate snapshot copies with low granularity.\n",
|
||
"\n",
|
||
"While working with most other vector databases, if we loaded in the wrong data (or any other such mistakes), we have to blow away the index, correct the mistake, and then completely rebuild it. It's **really difficult** to rollback to an earlier state, and any such corrective action **destroys historical data and evidence**, which may be useful down the line to debug and diagnose issues.\n",
|
||
"\n",
|
||
"To our knowledge, LanceDB is the first and only vector database that supports full reproducibility and rollbacks natively.\n",
|
||
"Taking advantage of the Lance columnar data format, LanceDB supports:\n",
|
||
"- Automatic versioning\n",
|
||
"- Instant rollback\n",
|
||
"- Appends, updates, deletions\n",
|
||
"- Schema evolution\n",
|
||
"\n",
|
||
"This makes auditing, tracking, and reproducibility a breeze!\n",
|
||
"\n",
|
||
"Let's see how this all works."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cafebbce-d324-485d-90ec-503695875f47",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Pickle Rick!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0e74818f-109e-4b09-b5f8-dd1875c512e3",
|
||
"metadata": {},
|
||
"source": [
|
||
"We'll start with a local LanceDB connection"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "1f57d988-56b9-4384-8a7b-000d5f91034a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import lancedb\n",
|
||
"db = lancedb.connect(\"~/.lancedb\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9c4c443d-2f14-455d-b766-bacbaad43d20",
|
||
"metadata": {},
|
||
"source": [
|
||
"We've got a CSV file with a bunch of quotes from Rick and Morty"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "08556aeb-6bdc-451c-99f5-163374fdec55",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"id,quote,author\n",
|
||
"1,\"Nobody exists on purpose. Nobody belongs anywhere.\",Morty\n",
|
||
"2,\"We're all going to die. Come watch TV.\",Morty\n",
|
||
"3,\"Losers look stuff up while the rest of us are carpin' all them diems.\",Summer\n",
|
||
"4,\"He's not a hot girl. He can't just bail on his life and set up shop in someone else's.\",Beth\n",
|
||
"5,\"When you are an a—hole, it doesn't matter how right you are. Nobody wants to give you the satisfaction.\",Morty\n",
|
||
"6,\"God's turning people into insect monsters, Beth. I'm the one beating them to death. Thank me.\",Jerry\n",
|
||
"7,\"Camping is just being homeless without the change.\",Summer\n",
|
||
"8,\"This seems like a good time for a drink and a cold, calculated speech with sinister overtones. A speech about politics, about order, brotherhood, power ... but speeches are for campaigning. Now is the time for action.\",Morty\n",
|
||
"9,\"Having a family doesn't mean that you stop being an individual. You know the best thing you can do for the people that depend on you? Be honest with them, even if it means setting them free.\",Mr. Meeseeks\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!head rick_and_morty_quotes.csv"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a5fcdcda-b0fe-4ac4-90b4-6b42cf2ef34d",
|
||
"metadata": {},
|
||
"source": [
|
||
"Let's load this into a pandas dataframe.\n",
|
||
"\n",
|
||
"It's got 3 columns, a quote id, the quote string, and the first name of the author of the quote:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "def3ae59-77d9-43f0-ba6d-415a1503856b",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>id</th>\n",
|
||
" <th>quote</th>\n",
|
||
" <th>author</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>1</td>\n",
|
||
" <td>Nobody exists on purpose. Nobody belongs anywh...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>2</td>\n",
|
||
" <td>We're all going to die. Come watch TV.</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>3</td>\n",
|
||
" <td>Losers look stuff up while the rest of us are ...</td>\n",
|
||
" <td>Summer</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>4</td>\n",
|
||
" <td>He's not a hot girl. He can't just bail on his...</td>\n",
|
||
" <td>Beth</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>5</td>\n",
|
||
" <td>When you are an a—hole, it doesn't matter how ...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" id quote author\n",
|
||
"0 1 Nobody exists on purpose. Nobody belongs anywh... Morty\n",
|
||
"1 2 We're all going to die. Come watch TV. Morty\n",
|
||
"2 3 Losers look stuff up while the rest of us are ... Summer\n",
|
||
"3 4 He's not a hot girl. He can't just bail on his... Beth\n",
|
||
"4 5 When you are an a—hole, it doesn't matter how ... Morty"
|
||
]
|
||
},
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"import pandas as pd\n",
|
||
"df = pd.read_csv(\"rick_and_morty_quotes.csv\")\n",
|
||
"df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4ba9ffac-c779-49e3-91a7-f1c00f3fda41",
|
||
"metadata": {},
|
||
"source": [
|
||
"Creating a LanceDB table from a pandas dataframe is straightforward using `create_table`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "bd981f6d-b921-4b1d-b63a-6c1d59f3a51d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>id</th>\n",
|
||
" <th>quote</th>\n",
|
||
" <th>author</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>1</td>\n",
|
||
" <td>Nobody exists on purpose. Nobody belongs anywh...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>2</td>\n",
|
||
" <td>We're all going to die. Come watch TV.</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>3</td>\n",
|
||
" <td>Losers look stuff up while the rest of us are ...</td>\n",
|
||
" <td>Summer</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>4</td>\n",
|
||
" <td>He's not a hot girl. He can't just bail on his...</td>\n",
|
||
" <td>Beth</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>5</td>\n",
|
||
" <td>When you are an a—hole, it doesn't matter how ...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" id quote author\n",
|
||
"0 1 Nobody exists on purpose. Nobody belongs anywh... Morty\n",
|
||
"1 2 We're all going to die. Come watch TV. Morty\n",
|
||
"2 3 Losers look stuff up while the rest of us are ... Summer\n",
|
||
"3 4 He's not a hot girl. He can't just bail on his... Beth\n",
|
||
"4 5 When you are an a—hole, it doesn't matter how ... Morty"
|
||
]
|
||
},
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"db.drop_table(\"rick_and_morty\", ignore_missing=True)\n",
|
||
"table = db.create_table(\"rick_and_morty\", df)\n",
|
||
"table.head().to_pandas()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "38d055be-ae3e-4190-b1cf-abf14cdf8975",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Updates"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "842550fb-da81-44ea-9e98-d5dbaa6916c7",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now, since Rick is the smartest man in the multiverse, he deserves to have his quotes attributed to his full name: Richard Daniel Sanchez.\n",
|
||
"\n",
|
||
"This can be done via `LanceTable.update`. It needs two arguments:\n",
|
||
"\n",
|
||
"1. A `where` string filter (sql syntax) to determine the rows to update\n",
|
||
"2. A dict of `values` where the keys are the column names to update and the values are the new values"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "9eac4708-a8c4-49aa-bc13-8e60c5bf34a0",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>id</th>\n",
|
||
" <th>quote</th>\n",
|
||
" <th>author</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>1</td>\n",
|
||
" <td>Nobody exists on purpose. Nobody belongs anywh...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>2</td>\n",
|
||
" <td>We're all going to die. Come watch TV.</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>3</td>\n",
|
||
" <td>Losers look stuff up while the rest of us are ...</td>\n",
|
||
" <td>Summer</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>4</td>\n",
|
||
" <td>He's not a hot girl. He can't just bail on his...</td>\n",
|
||
" <td>Beth</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>5</td>\n",
|
||
" <td>When you are an a—hole, it doesn't matter how ...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>...</th>\n",
|
||
" <td>...</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>56</th>\n",
|
||
" <td>57</td>\n",
|
||
" <td>If I let you make me nervous, then we can't ge...</td>\n",
|
||
" <td>Richard Daniel Sanchez</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>57</th>\n",
|
||
" <td>58</td>\n",
|
||
" <td>Oh, boy, so you actually learned something tod...</td>\n",
|
||
" <td>Richard Daniel Sanchez</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>58</th>\n",
|
||
" <td>59</td>\n",
|
||
" <td>I can't abide bureaucracy. I don't like being ...</td>\n",
|
||
" <td>Richard Daniel Sanchez</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>59</th>\n",
|
||
" <td>60</td>\n",
|
||
" <td>I think you have to think ahead and live in th...</td>\n",
|
||
" <td>Richard Daniel Sanchez</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>60</th>\n",
|
||
" <td>61</td>\n",
|
||
" <td>I know that new situations can be intimidating...</td>\n",
|
||
" <td>Richard Daniel Sanchez</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"<p>61 rows × 3 columns</p>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" id quote \\\n",
|
||
"0 1 Nobody exists on purpose. Nobody belongs anywh... \n",
|
||
"1 2 We're all going to die. Come watch TV. \n",
|
||
"2 3 Losers look stuff up while the rest of us are ... \n",
|
||
"3 4 He's not a hot girl. He can't just bail on his... \n",
|
||
"4 5 When you are an a—hole, it doesn't matter how ... \n",
|
||
".. .. ... \n",
|
||
"56 57 If I let you make me nervous, then we can't ge... \n",
|
||
"57 58 Oh, boy, so you actually learned something tod... \n",
|
||
"58 59 I can't abide bureaucracy. I don't like being ... \n",
|
||
"59 60 I think you have to think ahead and live in th... \n",
|
||
"60 61 I know that new situations can be intimidating... \n",
|
||
"\n",
|
||
" author \n",
|
||
"0 Morty \n",
|
||
"1 Morty \n",
|
||
"2 Summer \n",
|
||
"3 Beth \n",
|
||
"4 Morty \n",
|
||
".. ... \n",
|
||
"56 Richard Daniel Sanchez \n",
|
||
"57 Richard Daniel Sanchez \n",
|
||
"58 Richard Daniel Sanchez \n",
|
||
"59 Richard Daniel Sanchez \n",
|
||
"60 Richard Daniel Sanchez \n",
|
||
"\n",
|
||
"[61 rows x 3 columns]"
|
||
]
|
||
},
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.update(where=\"author='Rick'\", values={\"author\": \"Richard Daniel Sanchez\"})\n",
|
||
"table.to_pandas()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ac6499ce-af6d-4934-9051-be5f159ce623",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Schema evolution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0402226b-6d0c-41c5-9257-069c4bf16825",
|
||
"metadata": {},
|
||
"source": [
|
||
"Ok so this is a vector database, so we need actual vectors.\n",
|
||
"We'll use sentence transformers here to avoid having to deal with api keys and all that."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "85db4ed9-8f80-4b56-9867-1381fa1c4c7d",
|
||
"metadata": {},
|
||
"source": [
|
||
"Let's create a basic model using the \"all-MiniLM-L6-v2\" model and embed the quotes"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "998f4eb5-31cd-49ae-9f7c-2ec4d6652ef6",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/ayush/Documents/lancedb/env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
||
" from .autonotebook import tqdm as notebook_tqdm\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from sentence_transformers import SentenceTransformer\n",
|
||
"model = SentenceTransformer(\"all-MiniLM-L6-v2\", device=\"cpu\")\n",
|
||
"vectors = model.encode(df.quote.values.tolist(),\n",
|
||
" convert_to_numpy=True,\n",
|
||
" normalize_embeddings=True).tolist()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "539e2a0e-529b-439b-ba8c-a388907c4860",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can then convert the vectors into a pyarrow Table and merge it to the LanceDB Table.\n",
|
||
"\n",
|
||
"For the merge to work successfully, we need to have an overlapping column. Here the natural choice is to use the id column"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "ccbea593-85cf-484c-989f-9836a31c7906",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from lance.vector import vec_to_table\n",
|
||
"import numpy as np\n",
|
||
"import pyarrow as pa"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "727c8230-7e41-436a-8666-60ee46e7041b",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>vector</th>\n",
|
||
" <th>id</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>[0.044295236, -0.0831885, -0.03597761, -0.0396...</td>\n",
|
||
" <td>1</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>[0.057405394, -0.09669633, 0.00515391, -0.0213...</td>\n",
|
||
" <td>2</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>[0.057896998, -0.033441037, 0.01376669, -0.015...</td>\n",
|
||
" <td>3</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>[0.038649295, 0.01286428, -0.03261163, 0.01939...</td>\n",
|
||
" <td>4</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>[0.07633445, 0.03451182, -0.0037649637, 0.0203...</td>\n",
|
||
" <td>5</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" vector id\n",
|
||
"0 [0.044295236, -0.0831885, -0.03597761, -0.0396... 1\n",
|
||
"1 [0.057405394, -0.09669633, 0.00515391, -0.0213... 2\n",
|
||
"2 [0.057896998, -0.033441037, 0.01376669, -0.015... 3\n",
|
||
"3 [0.038649295, 0.01286428, -0.03261163, 0.01939... 4\n",
|
||
"4 [0.07633445, 0.03451182, -0.0037649637, 0.0203... 5"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"embeddings = vec_to_table(vectors)\n",
|
||
"embeddings = embeddings.append_column(\"id\", pa.array(np.arange(len(table))+1))\n",
|
||
"embeddings.to_pandas().head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "518da48d-6481-4c1e-8ba4-800d5e0542cf",
|
||
"metadata": {},
|
||
"source": [
|
||
"And now we'll use the `LanceTable.merge` function to add the vector column into the LanceTable."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "a4326a70-9863-47e8-8f3f-565e35d558cf",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>id</th>\n",
|
||
" <th>quote</th>\n",
|
||
" <th>author</th>\n",
|
||
" <th>vector</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>1</td>\n",
|
||
" <td>Nobody exists on purpose. Nobody belongs anywh...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" <td>[0.044295236, -0.0831885, -0.03597761, -0.0396...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>2</td>\n",
|
||
" <td>We're all going to die. Come watch TV.</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" <td>[0.057405394, -0.09669633, 0.00515391, -0.0213...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>3</td>\n",
|
||
" <td>Losers look stuff up while the rest of us are ...</td>\n",
|
||
" <td>Summer</td>\n",
|
||
" <td>[0.057896998, -0.033441037, 0.01376669, -0.015...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>4</td>\n",
|
||
" <td>He's not a hot girl. He can't just bail on his...</td>\n",
|
||
" <td>Beth</td>\n",
|
||
" <td>[0.038649295, 0.01286428, -0.03261163, 0.01939...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>5</td>\n",
|
||
" <td>When you are an a—hole, it doesn't matter how ...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" <td>[0.07633445, 0.03451182, -0.0037649637, 0.0203...</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" id quote author \\\n",
|
||
"0 1 Nobody exists on purpose. Nobody belongs anywh... Morty \n",
|
||
"1 2 We're all going to die. Come watch TV. Morty \n",
|
||
"2 3 Losers look stuff up while the rest of us are ... Summer \n",
|
||
"3 4 He's not a hot girl. He can't just bail on his... Beth \n",
|
||
"4 5 When you are an a—hole, it doesn't matter how ... Morty \n",
|
||
"\n",
|
||
" vector \n",
|
||
"0 [0.044295236, -0.0831885, -0.03597761, -0.0396... \n",
|
||
"1 [0.057405394, -0.09669633, 0.00515391, -0.0213... \n",
|
||
"2 [0.057896998, -0.033441037, 0.01376669, -0.015... \n",
|
||
"3 [0.038649295, 0.01286428, -0.03261163, 0.01939... \n",
|
||
"4 [0.07633445, 0.03451182, -0.0037649637, 0.0203... "
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.merge(embeddings, left_on=\"id\")\n",
|
||
"table.head().to_pandas()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f590fec8-0ed0-4148-b940-c81abe7b421c",
|
||
"metadata": {},
|
||
"source": [
|
||
"If we look at the schema, we see that `all-MiniLM-L6-v2` produces 384-dimensional vectors"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "ca9596a0-b4a0-4a5e-8d9e-967cd13b1eae",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"id: int64\n",
|
||
"quote: string\n",
|
||
"author: string\n",
|
||
"vector: fixed_size_list<item: float>[384]\n",
|
||
" child 0, item: float"
|
||
]
|
||
},
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.schema"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f046002c-872c-4c39-ab85-e03c3b45b477",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Rollback\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "dbfc298c-ada2-411b-925f-e53dc9d35f3c",
|
||
"metadata": {},
|
||
"source": [
|
||
"Suppose we used the table and found that the `all-MiniLM-L6-v2` model doesn't produce ideal results. Instead we want to try a larger model. How do we use the new embeddings without losing the change history?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "dfb116e4-b3b2-4b7e-bbf8-d3e63ca2aa14",
|
||
"metadata": {},
|
||
"source": [
|
||
"First, major operations are automatically versioned in LanceDB.\n",
|
||
"Version 1 is the table creation. This contains no rows but just records the schema and metadata.\n",
|
||
"Version 2 is the initial insertion of data.\n",
|
||
"Versions 3 and 4 represents the update (deletion + append)\n",
|
||
"Version 5 is adding the new column."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "a411902b-43d0-4889-8e34-bc5f3c409726",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[{'version': 1,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 39, 40549),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 2,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 39, 63675),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 3,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 53, 979216),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 4,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 53, 988601),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 5,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 35, 44, 475220),\n",
|
||
" 'metadata': {}}]"
|
||
]
|
||
},
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.list_versions()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7bd5e954-ac0f-4973-81c6-ad6120412d40",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can restore version 4, before we added the old vector column"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "ad0682cc-7599-459c-bbd8-1cd1f296c845",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>id</th>\n",
|
||
" <th>quote</th>\n",
|
||
" <th>author</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>1</td>\n",
|
||
" <td>Nobody exists on purpose. Nobody belongs anywh...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>2</td>\n",
|
||
" <td>We're all going to die. Come watch TV.</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>3</td>\n",
|
||
" <td>Losers look stuff up while the rest of us are ...</td>\n",
|
||
" <td>Summer</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>4</td>\n",
|
||
" <td>He's not a hot girl. He can't just bail on his...</td>\n",
|
||
" <td>Beth</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>5</td>\n",
|
||
" <td>When you are an a—hole, it doesn't matter how ...</td>\n",
|
||
" <td>Morty</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" id quote author\n",
|
||
"0 1 Nobody exists on purpose. Nobody belongs anywh... Morty\n",
|
||
"1 2 We're all going to die. Come watch TV. Morty\n",
|
||
"2 3 Losers look stuff up while the rest of us are ... Summer\n",
|
||
"3 4 He's not a hot girl. He can't just bail on his... Beth\n",
|
||
"4 5 When you are an a—hole, it doesn't matter how ... Morty"
|
||
]
|
||
},
|
||
"execution_count": 12,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.restore(4)\n",
|
||
"table.head().to_pandas()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b0a51146-40d0-4f16-9555-5ce68c2c9eee",
|
||
"metadata": {},
|
||
"source": [
|
||
"Notice that we now have one more, not less versions. When we restore an old version, we're not deleting the version history, we're just creating a new version where the schema and data is equivalent to the restored old version. In this way, we can keep track of all of the changes and always rollback to a previous state."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "d5bfb448-20b9-45e9-90ba-8a73abb86668",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[{'version': 1,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 39, 40549),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 2,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 39, 63675),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 3,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 53, 979216),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 4,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 33, 53, 988601),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 5,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 35, 44, 475220),\n",
|
||
" 'metadata': {}},\n",
|
||
" {'version': 6,\n",
|
||
" 'timestamp': datetime.datetime(2023, 10, 20, 14, 36, 15, 658370),\n",
|
||
" 'metadata': {}}]"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.list_versions()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6713cb53-8cb9-4235-9c55-337c311f0af6",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Switching Models\n",
|
||
"\n",
|
||
"Now we'll switch to the `all-mpnet-base-v2` model and add the vectors to the restored dataset again"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "1fa2950d-3002-4903-b6c3-2760ce60d079",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Downloading (…)a8e1d/.gitattributes: 100%|██████████| 1.18k/1.18k [00:00<00:00, 6.02MB/s]\n",
|
||
"Downloading (…)_Pooling/config.json: 100%|██████████| 190/190 [00:00<00:00, 411kB/s]\n",
|
||
"Downloading (…)b20bca8e1d/README.md: 100%|██████████| 10.6k/10.6k [00:00<00:00, 14.2MB/s]\n",
|
||
"Downloading (…)0bca8e1d/config.json: 100%|██████████| 571/571 [00:00<00:00, 1.83MB/s]\n",
|
||
"Downloading (…)ce_transformers.json: 100%|██████████| 116/116 [00:00<00:00, 991kB/s]\n",
|
||
"Downloading (…)e1d/data_config.json: 100%|██████████| 39.3k/39.3k [00:00<00:00, 188kB/s]\n",
|
||
"Downloading pytorch_model.bin: 100%|██████████| 438M/438M [00:34<00:00, 12.6MB/s] \n",
|
||
"Downloading (…)nce_bert_config.json: 100%|██████████| 53.0/53.0 [00:00<00:00, 150kB/s]\n",
|
||
"Downloading (…)cial_tokens_map.json: 100%|██████████| 239/239 [00:00<00:00, 624kB/s]\n",
|
||
"Downloading (…)a8e1d/tokenizer.json: 100%|██████████| 466k/466k [00:00<00:00, 719kB/s]\n",
|
||
"Downloading (…)okenizer_config.json: 100%|██████████| 363/363 [00:00<00:00, 1.06MB/s]\n",
|
||
"Downloading (…)8e1d/train_script.py: 100%|██████████| 13.1k/13.1k [00:00<00:00, 21.8MB/s]\n",
|
||
"Downloading (…)b20bca8e1d/vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 606kB/s]\n",
|
||
"Downloading (…)bca8e1d/modules.json: 100%|██████████| 349/349 [00:00<00:00, 946kB/s]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"model = SentenceTransformer(\"all-mpnet-base-v2\", device=\"cpu\")\n",
|
||
"vectors = model.encode(df.quote.values.tolist(),\n",
|
||
" convert_to_numpy=True,\n",
|
||
" normalize_embeddings=True).tolist()\n",
|
||
"embeddings = vec_to_table(vectors)\n",
|
||
"embeddings = embeddings.append_column(\"id\", pa.array(np.arange(len(table))+1))\n",
|
||
"table.merge(embeddings, left_on=\"id\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "694c46e0-a1c3-4869-a1eb-562f14606ad4",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"id: int64\n",
|
||
"quote: string\n",
|
||
"author: string\n",
|
||
"vector: fixed_size_list<item: float>[768]\n",
|
||
" child 0, item: float"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.schema"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5e4085a5-a2e7-4520-acfc-eabaae2caa7d",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Deletion\n",
|
||
"\n",
|
||
"What if the whole show was just Rick-isms? \n",
|
||
"Let's delete any quote not said by Rick"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "9d11ddf1-b352-496c-91d7-99c70cbf304b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"table.delete(\"author != 'Richard Daniel Sanchez'\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "77d2f591-e492-423e-b995-2a18ae8cb831",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can see that the number of rows has been reduced to 30"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "20bcce48-a5df-43c7-9ab9-7d59a83055e9",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"30"
|
||
]
|
||
},
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"len(table)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ef8457b2-1228-4a25-824e-477a07681b48",
|
||
"metadata": {},
|
||
"source": [
|
||
"Ok we had our fun, let's get back to the full quote set"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "6e279635-75b0-400c-8b43-4aa069282ccd",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"table.restore(7)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "6a65b627-57a2-43b2-8acc-3805591845ad",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"61"
|
||
]
|
||
},
|
||
"execution_count": 19,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"len(table)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ae1a6ee8-8868-49de-82ab-17a0f61f3a47",
|
||
"metadata": {},
|
||
"source": [
|
||
"## History\n",
|
||
"\n",
|
||
"We now have 9 versions in the data. We can review the operations that corresponds to each version below:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "f595c9b8-91ec-48c1-9790-c40e1bd24b60",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"9"
|
||
]
|
||
},
|
||
"execution_count": 20,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"table.version"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "774f4eb0-03d4-4fda-a825-6217bf096619",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"Versions:\n",
|
||
"- 1 - Create\n",
|
||
"- 2 - Append\n",
|
||
"- 3 - Update (deletion)\n",
|
||
"- 4 - Update (append)\n",
|
||
"- 5 - Merge (vector column)\n",
|
||
"- 6 - Restore (4)\n",
|
||
"- 7 - Merge (new vector column)\n",
|
||
"- 8 - Deletion\n",
|
||
"- 9 - Restore"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fb0131e6-2b73-442a-b4c6-6976a9cf4c7e",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Summary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "97a1cf79-b46b-40cd-ada0-54edef358627",
|
||
"metadata": {},
|
||
"source": [
|
||
"We never had to explicitly manage the versioning. And we never had to create expensive and slow snapshots. LanceDB automatically tracks the full history of operations I created and supports fast rollbacks. In production this is critical for debugging issues and minimizing downtime by rolling back to a previously successful state in seconds."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.6"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|