![](\"https://llmstack.ai/assets/images/rag-f517f1f834bdbb94a87765e0edd40ff2.png\")
\n",
+ "\n",
+ "[source](https://llmstack.ai/assets/images/rag-f517f1f834bdbb94a87765e0edd40ff2.png)\n",
+ "\n",
+ "## How do you go about improving retreival performance\n",
+ "Some of the common techniques are:\n",
+ "\n",
+ "- Using different search types - vector/semantic, FTS (BM25)\n",
+ "- Hybrid search\n",
+ "- Reranking\n",
+ "- Fine-tuning the embedding models\n",
+ "- Using different embedding models\n",
+ "\n",
+ "Obviously, the above list is not exhaustive. There are other subtler ways that can improve retrieval performance like experimenting chunking algorithms, using different distance/similarity metrics etc. But for brevity, we'll only cover high level and more impactful techniques here.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "3ZCm3-Bog9g7"
+ },
+ "source": [
+ "# LanceDB\n",
+ "- Multimodal DB for AI\n",
+ "- Powered by an innovative & open-source in-house file format\n",
+ "- 0 Setup\n",
+ "- Scales up on disk storage\n",
+ "- Native support for vector, full-text(BM25) and hybrid search\n",
+ "\n",
+ "![](\"https://lancedb.github.io/lancedb/assets/lancedb_and_lance.png\"\n",)
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "b1fzhbQc4O1u"
+ },
+ "source": [
+ "## The dataset\n",
+ "The dataset we'll use is a synthetic QA dataset generated from LLama2 review paper. The paper was divided into chunks, with each chunk being a unique context. An LLM was prompted to ask questions relevant to the context for testing a retreiver.\n",
+ "The exact code and other utility functions for this can be found in [this](https://github.com/lancedb/ragged) repo\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "f_qnH-Dfhi9Z",
+ "outputId": "9d41e17e-c994-473d-cf17-93568e025252"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "--2024-06-26 13:31:34-- https://raw.githubusercontent.com/AyushExel/assets/main/data_qa.csv\n",
+ "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
+ "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
+ "HTTP request sent, awaiting response... 200 OK\n",
+ "Length: 680439 (664K) [text/plain]\n",
+ "Saving to: ‘data_qa.csv’\n",
+ "\n",
+ "data_qa.csv 100%[===================>] 664.49K --.-KB/s in 0.009s \n",
+ "\n",
+ "2024-06-26 13:31:34 (71.7 MB/s) - ‘data_qa.csv’ saved [680439/680439]\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "!wget https://raw.githubusercontent.com/AyushExel/assets/main/data_qa.csv"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "ZNNAUc6f7ILI"
+ },
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "data = pd.read_csv(\"data_qa.csv\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 580
+ },
+ "id": "4Bp9Fdhz7QsM",
+ "outputId": "4cb6b384-7991-4911-e5fd-7fbd42487916"
+ },
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "summary": "{\n \"name\": \"data\",\n \"rows\": 220,\n \"fields\": [\n {\n \"column\": \"Unnamed: 0\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 63,\n \"min\": 0,\n \"max\": 219,\n \"num_unique_values\": 220,\n \"samples\": [\n 132,\n 148,\n 93\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"query\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 220,\n \"samples\": [\n \"What type of examination did scholars perform on ChatGPT, and when was the resulting scholarly paper published?\",\n \"How do the performance capabilities of the different models compare in evaluating tasks associated with logical reasoning and reading comprehension, specifically noted in tests like LSAT and SAT?\",\n \"What steps are recommended for users to ensure the responsible use of AI models like Llama 2 in projects or commercial applications?\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"context\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 110,\n \"samples\": [\n \"Dialogue Turn Baseline + GAtt\\n2 100% 100%\\n4 10% 100%\\n6 0% 100%\\n20 0% 100%\\nTable30: GAttresults. Llama 2-Chat withGAttisabletorefertoattributes100%ofthetime,forupto20\\nturns from our human evaluation. We limited the evaluated attributes to public figures and hobbies.\\nTheattentionnowspansbeyond20turns. Wetestedthemodelabilitytorememberthesystemarguments\\ntroughahumanevaluation. Thearguments(e.g. hobbies,persona)aredefinedduringthefirstmessage,and\\nthen from turn 2 to 20. We explicitly asked the model to refer to them (e.g. \\u201cWhat is your favorite hobby?\\u201d,\\n\\u201cWhatisyourname?\\u201d),tomeasurethemulti-turnmemoryabilityof Llama 2-Chat . Wereporttheresults\\ninTable30. EquippedwithGAtt, Llama 2-Chat maintains100%accuracy,alwaysreferringtothedefined\\nattribute,andso,upto20turns(wedidnotextendthehumanevaluationmore,andalltheexampleshad\\nlessthan4048tokensintotalovertheturns). Asacomparison, Llama 2-Chat withoutGAttcannotanymore\\nrefer to the attributes after only few turns: from 100% at turn t+1, to 10% at turn t+3 and then 0%.\\nGAttZero-shotGeneralisation. Wetriedatinferencetimetosetconstrainnotpresentinthetrainingof\\nGAtt. For instance, \\u201canswer in one sentence only\\u201d, for which the model remained consistent, as illustrated in\\nFigure 28.\\nWe applied first GAtt to Llama 1 , which was pretrained with a context length of 2048 tokens and then\\nfine-tuned with 4096 max length. We tested if GAtt works beyond 2048 tokens, and the model arguably\\nmanaged to understand attributes beyond this window. This promising result indicates that GAtt could be\\nadapted as an efficient technique for long context attention.\\nA.3.6 How Far Can Model-Based Evaluation Go?\\nTo measure the robustness of our reward model, we collected a test set of prompts for both helpfulness and\\nsafety,andaskedannotatorstojudgequalityoftheanswersbasedona7pointLikert-scale(thehigherthe\\nbetter)usingtriplereviews. AsillustratedinFigure29(inAppendix),weobservethatourrewardmodels\\noverallarewellcalibratedwithhumanpreference. Notethatthisenablesustousetherewardasapoint-wise\\nmetric, despite being trained with a Pairwise Ranking Loss.\\n0.0% 2.0% 4.0% 6.0% 8.0%\\nDensity0.00.20.40.60.81.0Reward Model ScoreNo Margin\\n0.0% 2.0% 4.0% 6.0% 8.0%\\nDensity0.00.20.40.60.81.0\\nMargin Small\\n0.0% 2.0% 4.0% 6.0% 8.0%\\nDensity0.00.20.40.60.81.0\\nMargin Large\\nFigure 27: Reward model score distribution shift caused by incorporating preference rating based margin\\ninrankingloss. Withthemarginterm, weobserveabinary splitpatterninrewarddistribution, especially\\nwith a larger margin.\\n54\",\n \"Model Size CodeCommonsense\\nReasoningWorld\\nKnowledgeReading\\nComprehensionMath MMLU BBH AGI Eval\\nMPT7B 20.5 57.4 41.0 57.5 4.9 26.8 31.0 23.5\\n30B 28.9 64.9 50.0 64.7 9.1 46.9 38.0 33.8\\nFalcon7B 5.6 56.1 42.8 36.0 4.6 26.2 28.0 21.2\\n40B 15.2 69.2 56.7 65.7 12.6 55.4 37.1 37.0\\nLlama 17B 14.1 60.8 46.2 58.5 6.95 35.1 30.3 23.9\\n13B 18.9 66.1 52.6 62.3 10.9 46.9 37.0 33.9\\n33B 26.0 70.0 58.4 67.6 21.4 57.8 39.8 41.7\\n65B 30.7 70.7 60.5 68.6 30.8 63.4 43.5 47.6\\nLlama 27B 16.8 63.9 48.9 61.3 14.6 45.3 32.6 29.3\\n13B 24.5 66.9 55.4 65.8 28.7 54.8 39.4 39.1\\n34B 27.8 69.9 58.7 68.0 24.2 62.6 44.1 43.4\\n70B37.5 71.9 63.6 69.4 35.2 68.9 51.2 54.2\\nTable3: Overallperformanceongroupedacademicbenchmarkscomparedtoopen-sourcebasemodels.\\n\\u2022Popular Aggregated Benchmarks . We report the overall results for MMLU (5 shot) (Hendrycks\\net al., 2020), Big Bench Hard (BBH) (3 shot) (Suzgun et al., 2022), and AGI Eval (3\\u20135 shot) (Zhong\\net al., 2023). For AGI Eval, we only evaluate on the English tasks and report the average.\\nAs shown in Table 3, Llama 2 models outperform Llama 1 models. In particular, Llama 2 70B improves the\\nresultsonMMLUandBBHby \\u22485and\\u22488points,respectively,comparedto Llama 1 65B.Llama 2 7Band30B\\nmodelsoutperformMPTmodelsofthecorrespondingsizeonallcategoriesbesidescodebenchmarks. Forthe\\nFalcon models, Llama 2 7B and 34B outperform Falcon 7B and 40B models on all categories of benchmarks.\\nAdditionally, Llama 2 70B model outperforms all open-source models.\\nIn addition to open-source models, we also compare Llama 2 70B results to closed-source models. As shown\\nin Table 4, Llama 2 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a significant\\ngaponcodingbenchmarks. Llama 2 70BresultsareonparorbetterthanPaLM(540B)(Chowdheryetal.,\\n2022)onalmostallbenchmarks. Thereisstillalargegapinperformancebetween Llama 2 70BandGPT-4\\nand PaLM-2-L.\\nWe also analysed the potential data contamination and share the details in Section A.6.\",\n \"Figure 1: Helpfulness human evaluation results for Llama\\n2-Chatcomparedtootheropen-sourceandclosed-source\\nmodels. Human raters compared model generations on ~4k\\npromptsconsistingofbothsingleandmulti-turnprompts.\\nThe95%confidenceintervalsforthisevaluationarebetween\\n1%and2%. MoredetailsinSection3.4.2. Whilereviewing\\nthese results, it is important to note that human evaluations\\ncanbenoisyduetolimitationsofthepromptset,subjectivity\\nof the review guidelines, subjectivity of individual raters,\\nand the inherent difficulty of comparing generations.\\nFigure 2: Win-rate % for helpfulness and\\nsafety between commercial-licensed base-\\nlines and Llama 2-Chat , according to GPT-\\n4. Tocomplementthehumanevaluation,we\\nused a more capable model, not subject to\\nourownguidance. Greenareaindicatesour\\nmodelisbetteraccordingtoGPT-4. Toremove\\nties, we used win/ (win+loss). The orders in\\nwhichthemodelresponsesarepresentedto\\nGPT-4arerandomlyswappedtoalleviatebias.\\n1 Introduction\\nLarge Language Models (LLMs) have shown great promise as highly capable AI assistants that excel in\\ncomplex reasoning tasks requiring expert knowledge across a wide range of fields, including in specialized\\ndomains such as programming and creative writing. They enable interaction with humans through intuitive\\nchat interfaces, which has led to rapid and widespread adoption among the general public.\\nThecapabilitiesofLLMsareremarkableconsideringtheseeminglystraightforwardnatureofthetraining\\nmethodology. Auto-regressivetransformersarepretrainedonanextensivecorpusofself-superviseddata,\\nfollowed by alignment with human preferences via techniques such as Reinforcement Learning with Human\\nFeedback(RLHF).Althoughthetrainingmethodologyissimple,highcomputationalrequirementshave\\nlimited the development of LLMs to a few players. There have been public releases of pretrained LLMs\\n(such as BLOOM (Scao et al., 2022), LLaMa-1 (Touvron et al., 2023), and Falcon (Penedo et al., 2023)) that\\nmatch the performance of closed pretrained competitors like GPT-3 (Brown et al., 2020) and Chinchilla\\n(Hoffmann et al., 2022), but none of these models are suitable substitutes for closed \\u201cproduct\\u201d LLMs, such\\nasChatGPT,BARD,andClaude. TheseclosedproductLLMsareheavilyfine-tunedtoalignwithhuman\\npreferences, which greatly enhances their usability and safety. This step can require significant costs in\\ncomputeandhumanannotation,andisoftennottransparentoreasilyreproducible,limitingprogresswithin\\nthe community to advance AI alignment research.\\nIn this work, we develop and release Llama 2, a family of pretrained and fine-tuned LLMs, Llama 2 and\\nLlama 2-Chat , at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested,\\nLlama 2-Chat models generally perform better than existing open-source models. They also appear to\\nbe on par with some of the closed-source models, at least on the human evaluations we performed (see\\nFigures1and3). Wehavetakenmeasurestoincreasethesafetyofthesemodels,usingsafety-specificdata\\nannotation and tuning, as well as conducting red-teaming and employing iterative evaluations. Additionally,\\nthispapercontributesathoroughdescriptionofourfine-tuningmethodologyandapproachtoimproving\\nLLM safety. We hope that this openness will enable the community to reproduce fine-tuned LLMs and\\ncontinue to improve the safety of those models, paving the way for more responsible development of LLMs.\\nWealsosharenovelobservationswemadeduringthedevelopmentof Llama 2 andLlama 2-Chat ,suchas\\nthe emergence of tool usage and temporal organization of knowledge.\\n3\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"answer\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 220,\n \"samples\": [\n \"Scholars performed a diagnostic analysis to investigate the AI ethics associated with ChatGPT. Their findings were compiled into a research paper that became accessible as a preprint on arXiv in January 2023.\",\n \"The MPT 30B model demonstrates considerable proficiency in logical reasoning and reading comprehension tasks, scoring highly on LSAT-LR, LSAT-RC, and SAT-en tests compared to its peers, such as Falcon 40B and Llama 17B. This is indicative of its advanced analytical and comprehension abilities. Conversely, while Falcon 40B shows strengths in LSAT-LR with a score second only to MPT 30B, it trails in SAT-en performance. This variability underscores the diverse capabilities of models based on their structural design and training paradigms.\",\n \"Users intending to deploy models like Llama 2 are advised to strictly adhere to guidelines laid out in the Responsible Use Guide. This includes employing enhanced safety measures at both the input and output stages of model interaction, as well as carefully tuning the model according to specific use-case requirements to prevent any potential misuse. Additionally, users must comply with the terms set in the Acceptable Use Policy, ensuring their applications do not contravene applicable laws, regulations, and ethical standards. Leveraging provided code examples can further assist developers in replicating the necessary safety protocols and maintaining ethical integrity in their applications.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
+ "type": "dataframe",
+ "variable_name": "data"
+ },
+ "text/html": [
+ "\n",
+ " | \n", + " | Unnamed: 0 | \n", + "query | \n", + "context | \n", + "answer | \n", + "
|---|---|---|---|---|
| 0 | \n", + "0 | \n", + "How does the performance of Llama 2-Chat model... | \n", + "Llama 2 : Open Foundation and Fine-Tuned Chat ... | \n", + "Llama 2-Chat models have shown to exceed the p... | \n", + "
| 1 | \n", + "1 | \n", + "What benefits does the enhancement and safety ... | \n", + "Llama 2 : Open Foundation and Fine-Tuned Chat ... | \n", + "The safety and enhancement measures implemente... | \n", + "
| 2 | \n", + "2 | \n", + "How does one ensure the reliability and robust... | \n", + "Contents\\n1 Introduction 3\\n2 Pretraining 5\\n2... | \n", + "In the initial steps of model development, the... | \n", + "
| 3 | \n", + "3 | \n", + "What methodologies are employed to align machi... | \n", + "Contents\\n1 Introduction 3\\n2 Pretraining 5\\n2... | \n", + "Machine learning models can be aligned with de... | \n", + "
| 4 | \n", + "4 | \n", + "What are some of the primary insights gained f... | \n", + ". . . . . . . . 23\\n4.3 Red Teaming . . . . . ... | \n", + "The key insights gained from evaluating platfo... | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 215 | \n", + "215 | \n", + "How are the terms 'clean', 'not clean', 'dirty... | \n", + "Giventhe\\nembarrassinglyparallelnatureofthetas... | \n", + "In the discussed dataset analysis, samples are... | \n", + "
| 216 | \n", + "216 | \n", + "How does the size of the model influence the a... | \n", + "Dataset Model Subset Type Avg. Contam. % n ¯X ... | \n", + "The size of the model significantly influences... | \n", + "
| 217 | \n", + "217 | \n", + "What impact does the model contamination have ... | \n", + "Dataset Model Subset Type Avg. Contam. % n ¯X ... | \n", + "Model contamination affects various contaminat... | \n", + "
| 218 | \n", + "218 | \n", + "What are the different sizes and types availab... | \n", + "A.7 Model Card\\nTable 52 presents a model card... | \n", + "Llama 2 is available in three distinct paramet... | \n", + "
| 219 | \n", + "219 | \n", + "Could you discuss the sustainability measures ... | \n", + "A.7 Model Card\\nTable 52 presents a model card... | \n", + "Throughout the training of Llama 2, which invo... | \n", + "
220 rows × 4 columns
\n", + "![](\"https://blog.lancedb.com/content/images/2024/02/1_Zh4Jju6uiCYFO9HHvO5sIA.webp\")
\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "ydLNeAr4acYj",
+ "outputId": "34263ad1-1c21-43a5-f592-16e7f7ca8c45"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 220/220 [00:10<00:00, 20.37it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ " Hybrid Search with LinearCombinationReranker Hit Rate: 0.6454545454545455\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "from lancedb.rerankers import LinearCombinationReranker # LanceDB hybrid search uses LinearCombinationReranker by default\n",
+ "\n",
+ "reranker = LinearCombinationReranker(weight=0.7)\n",
+ "hit_rate_hybrid = hit_rate(data, tbl, \"hybrid\", reranker=reranker)\n",
+ "\n",
+ "print(f\"\\n Hybrid Search with LinearCombinationReranker Hit Rate: {hit_rate_hybrid}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Wswq157ptjTZ"
+ },
+ "source": [
+ "## Trying out different rerankers\n",
+ "\n",
+ "### 1. Cross Encoder Reranker\n",
+ "\n",
+ " ![](\"https://raw.githubusercontent.com/UKPLab/sentence-transformers/master/docs/img/Bi_vs_Cross-Encoder.png\")
\n",
+ "\n",
+ "Bi-Encoders produce for a given sentence a sentence embedding. We pass to a BERT independently the sentences A and B, which result in the sentence embeddings u and v. These sentence embedding can then be compared using cosine similarity.\n",
+ "\n",
+ "In contrast, for a Cross-Encoder, we pass both sentences simultaneously to the Transformer network. It produces then an output value between 0 and 1 indicating the similarity of the input sentence pair:\n",
+ "\n",
+ "A Cross-Encoder does not produce a sentence embedding. Also, we are not able to pass individual sentences to a Cross-Encoder."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 286,
+ "referenced_widgets": [
+ "1833d9358c9b4cb581cc618ac68d7e30",
+ "8609f6b2059943d9bad3254b3d7ec629",
+ "25aefdd4c5644c709e03115bc8ae5f99",
+ "7ba55480b151488782c14afdc459b182",
+ "5414385b19d54478a98969b4d45736fa",
+ "c17520bc49fd4539bceaa78318cce796",
+ "8b7c9f8f944e4c4eb4cff354b42e086a",
+ "559ce10ceb2e4b558ae7124b2d359031",
+ "db0cca3c1f9a4fc695bb7362409b2cbe",
+ "27d1b3b7b4cd4588910f3aa07ab4fc3e",
+ "caab49ce2bf940eb8ac988f88a0dadef",
+ "6e37c2f25d3946fca0564fe3363f4725",
+ "08d187281e1e463785b7a5e88f47f316",
+ "f43a7a4b03b240b7bcef5befa7c91b46",
+ "c800dfef86de45a8b13b6621ef4e1a84",
+ "7926d3c178a949e182328fef895b9cbe",
+ "42caf4b7c7804a91bf320f38f7481ecb",
+ "45c142aca09946929dd91e2ad8463fac",
+ "e215079b606d43738715e21d784b8d2c",
+ "bd455814e03b4dea8169ba39b782e5dc",
+ "3f6c357e36ad48c2a73cbf7f9fe74716",
+ "d202239dc6ac4107b5c5f930cee827d7",
+ "e5e24aa6f68f41e3ba3f7f0585eddf8f",
+ "426a927035474cb2ad4936ca2041d2e1",
+ "cf476ad98f34482ebc14638fa5b7a35f",
+ "52f061513e3845d7819d0d103afe8258",
+ "5c476489e4964893a70aef719c38ac46",
+ "4f7ce5f08915421eb115d3e98cf60007",
+ "ea80b988836247838d21652c75b319ea",
+ "306c491ee6a04737aa400d6ff0a8c463",
+ "f6ecbdd2cad14aa4ad2b184f1889c8f5",
+ "e48fd3d0d8054fb297361c1e2d17f6ba",
+ "a9cb97c0b36e4da7bae4571baf0ec9d4",
+ "607b5519211d47a6ad75d5b2739ee907",
+ "079979603b3a46c89934f3718cdde9a5",
+ "3cca14175fd54dc3b2967d0e14a0df66",
+ "53d7258eb0924da38642be8c714bc6c3",
+ "5b5c3f2bbdbd456ebc26c6069869dcbd",
+ "d82d228598ae4eedb1a92c18aa03e985",
+ "2b361f073b1c49129ede34d359caff88",
+ "d2948d1af00e42e78058248f45059c30",
+ "48fee23cb275447bb43942a38cb94a17",
+ "ea726496c3e8476091d7be2c96def0d7",
+ "0d6b1481bbcd4a949f48396b7a385f6f",
+ "c70a82bbcb974214b1563babd29d50ef",
+ "43447d181ae44d71a449957f5eed54d6",
+ "706d91b06672487aa62dc974b4b28342",
+ "7c8b3122064a4289a9d8b246f9c37dfe",
+ "3c1b70e0586945d5a240a769a61f1ef0",
+ "44907a6e01ee434383566a39de8c05e8",
+ "b2358f595c3f47399530d363f63166a0",
+ "08b6d24ba33942b7bb4f47a7f43c4ad5",
+ "1b5fb3da3e98440aba828920d512c7fa",
+ "ae501dd3797e40b3bd0a018b6212f80e",
+ "8aa85ff7d417493ea7bf43e347fca68d"
+ ]
+ },
+ "id": "dd0jh4gNtm41",
+ "outputId": "731f7c32-838d-4e18-c080-83bece99f506"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\r 0%| | 0/220 [00:00\n",
+ "\n",
+ " \n",
+ " 2. **Social media personalised timeline**\n",
+ " \n",
+ "\n",
+ " 3. **Recommend blogs, videos, etc. via push notifications**\n",
+ "\n",
+ " \"YouTube now gives notifications for \"recommended\", non-subscribed channels\" - https://www.reddit.com/r/assholedesign/comments/807zpe/youtube_now_gives_notifications_for_recommended/\n",
+ " \n",
+ " ![](\"https://preview.redd.it/q8uq4e4bpfi01.png?auto=webp&s=95d1a3c1d05971d8040bb650e4287518b3a0f312\")
\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": []
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "gpuType": "T4",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}