{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train your model\n", "\n", "The first step is, of course, to install `great-ai` in your Python environment." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: great-ai in /data/projects/great-ai/.env/lib/python3.10/site-packages (0.1.0)\n", "Requirement already satisfied: syntok>=1.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.4.4)\n", "Requirement already satisfied: uvicorn[standard]>=0.18.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.18.2)\n", "Requirement already satisfied: langcodes[data]>=3.3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (3.3.0)\n", "Requirement already satisfied: typeguard>=2.10.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.13.3)\n", "Requirement already satisfied: pymongo>=4.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.1.1)\n", "Requirement already satisfied: async_lru>=1.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.0.3)\n", "Requirement already satisfied: plotly>=5.8.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (5.9.0)\n", "Requirement already satisfied: fastapi>=0.70.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.78.0)\n", "Requirement already satisfied: httpx>=0.20.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.23.0)\n", "Requirement already satisfied: matplotlib in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (3.5.2)\n", "Requirement already satisfied: scikit-learn in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.1.1)\n", "Requirement already satisfied: dill>=0.3.5.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.3.5.1)\n", "Requirement already satisfied: numpy in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.23.0)\n", "Requirement already satisfied: watchdog>=2.1.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.1.9)\n", "Requirement already satisfied: tqdm in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.64.0)\n", "Requirement already satisfied: dash>=2.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.5.1)\n", "Requirement already satisfied: unidecode>=1.3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.3.4)\n", "Requirement already satisfied: pandas in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.4.3)\n", "Requirement already satisfied: langdetect>=1.0.9 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.0.9)\n", "Requirement already satisfied: ipython in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (8.4.0)\n", "Requirement already satisfied: tinydb>=4.7.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.7.0)\n", "Requirement already satisfied: nbconvert in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (6.5.0)\n", "Requirement already satisfied: boto3>=1.23.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.24.25)\n", "Requirement already satisfied: botocore<1.28.0,>=1.27.25 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (1.27.25)\n", "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (0.6.0)\n", "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (1.0.1)\n", "Requirement already satisfied: Flask>=1.0.4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.1.2)\n", "Requirement already satisfied: dash-table==5.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (5.0.0)\n", "Requirement already satisfied: dash-core-components==2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.0.0)\n", "Requirement already satisfied: flask-compress in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (1.12)\n", "Requirement already satisfied: dash-html-components==2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.0.0)\n", "Requirement already satisfied: starlette==0.19.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from fastapi>=0.70.0->great-ai) (0.19.1)\n", "Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from fastapi>=0.70.0->great-ai) (1.9.1)\n", "Requirement already satisfied: anyio<5,>=3.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from starlette==0.19.1->fastapi>=0.70.0->great-ai) (3.6.1)\n", "Requirement already satisfied: sniffio in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (1.2.0)\n", "Requirement already satisfied: certifi in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (2022.6.15)\n", "Requirement already satisfied: httpcore<0.16.0,>=0.15.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (0.15.0)\n", "Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (1.5.0)\n", "Requirement already satisfied: language-data<2.0,>=1.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from langcodes[data]>=3.3.0->great-ai) (1.1)\n", "Requirement already satisfied: six in /data/projects/great-ai/.env/lib/python3.10/site-packages (from langdetect>=1.0.9->great-ai) (1.16.0)\n", "Requirement already satisfied: tenacity>=6.2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from plotly>=5.8.0->great-ai) (8.0.1)\n", "Requirement already satisfied: regex>2016 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from syntok>=1.4.0->great-ai) (2022.6.2)\n", "Requirement already satisfied: click>=7.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (8.1.3)\n", "Requirement already satisfied: h11>=0.8 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.12.0)\n", "Requirement already satisfied: websockets>=10.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (10.3)\n", "Requirement already satisfied: watchfiles>=0.13 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.15.0)\n", "Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.16.0)\n", "Requirement already satisfied: httptools>=0.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.4.0)\n", "Requirement already satisfied: PyYAML>=5.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (6.0)\n", "Requirement already satisfied: python-dotenv>=0.13 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.20.0)\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (3.0.30)\n", "Requirement already satisfied: pickleshare in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.7.5)\n", "Requirement already satisfied: stack-data in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.3.0)\n", "Requirement already satisfied: backcall in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.2.0)\n", "Requirement already satisfied: jedi>=0.16 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.18.1)\n", "Requirement already satisfied: pexpect>4.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (4.8.0)\n", "Requirement already satisfied: setuptools>=18.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (63.1.0)\n", "Requirement already satisfied: traitlets>=5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (5.3.0)\n", "Requirement already satisfied: matplotlib-inline in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.1.3)\n", "Requirement already satisfied: decorator in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (5.1.1)\n", "Requirement already satisfied: pygments>=2.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (2.12.0)\n", "Requirement already satisfied: fonttools>=4.22.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (4.34.4)\n", "Requirement already satisfied: packaging>=20.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (21.3)\n", "Requirement already satisfied: pillow>=6.2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (9.2.0)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (1.4.3)\n", "Requirement already satisfied: cycler>=0.10 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (0.11.0)\n", "Requirement already satisfied: pyparsing>=2.2.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (3.0.9)\n", "Requirement already satisfied: python-dateutil>=2.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (2.8.2)\n", "Requirement already satisfied: nbclient>=0.5.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.6.6)\n", "Requirement already satisfied: tinycss2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (1.1.1)\n", "Requirement already satisfied: defusedxml in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.7.1)\n", "Requirement already satisfied: jinja2>=3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (3.1.2)\n", "Requirement already satisfied: bleach in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (5.0.1)\n", "Requirement already satisfied: beautifulsoup4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (4.11.1)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (2.1.1)\n", "Requirement already satisfied: mistune<2,>=0.8.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.8.4)\n", "Requirement already satisfied: jupyter-core>=4.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (4.11.1)\n", "Requirement already satisfied: nbformat>=5.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (5.4.0)\n", "Requirement already satisfied: jupyterlab-pygments in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.2.2)\n", "Requirement already satisfied: pandocfilters>=1.4.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (1.5.0)\n", "Requirement already satisfied: entrypoints>=0.2.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.4)\n", "Requirement already satisfied: pytz>=2020.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pandas->great-ai) (2022.1)\n", "Requirement already satisfied: scipy>=1.3.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (1.8.1)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (3.1.0)\n", "Requirement already satisfied: joblib>=1.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (1.1.0)\n", "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from botocore<1.28.0,>=1.27.25->boto3>=1.23.0->great-ai) (1.26.10)\n", "Requirement already satisfied: itsdangerous>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from Flask>=1.0.4->dash>=2.4.0->great-ai) (2.1.2)\n", "Requirement already satisfied: Werkzeug>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from Flask>=1.0.4->dash>=2.4.0->great-ai) (2.1.2)\n", "Requirement already satisfied: idna>=2.8 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette==0.19.1->fastapi>=0.70.0->great-ai) (3.3)\n", "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jedi>=0.16->ipython->great-ai) (0.8.3)\n", "Requirement already satisfied: marisa-trie<0.8.0,>=0.7.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from language-data<2.0,>=1.1->langcodes[data]>=3.3.0->great-ai) (0.7.7)\n", "Requirement already satisfied: jupyter-client>=6.1.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbclient>=0.5.0->nbconvert->great-ai) (7.3.4)\n", "Requirement already satisfied: nest-asyncio in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbclient>=0.5.0->nbconvert->great-ai) (1.5.5)\n", "Requirement already satisfied: jsonschema>=2.6 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbformat>=5.1->nbconvert->great-ai) (4.6.2)\n", "Requirement already satisfied: fastjsonschema in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbformat>=5.1->nbconvert->great-ai) (2.15.3)\n", "Requirement already satisfied: ptyprocess>=0.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pexpect>4.3->ipython->great-ai) (0.7.0)\n", "Requirement already satisfied: wcwidth in /data/projects/great-ai/.env/lib/python3.10/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->great-ai) (0.2.5)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2->fastapi>=0.70.0->great-ai) (4.3.0)\n", "Requirement already satisfied: soupsieve>1.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from beautifulsoup4->nbconvert->great-ai) (2.3.2.post1)\n", "Requirement already satisfied: webencodings in /data/projects/great-ai/.env/lib/python3.10/site-packages (from bleach->nbconvert->great-ai) (0.5.1)\n", "Requirement already satisfied: brotli in /data/projects/great-ai/.env/lib/python3.10/site-packages (from flask-compress->dash>=2.4.0->great-ai) (1.0.9)\n", "Requirement already satisfied: asttokens in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (2.0.5)\n", "Requirement already satisfied: executing in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (0.8.3)\n", "Requirement already satisfied: pure-eval in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (0.2.2)\n", "Requirement already satisfied: attrs>=17.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->great-ai) (21.4.0)\n", "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->great-ai) (0.18.1)\n", "Requirement already satisfied: pyzmq>=23.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jupyter-client>=6.1.5->nbclient>=0.5.0->nbconvert->great-ai) (23.2.0)\n", "Requirement already satisfied: tornado>=6.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jupyter-client>=6.1.5->nbclient>=0.5.0->nbconvert->great-ai) (6.2)\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install great-ai" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a seperate JSON). \n", "\n", "Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also cleaned to remove any LaTeX, XML, unicode, PDF-extraction artifacts." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import json\n", "from great_ai.utilities import clean\n", "\n", "\n", "def preprocess(line):\n", " data_point = json.loads(line)\n", "\n", " sentence = data_point[\"text\"]\n", " label = data_point[\"label\"]\n", "\n", " return clean(sentence), label" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can load the dataset and extract the *training* samples from it. Since we're impatient, we can do it in parallel using the `simple_parallel_map` function." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 84000/84000 [00:08<00:00, 9771.59it/s] \n" ] } ], "source": [ "from great_ai.utilities import simple_parallel_map\n", "\n", "with open(\"data/train.txt\", encoding=\"utf-8\") as f:\n", " training_data = simple_parallel_map(preprocess, f.readlines())\n", "\n", "X_train = [d[0] for d in training_data]\n", "y_train = [d[1] for d in training_data]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's do the same for the *test* data." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 22399/22399 [00:02<00:00, 7858.71it/s] \n" ] } ], "source": [ "with open(\"data/test.txt\", encoding=\"utf-8\") as f:\n", " test_data = simple_parallel_map(preprocess, f.readlines())\n", "\n", "X_test = [d[0] for d in test_data]\n", "y_test = [d[1] for d in test_data]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After obtaining some clean data, it's time to create and train a model on it. We are going to use [scikit-learn](https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html) for this purpose.\n", "\n", "In this case, we opted for a [linear Support Vector Machine](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) because it's known to be an adequate baseline for simple classification tasks." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),\n",
" ('linearsvc', LinearSVC())])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),\n",
" ('linearsvc', LinearSVC())])TfidfVectorizer()
LinearSVC()