Train your model¶
The first step is, of course, to install great-ai in your Python environment.
%pip install great-ai
Requirement already satisfied: great-ai in /data/projects/great-ai/.env/lib/python3.10/site-packages (0.1.0) Requirement already satisfied: syntok>=1.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.4.4) Requirement already satisfied: uvicorn[standard]>=0.18.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.18.2) Requirement already satisfied: langcodes[data]>=3.3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (3.3.0) Requirement already satisfied: typeguard>=2.10.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.13.3) Requirement already satisfied: pymongo>=4.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.1.1) Requirement already satisfied: async_lru>=1.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.0.3) Requirement already satisfied: plotly>=5.8.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (5.9.0) Requirement already satisfied: fastapi>=0.70.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.78.0) Requirement already satisfied: httpx>=0.20.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.23.0) Requirement already satisfied: matplotlib in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (3.5.2) Requirement already satisfied: scikit-learn in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.1.1) Requirement already satisfied: dill>=0.3.5.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (0.3.5.1) Requirement already satisfied: numpy in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.23.0) Requirement already satisfied: watchdog>=2.1.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.1.9) Requirement already satisfied: tqdm in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.64.0) Requirement already satisfied: dash>=2.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (2.5.1) Requirement already satisfied: unidecode>=1.3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.3.4) Requirement already satisfied: pandas in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.4.3) Requirement already satisfied: langdetect>=1.0.9 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.0.9) Requirement already satisfied: ipython in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (8.4.0) Requirement already satisfied: tinydb>=4.7.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (4.7.0) Requirement already satisfied: nbconvert in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (6.5.0) Requirement already satisfied: boto3>=1.23.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from great-ai) (1.24.25) Requirement already satisfied: botocore<1.28.0,>=1.27.25 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (1.27.25) Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (0.6.0) Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from boto3>=1.23.0->great-ai) (1.0.1) Requirement already satisfied: Flask>=1.0.4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.1.2) Requirement already satisfied: dash-table==5.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (5.0.0) Requirement already satisfied: dash-core-components==2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.0.0) Requirement already satisfied: flask-compress in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (1.12) Requirement already satisfied: dash-html-components==2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from dash>=2.4.0->great-ai) (2.0.0) Requirement already satisfied: starlette==0.19.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from fastapi>=0.70.0->great-ai) (0.19.1) Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from fastapi>=0.70.0->great-ai) (1.9.1) Requirement already satisfied: anyio<5,>=3.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from starlette==0.19.1->fastapi>=0.70.0->great-ai) (3.6.1) Requirement already satisfied: sniffio in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (1.2.0) Requirement already satisfied: certifi in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (2022.6.15) Requirement already satisfied: httpcore<0.16.0,>=0.15.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (0.15.0) Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from httpx>=0.20.0->great-ai) (1.5.0) Requirement already satisfied: language-data<2.0,>=1.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from langcodes[data]>=3.3.0->great-ai) (1.1) Requirement already satisfied: six in /data/projects/great-ai/.env/lib/python3.10/site-packages (from langdetect>=1.0.9->great-ai) (1.16.0) Requirement already satisfied: tenacity>=6.2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from plotly>=5.8.0->great-ai) (8.0.1) Requirement already satisfied: regex>2016 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from syntok>=1.4.0->great-ai) (2022.6.2) Requirement already satisfied: click>=7.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (8.1.3) Requirement already satisfied: h11>=0.8 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.12.0) Requirement already satisfied: websockets>=10.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (10.3) Requirement already satisfied: watchfiles>=0.13 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.15.0) Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.16.0) Requirement already satisfied: httptools>=0.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.4.0) Requirement already satisfied: PyYAML>=5.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (6.0) Requirement already satisfied: python-dotenv>=0.13 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.0->great-ai) (0.20.0) Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (3.0.30) Requirement already satisfied: pickleshare in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.7.5) Requirement already satisfied: stack-data in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.3.0) Requirement already satisfied: backcall in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.2.0) Requirement already satisfied: jedi>=0.16 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.18.1) Requirement already satisfied: pexpect>4.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (4.8.0) Requirement already satisfied: setuptools>=18.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (63.1.0) Requirement already satisfied: traitlets>=5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (5.3.0) Requirement already satisfied: matplotlib-inline in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (0.1.3) Requirement already satisfied: decorator in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (5.1.1) Requirement already satisfied: pygments>=2.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from ipython->great-ai) (2.12.0) Requirement already satisfied: fonttools>=4.22.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (4.34.4) Requirement already satisfied: packaging>=20.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (21.3) Requirement already satisfied: pillow>=6.2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (9.2.0) Requirement already satisfied: kiwisolver>=1.0.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (1.4.3) Requirement already satisfied: cycler>=0.10 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (0.11.0) Requirement already satisfied: pyparsing>=2.2.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (3.0.9) Requirement already satisfied: python-dateutil>=2.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from matplotlib->great-ai) (2.8.2) Requirement already satisfied: nbclient>=0.5.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.6.6) Requirement already satisfied: tinycss2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (1.1.1) Requirement already satisfied: defusedxml in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.7.1) Requirement already satisfied: jinja2>=3.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (3.1.2) Requirement already satisfied: bleach in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (5.0.1) Requirement already satisfied: beautifulsoup4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (4.11.1) Requirement already satisfied: MarkupSafe>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (2.1.1) Requirement already satisfied: mistune<2,>=0.8.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.8.4) Requirement already satisfied: jupyter-core>=4.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (4.11.1) Requirement already satisfied: nbformat>=5.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (5.4.0) Requirement already satisfied: jupyterlab-pygments in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.2.2) Requirement already satisfied: pandocfilters>=1.4.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (1.5.0) Requirement already satisfied: entrypoints>=0.2.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbconvert->great-ai) (0.4) Requirement already satisfied: pytz>=2020.1 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pandas->great-ai) (2022.1) Requirement already satisfied: scipy>=1.3.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (1.8.1) Requirement already satisfied: threadpoolctl>=2.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (3.1.0) Requirement already satisfied: joblib>=1.0.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from scikit-learn->great-ai) (1.1.0) Requirement already satisfied: urllib3<1.27,>=1.25.4 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from botocore<1.28.0,>=1.27.25->boto3>=1.23.0->great-ai) (1.26.10) Requirement already satisfied: itsdangerous>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from Flask>=1.0.4->dash>=2.4.0->great-ai) (2.1.2) Requirement already satisfied: Werkzeug>=2.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from Flask>=1.0.4->dash>=2.4.0->great-ai) (2.1.2) Requirement already satisfied: idna>=2.8 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette==0.19.1->fastapi>=0.70.0->great-ai) (3.3) Requirement already satisfied: parso<0.9.0,>=0.8.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jedi>=0.16->ipython->great-ai) (0.8.3) Requirement already satisfied: marisa-trie<0.8.0,>=0.7.7 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from language-data<2.0,>=1.1->langcodes[data]>=3.3.0->great-ai) (0.7.7) Requirement already satisfied: jupyter-client>=6.1.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbclient>=0.5.0->nbconvert->great-ai) (7.3.4) Requirement already satisfied: nest-asyncio in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbclient>=0.5.0->nbconvert->great-ai) (1.5.5) Requirement already satisfied: jsonschema>=2.6 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbformat>=5.1->nbconvert->great-ai) (4.6.2) Requirement already satisfied: fastjsonschema in /data/projects/great-ai/.env/lib/python3.10/site-packages (from nbformat>=5.1->nbconvert->great-ai) (2.15.3) Requirement already satisfied: ptyprocess>=0.5 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pexpect>4.3->ipython->great-ai) (0.7.0) Requirement already satisfied: wcwidth in /data/projects/great-ai/.env/lib/python3.10/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->great-ai) (0.2.5) Requirement already satisfied: typing-extensions>=3.7.4.3 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2->fastapi>=0.70.0->great-ai) (4.3.0) Requirement already satisfied: soupsieve>1.2 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from beautifulsoup4->nbconvert->great-ai) (2.3.2.post1) Requirement already satisfied: webencodings in /data/projects/great-ai/.env/lib/python3.10/site-packages (from bleach->nbconvert->great-ai) (0.5.1) Requirement already satisfied: brotli in /data/projects/great-ai/.env/lib/python3.10/site-packages (from flask-compress->dash>=2.4.0->great-ai) (1.0.9) Requirement already satisfied: asttokens in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (2.0.5) Requirement already satisfied: executing in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (0.8.3) Requirement already satisfied: pure-eval in /data/projects/great-ai/.env/lib/python3.10/site-packages (from stack-data->ipython->great-ai) (0.2.2) Requirement already satisfied: attrs>=17.4.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->great-ai) (21.4.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->great-ai) (0.18.1) Requirement already satisfied: pyzmq>=23.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jupyter-client>=6.1.5->nbclient>=0.5.0->nbconvert->great-ai) (23.2.0) Requirement already satisfied: tornado>=6.0 in /data/projects/great-ai/.env/lib/python3.10/site-packages (from jupyter-client>=6.1.5->nbclient>=0.5.0->nbconvert->great-ai) (6.2) Note: you may need to restart the kernel to use updated packages.
First, we have to get some data. After downloading it from here, we might notice that the dataset is in JSON Lines format (each line is a seperate JSON).
Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also cleaned to remove any LaTeX, XML, unicode, PDF-extraction artifacts.
import json
from great_ai.utilities import clean
def preprocess(line):
data_point = json.loads(line)
sentence = data_point["text"]
label = data_point["label"]
return clean(sentence), label
Now, we can load the dataset and extract the training samples from it. Since we're impatient, we can do it in parallel using the simple_parallel_map function.
from great_ai.utilities import simple_parallel_map
with open("data/train.txt", encoding="utf-8") as f:
training_data = simple_parallel_map(preprocess, f.readlines())
X_train = [d[0] for d in training_data]
y_train = [d[1] for d in training_data]
100%|██████████| 84000/84000 [00:08<00:00, 9771.59it/s]
Let's do the same for the test data.
with open("data/test.txt", encoding="utf-8") as f:
test_data = simple_parallel_map(preprocess, f.readlines())
X_test = [d[0] for d in test_data]
y_test = [d[1] for d in test_data]
100%|██████████| 22399/22399 [00:02<00:00, 7858.71it/s]
After obtaining some clean data, it's time to create and train a model on it. We are going to use scikit-learn for this purpose.
In this case, we opted for a linear Support Vector Machine because it's known to be an adequate baseline for simple classification tasks.
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
model = make_pipeline(TfidfVectorizer(), LinearSVC())
# todo: hyperparameter-optimisation
model.fit(X_train, y_train)
Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),
('linearsvc', LinearSVC())])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),
('linearsvc', LinearSVC())])TfidfVectorizer()
LinearSVC()
It's already finished. Let's see how well it performs. But first, we need to configure matplotlib to make the charts more legible.
ConfusionMatrixDisplayrelies onmatplotlibfor rendering.
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams["figure.figsize"] = (30, 15)
plt.rcParams["font.size"] = 12
Next, we check the quality of the model on the test split. We can use the classification report to check the common metrics, such as the macro F1-score. We also draw the confusion matrix to get some insights into the types of mistakes being made by the model.
from sklearn import metrics
y_predicted = model.predict(X_test)
print(metrics.classification_report(y_test, y_predicted))
metrics.ConfusionMatrixDisplay.from_predictions(y_test, y_predicted)
None
precision recall f1-score support
business 0.67 0.69 0.68 3198
economics 0.69 0.71 0.70 3189
geography 0.70 0.73 0.72 3207
medicine 0.90 0.91 0.90 3187
politics 0.63 0.57 0.60 3169
psychology 0.73 0.73 0.73 3252
sociology 0.51 0.51 0.51 3197
accuracy 0.69 22399
macro avg 0.69 0.69 0.69 22399
weighted avg 0.69 0.69 0.69 22399
Great work, we can be rightfully satisfied with our model. Seeing the results, we achieved an F1-score of 0.69 which is about 5% better than SciBERT's 0.6571!
You might wonder that "this is great, but besides some utility functions (clean, simple_parallel_map, ...) what more value does GreatAI add?". This would be a valid argument because the scope of GreatAI actually only starts here.
Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey-zone for software engineering.
In order to use this model in production, we have to make it accessible on some possibly shared infrastructure.
from great_ai import save_model
save_model(model, key="my-domain-predictor")
Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️ Cannot find credentials files, defaulting to using ParallelTinyDbDriver The selected tracing database (ParallelTinyDbDriver) is not recommended for production Cannot find credentials files, defaulting to using LargeFileLocal GreatAI (v0.1.3): configured ✅ 🔩 tracing_database: ParallelTinyDbDriver 🔩 large_file_implementation: LargeFileLocal 🔩 is_production: False 🔩 should_log_exception_stack: True 🔩 prediction_cache_size: 512 🔩 dashboard_table_size: 50 You still need to check whether you follow all best practices before trusting your deployment. > Find out more at https://se-ml.github.io/practices Fetching cached versions of my-domain-predictor Copying file for my-domain-predictor-6 Compressing my-domain-predictor-6 Model my-domain-predictor uploaded with version 6
'my-domain-predictor:6'