{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
"\n",
"## Part 2: train a model\n",
"\n",
"In [Part 1](data.ipynb), we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3. \n",
"\n",
"In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"from sklearn import metrics, set_config\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.model_selection import GridSearchCV\n",
"import re\n",
"import plotly.express as px\n",
"from great_ai import save_model, configure, LargeFile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m2022-05-27 21:28:11,641 | INFO | Environment variable ENVIRONMENT is not set, defaulting to development mode\u001b[0m\n",
"\u001b[38;5;226m2022-05-27 21:28:11,642 | WARNING | Running in development mode ‼️\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:11,643 | INFO | Options: configured ✅\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:11,683 | INFO | Fetching online versions of semantic-scholar-dataset-small\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:11,959 | INFO | Found versions: [3, 4]\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:11,960 | INFO | Latest version of semantic-scholar-dataset-small is 4\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:11,961 | INFO | File semantic-scholar-dataset-small-4 found in cache\u001b[0m\n"
]
}
],
"source": [
"DATASET_KEY = \"semantic-scholar-dataset-small\"\n",
"MODEL_KEY = \"small-domain-prediction-v2\"\n",
"\n",
"%matplotlib inline\n",
"\n",
"plt.rcParams[\"figure.figsize\"] = (30, 15)\n",
"plt.rcParams[\"figure.facecolor\"] = \"white\"\n",
"plt.rcParams[\"font.size\"] = 12\n",
"plt.rcParams[\"axes.xmargin\"] = 0\n",
"\n",
"configure()\n",
"corpus_path = LargeFile(DATASET_KEY).get()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preprocessing"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m2022-05-27 21:28:11,997 | INFO | Fetching online versions of semantic-scholar-dataset-small\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:12,289 | INFO | Found versions: [3, 4]\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:12,290 | INFO | Latest version of semantic-scholar-dataset-small is 4\u001b[0m\n",
"\u001b[38;5;39m2022-05-27 21:28:12,290 | INFO | File semantic-scholar-dataset-small-4 found in cache\u001b[0m\n"
]
}
],
"source": [
"with LargeFile(DATASET_KEY) as f:\n",
" data = json.load(f)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"alignmentgroup": "True",
"bingroup": "x",
"hovertemplate": "variable=0
value=%{x}
count=%{y}