{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
"\n",
"## Part 2: train a model\n",
"\n",
"In [Part 1](data.ipynb), we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3. \n",
"\n",
"In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"from collections import Counter\n",
"from sklearn import metrics, set_config\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.model_selection import GridSearchCV\n",
"import plotly.express as px\n",
"from great_ai import save_model, configure, LargeFile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;226m2022-05-28 16:39:38,557 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
"\u001b[38;5;39m2022-05-28 16:39:38,559 | INFO | Options: configured ✅\u001b[0m\n"
]
}
],
"source": [
"DATASET_KEY = \"semantic-scholar-dataset-small\"\n",
"MODEL_KEY = \"small-domain-prediction\"\n",
"\n",
"%matplotlib inline\n",
"\n",
"plt.rcParams[\"figure.figsize\"] = (30, 15)\n",
"plt.rcParams[\"figure.facecolor\"] = \"white\"\n",
"plt.rcParams[\"font.size\"] = 12\n",
"plt.rcParams[\"axes.xmargin\"] = 0\n",
"\n",
"configure()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preprocessing"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39m2022-05-28 16:39:38,934 | INFO | Latest version of semantic-scholar-dataset-small is 4 (from versions: 3, 4)\u001b[0m\n",
"\u001b[38;5;39m2022-05-28 16:39:38,934 | INFO | File semantic-scholar-dataset-small-4 found in cache\u001b[0m\n"
]
}
],
"source": [
"with LargeFile(DATASET_KEY) as f:\n",
" data = json.load(f)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"alignmentgroup": "True",
"hovertemplate": "x=%{x}
y=%{y}
Pipeline(steps=[('vectorizer', TfidfVectorizer(max_df=0.1, min_df=5)),\n",
" ('classifier', MultinomialNB(alpha=0.1, fit_prior=False))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('vectorizer', TfidfVectorizer(max_df=0.1, min_df=5)),\n",
" ('classifier', MultinomialNB(alpha=0.1, fit_prior=False))])TfidfVectorizer(max_df=0.1, min_df=5)
MultinomialNB(alpha=0.1, fit_prior=False)