{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n", "\n", "> Part 2: train a model\n", "\n", "![position of this step in the lifecycle](diagrams/scope-train.svg)\n", "> The blue boxes show the steps implemented in this notebook.\n", "\n", "In [Part 1](data.ipynb), we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3. \n", "\n", "In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "MODEL_KEY = \"small-domain-prediction\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load data that has been extracted in [part 1](data.ipynb)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m2022-06-19 15:08:22,338 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n", "\u001b[38;5;226m2022-06-19 15:08:22,338 | WARNING | The selected persistence driver (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n", "\u001b[38;5;39m2022-06-19 15:08:22,339 | INFO | Options: configured ✅\u001b[0m\n" ] } ], "source": [ "from great_ai import query_ground_truth\n", "\n", "data = query_ground_truth(\"train\")\n", "X = [d.input for d in data for domain in d.feedback]\n", "y = [domain for d in data for domain in d.feedback]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "alignmentgroup": "True", "hovertemplate": "x=%{x}
y=%{y}", "legendgroup": "", "marker": { "color": "#636efa", "pattern": { "shape": "" } }, "name": "", "offsetgroup": "", "orientation": "v", "showlegend": false, "textposition": "auto", "type": "bar", "x": [ "Medicine", "Computer Science", "Biology", "Chemistry", "Materials Science", "Engineering", "Physics", "Psychology", "Mathematics", "Business", "Political Science", "Sociology", "Economics", "Geography", "Environmental Science", "Geology", "Art", "History", "Philosophy" ], "xaxis": "x", "y": [ 16996, 5836, 5354, 5135, 4200, 3663, 2643, 2463, 2386, 1362, 1304, 1297, 1087, 1060, 968, 943, 515, 497, 332 ], "yaxis": "y" } ], "layout": { "barmode": "relative", "height": 400, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "width": 1200, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "x" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "y" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "from collections import Counter\n", "import plotly.express as px\n", "\n", "df = pd.DataFrame(Counter(y).most_common(), columns=[\"domain\", \"count\"])\n", "px.bar(x=df[\"domain\"], y=df[\"count\"], width=1200, height=400).show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimise and train Multinomial Naive Bayes classifier" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import MultinomialNB\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "\n", "\n", "def create_pipeline() -> Pipeline:\n", " return Pipeline(\n", " steps=[\n", " (\"vectorizer\", TfidfVectorizer(sublinear_tf=True)),\n", " (\"classifier\", MultinomialNB()),\n", " ]\n", " )" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 3 folds for each of 24 candidates, totalling 72 fits\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_classifier__alphaparam_classifier__fit_priorparam_vectorizer__max_dfparam_vectorizer__min_dfparamssplit0_test_scoresplit1_test_scoresplit2_test_scoremean_test_scorestd_test_scorerank_test_score
77.7969240.3213143.7560430.0278600.5False0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.5080130.5090860.5144550.5105180.0028181
108.0556640.2069843.7485170.0880120.5False0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.5037290.5064170.5118950.5073470.0033982
117.7483600.4843613.8632160.0720480.5False0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.5022110.4989490.5037440.5016350.0020003
87.4006490.0873203.6584420.0117350.5False0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.5014320.4939700.5013860.4989290.0035074
198.1479690.4019803.9771190.2840281False0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.4864100.4918910.4925150.4902720.0027435
207.4724140.1303203.7711360.1464061False0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.4868680.4891420.4926650.4895580.0023856
237.3955850.3261622.3320310.2541461False0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.4894890.4899870.4885430.4893400.0005997
227.4520600.1620722.9374730.1164431False0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.4787480.4851740.4846850.4828690.0029218
67.8363800.3746694.0074290.2511990.5False0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4727930.4764600.4795830.4762790.0027759
27.8394440.1749643.9141050.3797350.5True0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4692240.4721790.4767580.4727200.00310010
57.9484540.4113643.9684440.0900300.5True0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4693100.4719160.4769000.4727080.00314911
97.3735920.1430283.7776980.0089900.5False0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4631560.4667830.4634600.4644660.00164312
17.4068390.0419733.8386340.1166340.5True0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4623150.4629750.4631920.4628270.00037313
47.4815330.2243443.7898210.0980980.5True0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4575040.4608530.4599410.4594330.00141414
147.7497250.4684694.1395340.1401731True0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.4429300.4431830.4495770.4452300.00307615
177.7038460.3575084.0341540.2655091True0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.4390580.4434290.4493360.4439410.00421216
187.5531050.0560774.0949780.1988131False0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.4214880.4275300.4223290.4237820.00267217
137.4380300.2375453.9158240.0298231True0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.4041520.4113730.4068580.4074610.00297918
218.0571890.4350523.3052220.3489951False0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.3971820.4052470.4015050.4013110.00329519
167.7107480.5592834.0344210.0735221True0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.3920460.3979950.3961140.3953850.00248320
07.6146470.4662523.8982200.0986180.5True0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3851560.3888660.3865090.3868440.00153321
38.1768930.2528214.0719520.2657730.5True0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3710790.3770720.3745310.3742280.00245622
127.6064350.2390013.8757930.1092251True0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.2770310.2880650.2879130.2843360.00516623
158.0779710.7337004.1357210.2353071True0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.2604150.2672010.2669810.2648660.00314824
\n", "
" ], "text/plain": [ " mean_fit_time std_fit_time mean_score_time std_score_time \\\n", "7 7.796924 0.321314 3.756043 0.027860 \n", "10 8.055664 0.206984 3.748517 0.088012 \n", "11 7.748360 0.484361 3.863216 0.072048 \n", "8 7.400649 0.087320 3.658442 0.011735 \n", "19 8.147969 0.401980 3.977119 0.284028 \n", "20 7.472414 0.130320 3.771136 0.146406 \n", "23 7.395585 0.326162 2.332031 0.254146 \n", "22 7.452060 0.162072 2.937473 0.116443 \n", "6 7.836380 0.374669 4.007429 0.251199 \n", "2 7.839444 0.174964 3.914105 0.379735 \n", "5 7.948454 0.411364 3.968444 0.090030 \n", "9 7.373592 0.143028 3.777698 0.008990 \n", "1 7.406839 0.041973 3.838634 0.116634 \n", "4 7.481533 0.224344 3.789821 0.098098 \n", "14 7.749725 0.468469 4.139534 0.140173 \n", "17 7.703846 0.357508 4.034154 0.265509 \n", "18 7.553105 0.056077 4.094978 0.198813 \n", "13 7.438030 0.237545 3.915824 0.029823 \n", "21 8.057189 0.435052 3.305222 0.348995 \n", "16 7.710748 0.559283 4.034421 0.073522 \n", "0 7.614647 0.466252 3.898220 0.098618 \n", "3 8.176893 0.252821 4.071952 0.265773 \n", "12 7.606435 0.239001 3.875793 0.109225 \n", "15 8.077971 0.733700 4.135721 0.235307 \n", "\n", " param_classifier__alpha param_classifier__fit_prior \\\n", "7 0.5 False \n", "10 0.5 False \n", "11 0.5 False \n", "8 0.5 False \n", "19 1 False \n", "20 1 False \n", "23 1 False \n", "22 1 False \n", "6 0.5 False \n", "2 0.5 True \n", "5 0.5 True \n", "9 0.5 False \n", "1 0.5 True \n", "4 0.5 True \n", "14 1 True \n", "17 1 True \n", "18 1 False \n", "13 1 True \n", "21 1 False \n", "16 1 True \n", "0 0.5 True \n", "3 0.5 True \n", "12 1 True \n", "15 1 True \n", "\n", " param_vectorizer__max_df param_vectorizer__min_df \\\n", "7 0.05 20 \n", "10 0.1 20 \n", "11 0.1 100 \n", "8 0.05 100 \n", "19 0.05 20 \n", "20 0.05 100 \n", "23 0.1 100 \n", "22 0.1 20 \n", "6 0.05 5 \n", "2 0.05 100 \n", "5 0.1 100 \n", "9 0.1 5 \n", "1 0.05 20 \n", "4 0.1 20 \n", "14 0.05 100 \n", "17 0.1 100 \n", "18 0.05 5 \n", "13 0.05 20 \n", "21 0.1 5 \n", "16 0.1 20 \n", "0 0.05 5 \n", "3 0.1 5 \n", "12 0.05 5 \n", "15 0.1 5 \n", "\n", " params split0_test_score \\\n", "7 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.508013 \n", "10 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.503729 \n", "11 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.502211 \n", "8 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.501432 \n", "19 {'classifier__alpha': 1, 'classifier__fit_prio... 0.486410 \n", "20 {'classifier__alpha': 1, 'classifier__fit_prio... 0.486868 \n", "23 {'classifier__alpha': 1, 'classifier__fit_prio... 0.489489 \n", "22 {'classifier__alpha': 1, 'classifier__fit_prio... 0.478748 \n", "6 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.472793 \n", "2 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.469224 \n", "5 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.469310 \n", "9 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.463156 \n", "1 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.462315 \n", "4 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.457504 \n", "14 {'classifier__alpha': 1, 'classifier__fit_prio... 0.442930 \n", "17 {'classifier__alpha': 1, 'classifier__fit_prio... 0.439058 \n", "18 {'classifier__alpha': 1, 'classifier__fit_prio... 0.421488 \n", "13 {'classifier__alpha': 1, 'classifier__fit_prio... 0.404152 \n", "21 {'classifier__alpha': 1, 'classifier__fit_prio... 0.397182 \n", "16 {'classifier__alpha': 1, 'classifier__fit_prio... 0.392046 \n", "0 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.385156 \n", "3 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.371079 \n", "12 {'classifier__alpha': 1, 'classifier__fit_prio... 0.277031 \n", "15 {'classifier__alpha': 1, 'classifier__fit_prio... 0.260415 \n", "\n", " split1_test_score split2_test_score mean_test_score std_test_score \\\n", "7 0.509086 0.514455 0.510518 0.002818 \n", "10 0.506417 0.511895 0.507347 0.003398 \n", "11 0.498949 0.503744 0.501635 0.002000 \n", "8 0.493970 0.501386 0.498929 0.003507 \n", "19 0.491891 0.492515 0.490272 0.002743 \n", "20 0.489142 0.492665 0.489558 0.002385 \n", "23 0.489987 0.488543 0.489340 0.000599 \n", "22 0.485174 0.484685 0.482869 0.002921 \n", "6 0.476460 0.479583 0.476279 0.002775 \n", "2 0.472179 0.476758 0.472720 0.003100 \n", "5 0.471916 0.476900 0.472708 0.003149 \n", "9 0.466783 0.463460 0.464466 0.001643 \n", "1 0.462975 0.463192 0.462827 0.000373 \n", "4 0.460853 0.459941 0.459433 0.001414 \n", "14 0.443183 0.449577 0.445230 0.003076 \n", "17 0.443429 0.449336 0.443941 0.004212 \n", "18 0.427530 0.422329 0.423782 0.002672 \n", "13 0.411373 0.406858 0.407461 0.002979 \n", "21 0.405247 0.401505 0.401311 0.003295 \n", "16 0.397995 0.396114 0.395385 0.002483 \n", "0 0.388866 0.386509 0.386844 0.001533 \n", "3 0.377072 0.374531 0.374228 0.002456 \n", "12 0.288065 0.287913 0.284336 0.005166 \n", "15 0.267201 0.266981 0.264866 0.003148 \n", "\n", " rank_test_score \n", "7 1 \n", "10 2 \n", "11 3 \n", "8 4 \n", "19 5 \n", "20 6 \n", "23 7 \n", "22 8 \n", "6 9 \n", "2 10 \n", "5 11 \n", "9 12 \n", "1 13 \n", "4 14 \n", "14 15 \n", "17 16 \n", "18 17 \n", "13 18 \n", "21 19 \n", "16 20 \n", "0 21 \n", "3 22 \n", "12 23 \n", "15 24 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "optimisation_pipeline = GridSearchCV(\n", " create_pipeline(),\n", " {\n", " \"vectorizer__min_df\": [5, 20, 100],\n", " \"vectorizer__max_df\": [0.05, 0.1],\n", " \"classifier__alpha\": [0.5, 1],\n", " \"classifier__fit_prior\": [True, False],\n", " },\n", " scoring=\"f1_macro\",\n", " cv=3,\n", " n_jobs=-1,\n", " verbose=1,\n", ")\n", "optimisation_pipeline.fit(X, y)\n", "\n", "results = pd.DataFrame(optimisation_pipeline.cv_results_)\n", "results.sort_values(\"rank_test_score\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('vectorizer',\n",
       "                 TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n",
       "                ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('vectorizer',\n", " TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n", " ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import set_config\n", "\n", "set_config(display=\"diagram\")\n", "\n", "classifier = create_pipeline()\n", "classifier.set_params(**optimisation_pipeline.best_params_)\n", "classifier.fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export the model using GreatAI" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m2022-06-19 15:12:58,312 | INFO | Fetching cached versions of small-domain-prediction\u001b[0m\n", "\u001b[38;5;39m2022-06-19 15:12:59,027 | INFO | Copying file for small-domain-prediction-12\u001b[0m\n", "\u001b[38;5;39m2022-06-19 15:12:59,039 | INFO | Compressing small-domain-prediction-12\u001b[0m\n", "\u001b[38;5;39m2022-06-19 15:12:59,842 | INFO | Model small-domain-prediction uploaded with version 12\u001b[0m\n" ] }, { "data": { "text/plain": [ "'small-domain-prediction:12'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from great_ai import save_model\n", "\n", "\n", "save_model(classifier, key=MODEL_KEY, keep_last_n=5)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad" } } }, "nbformat": 4, "nbformat_minor": 2 }