{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n", "\n", "> Part 2: train a model\n", "\n", "![position of this step in the lifecycle](../diagrams/scope-train.svg)\n", "> The blue boxes show the steps implemented in this notebook.\n", "\n", "In [Part 1](data.ipynb), we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3. \n", "\n", "In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load data that has been extracted in [part 1](data.ipynb)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m2022-06-25 14:50:29,879 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,880 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongodbDriver\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,881 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,881 | INFO | Settings: configured ✅\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,882 | INFO | 🔩 tracing_database: MongodbDriver\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,883 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,883 | INFO | 🔩 is_production: False\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,884 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,884 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:29,885 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n", "\u001b[38;5;226m2022-06-25 14:50:29,885 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n", "\u001b[38;5;226m2022-06-25 14:50:29,885 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n" ] } ], "source": [ "from great_ai import query_ground_truth\n", "\n", "data = query_ground_truth(\"train\")\n", "X = [d.input for d in data for domain in d.feedback]\n", "y = [domain for d in data for domain in d.feedback]" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "alignmentgroup": "True", "hovertemplate": "domain=%{x}
count=%{y}", "legendgroup": "", "marker": { "color": "#636efa", "pattern": { "shape": "" } }, "name": "", "offsetgroup": "", "orientation": "v", "showlegend": false, "textposition": "auto", "type": "bar", "x": [ "Medicine", "Computer Science", "Chemistry", "Biology", "Materials Science", "Engineering", "Physics", "Psychology", "Mathematics", "Sociology", "Business", "Political Science", "Economics", "Geology", "Geography", "Environmental Science", "History", "Art", "Philosophy" ], "xaxis": "x", "y": [ 4200, 1371, 1307, 1300, 1111, 876, 639, 617, 569, 334, 323, 311, 269, 256, 245, 244, 143, 137, 62 ], "yaxis": "y" } ], "layout": { "barmode": "relative", "height": 400, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "width": 1200, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "domain" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "count" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "from collections import Counter\n", "import plotly.express as px\n", "\n", "df = pd.DataFrame(Counter(y).most_common(), columns=[\"domain\", \"count\"])\n", "px.bar(df, \"domain\", \"count\", width=1200, height=400).show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimise and train Multinomial Naive Bayes classifier" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import MultinomialNB\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "\n", "\n", "def create_pipeline() -> Pipeline:\n", " return Pipeline(\n", " steps=[\n", " (\"vectorizer\", TfidfVectorizer(sublinear_tf=True)),\n", " (\"classifier\", MultinomialNB()),\n", " ]\n", " )" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 3 folds for each of 24 candidates, totalling 72 fits\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_classifier__alphaparam_classifier__fit_priorparam_vectorizer__max_dfparam_vectorizer__min_dfparamssplit0_test_scoresplit1_test_scoresplit2_test_scoremean_test_scorestd_test_scorerank_test_score
71.9622600.1474490.9353570.0636590.5False0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4850300.4638490.4818400.4769060.0093241
101.9426050.1110270.9523610.0668120.5False0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4828900.4595560.4793620.4739360.0102702
192.1451520.0689781.0022910.0473581False0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.4673300.4429940.4643020.4582080.0108293
221.9718880.1269500.7397950.0715511False0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.4548300.4229020.4506770.4428030.0141744
61.8612750.0133891.0589070.1111220.5False0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4561270.4224560.4438270.4408030.0139125
111.8253970.1057540.8922270.0570030.5False0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4382320.4404640.4226670.4337880.0079166
231.6933330.0096670.5014910.0065451False0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.4339150.4394700.4160310.4298050.0100017
82.0080450.1453300.9445590.1559250.5False0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4361780.4257240.4183960.4267660.0072978
201.7492000.0229590.8895320.0475171False0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.4282150.4253980.4110510.4215550.0075169
91.9608890.0980040.9859570.0809250.5False0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4306380.4066190.4202130.4191570.00983410
181.8077990.0648910.8818720.0308101False0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.4024020.3723530.3861890.3869810.01228011
12.0092320.0431250.8996760.0369770.5True0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3897970.3726190.3883580.3835910.00778112
41.8680870.0947391.0053530.1014660.5True0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3793230.3646670.3790600.3743500.00684813
211.9584300.0396390.8909630.0125461False0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.3669360.3433610.3638830.3580600.01046814
51.9406920.0183200.8988650.0306510.5True0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3548500.3492370.3427370.3489410.00495015
21.8556910.0295060.8664920.0380480.5True0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3490050.3418320.3286170.3398180.00844416
171.7985590.1034970.8882730.0690501True0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.3123320.2976550.3074710.3058190.00610417
142.0160410.2321380.9676300.1461441True0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.3049420.2969210.2971210.2996610.00373518
131.8295130.1126450.8858480.0277261True0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.3015390.2853960.2972720.2947360.00683019
01.9053620.0180520.8855520.0239850.5True0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.2956350.2767590.2962700.2895550.00905220
161.7936880.0499950.9213010.0609801True0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.2867460.2722600.2770840.2786960.00602321
32.0785680.0455490.9636910.0482810.5True0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.2764520.2655090.2689490.2703030.00456922
121.8395060.0489100.9218120.0103741True0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.1831960.1861440.1803230.1832210.00237623
151.9092790.1106391.0560870.1297381True0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.1652770.1658400.1670880.1660680.00075724
\n", "
" ], "text/plain": [ " mean_fit_time std_fit_time mean_score_time std_score_time \\\n", "7 1.962260 0.147449 0.935357 0.063659 \n", "10 1.942605 0.111027 0.952361 0.066812 \n", "19 2.145152 0.068978 1.002291 0.047358 \n", "22 1.971888 0.126950 0.739795 0.071551 \n", "6 1.861275 0.013389 1.058907 0.111122 \n", "11 1.825397 0.105754 0.892227 0.057003 \n", "23 1.693333 0.009667 0.501491 0.006545 \n", "8 2.008045 0.145330 0.944559 0.155925 \n", "20 1.749200 0.022959 0.889532 0.047517 \n", "9 1.960889 0.098004 0.985957 0.080925 \n", "18 1.807799 0.064891 0.881872 0.030810 \n", "1 2.009232 0.043125 0.899676 0.036977 \n", "4 1.868087 0.094739 1.005353 0.101466 \n", "21 1.958430 0.039639 0.890963 0.012546 \n", "5 1.940692 0.018320 0.898865 0.030651 \n", "2 1.855691 0.029506 0.866492 0.038048 \n", "17 1.798559 0.103497 0.888273 0.069050 \n", "14 2.016041 0.232138 0.967630 0.146144 \n", "13 1.829513 0.112645 0.885848 0.027726 \n", "0 1.905362 0.018052 0.885552 0.023985 \n", "16 1.793688 0.049995 0.921301 0.060980 \n", "3 2.078568 0.045549 0.963691 0.048281 \n", "12 1.839506 0.048910 0.921812 0.010374 \n", "15 1.909279 0.110639 1.056087 0.129738 \n", "\n", " param_classifier__alpha param_classifier__fit_prior \\\n", "7 0.5 False \n", "10 0.5 False \n", "19 1 False \n", "22 1 False \n", "6 0.5 False \n", "11 0.5 False \n", "23 1 False \n", "8 0.5 False \n", "20 1 False \n", "9 0.5 False \n", "18 1 False \n", "1 0.5 True \n", "4 0.5 True \n", "21 1 False \n", "5 0.5 True \n", "2 0.5 True \n", "17 1 True \n", "14 1 True \n", "13 1 True \n", "0 0.5 True \n", "16 1 True \n", "3 0.5 True \n", "12 1 True \n", "15 1 True \n", "\n", " param_vectorizer__max_df param_vectorizer__min_df \\\n", "7 0.05 20 \n", "10 0.1 20 \n", "19 0.05 20 \n", "22 0.1 20 \n", "6 0.05 5 \n", "11 0.1 100 \n", "23 0.1 100 \n", "8 0.05 100 \n", "20 0.05 100 \n", "9 0.1 5 \n", "18 0.05 5 \n", "1 0.05 20 \n", "4 0.1 20 \n", "21 0.1 5 \n", "5 0.1 100 \n", "2 0.05 100 \n", "17 0.1 100 \n", "14 0.05 100 \n", "13 0.05 20 \n", "0 0.05 5 \n", "16 0.1 20 \n", "3 0.1 5 \n", "12 0.05 5 \n", "15 0.1 5 \n", "\n", " params split0_test_score \\\n", "7 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.485030 \n", "10 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.482890 \n", "19 {'classifier__alpha': 1, 'classifier__fit_prio... 0.467330 \n", "22 {'classifier__alpha': 1, 'classifier__fit_prio... 0.454830 \n", "6 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.456127 \n", "11 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.438232 \n", "23 {'classifier__alpha': 1, 'classifier__fit_prio... 0.433915 \n", "8 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.436178 \n", "20 {'classifier__alpha': 1, 'classifier__fit_prio... 0.428215 \n", "9 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.430638 \n", "18 {'classifier__alpha': 1, 'classifier__fit_prio... 0.402402 \n", "1 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.389797 \n", "4 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.379323 \n", "21 {'classifier__alpha': 1, 'classifier__fit_prio... 0.366936 \n", "5 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.354850 \n", "2 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.349005 \n", "17 {'classifier__alpha': 1, 'classifier__fit_prio... 0.312332 \n", "14 {'classifier__alpha': 1, 'classifier__fit_prio... 0.304942 \n", "13 {'classifier__alpha': 1, 'classifier__fit_prio... 0.301539 \n", "0 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.295635 \n", "16 {'classifier__alpha': 1, 'classifier__fit_prio... 0.286746 \n", "3 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.276452 \n", "12 {'classifier__alpha': 1, 'classifier__fit_prio... 0.183196 \n", "15 {'classifier__alpha': 1, 'classifier__fit_prio... 0.165277 \n", "\n", " split1_test_score split2_test_score mean_test_score std_test_score \\\n", "7 0.463849 0.481840 0.476906 0.009324 \n", "10 0.459556 0.479362 0.473936 0.010270 \n", "19 0.442994 0.464302 0.458208 0.010829 \n", "22 0.422902 0.450677 0.442803 0.014174 \n", "6 0.422456 0.443827 0.440803 0.013912 \n", "11 0.440464 0.422667 0.433788 0.007916 \n", "23 0.439470 0.416031 0.429805 0.010001 \n", "8 0.425724 0.418396 0.426766 0.007297 \n", "20 0.425398 0.411051 0.421555 0.007516 \n", "9 0.406619 0.420213 0.419157 0.009834 \n", "18 0.372353 0.386189 0.386981 0.012280 \n", "1 0.372619 0.388358 0.383591 0.007781 \n", "4 0.364667 0.379060 0.374350 0.006848 \n", "21 0.343361 0.363883 0.358060 0.010468 \n", "5 0.349237 0.342737 0.348941 0.004950 \n", "2 0.341832 0.328617 0.339818 0.008444 \n", "17 0.297655 0.307471 0.305819 0.006104 \n", "14 0.296921 0.297121 0.299661 0.003735 \n", "13 0.285396 0.297272 0.294736 0.006830 \n", "0 0.276759 0.296270 0.289555 0.009052 \n", "16 0.272260 0.277084 0.278696 0.006023 \n", "3 0.265509 0.268949 0.270303 0.004569 \n", "12 0.186144 0.180323 0.183221 0.002376 \n", "15 0.165840 0.167088 0.166068 0.000757 \n", "\n", " rank_test_score \n", "7 1 \n", "10 2 \n", "19 3 \n", "22 4 \n", "6 5 \n", "11 6 \n", "23 7 \n", "8 8 \n", "20 9 \n", "9 10 \n", "18 11 \n", "1 12 \n", "4 13 \n", "21 14 \n", "5 15 \n", "2 16 \n", "17 17 \n", "14 18 \n", "13 19 \n", "0 20 \n", "16 21 \n", "3 22 \n", "12 23 \n", "15 24 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "optimisation_pipeline = GridSearchCV(\n", " create_pipeline(),\n", " {\n", " \"vectorizer__min_df\": [5, 20, 100],\n", " \"vectorizer__max_df\": [0.05, 0.1],\n", " \"classifier__alpha\": [0.5, 1],\n", " \"classifier__fit_prior\": [True, False],\n", " },\n", " scoring=\"f1_macro\",\n", " cv=3,\n", " n_jobs=-1,\n", " verbose=1,\n", ")\n", "optimisation_pipeline.fit(X, y)\n", "\n", "results = pd.DataFrame(optimisation_pipeline.cv_results_)\n", "results.sort_values(\"rank_test_score\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('vectorizer',\n",
       "                 TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n",
       "                ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('vectorizer',\n", " TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n", " ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import set_config\n", "\n", "set_config(display=\"diagram\")\n", "\n", "classifier = create_pipeline()\n", "classifier.set_params(**optimisation_pipeline.best_params_)\n", "classifier.fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export the model using GreatAI" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m2022-06-25 14:50:53,592 | INFO | Copying file for small-domain-prediction-0\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,613 | INFO | Compressing small-domain-prediction-0\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,917 | INFO | Uploading /tmp/tmpvxez8op8/small-domain-prediction-0.tar.gz to Mongo (GridFS)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,972 | INFO | Uploading small-domain-prediction-0.tar.gz 0.26/1.85 MB (14.1%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,974 | INFO | Uploading small-domain-prediction-0.tar.gz 0.52/1.85 MB (28.2%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,975 | INFO | Uploading small-domain-prediction-0.tar.gz 0.78/1.85 MB (42.3%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,977 | INFO | Uploading small-domain-prediction-0.tar.gz 1.04/1.85 MB (56.4%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,979 | INFO | Uploading small-domain-prediction-0.tar.gz 1.31/1.85 MB (70.5%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,980 | INFO | Uploading small-domain-prediction-0.tar.gz 1.57/1.85 MB (84.7%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,982 | INFO | Uploading small-domain-prediction-0.tar.gz 1.83/1.85 MB (98.8%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,982 | INFO | Uploading small-domain-prediction-0.tar.gz 1.85/1.85 MB (100.0%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 14:50:53,985 | INFO | Model small-domain-prediction uploaded with version 0\u001b[0m\n" ] }, { "data": { "text/plain": [ "'small-domain-prediction:0'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from great_ai import save_model\n", "\n", "\n", "save_model(classifier, key=\"small-domain-prediction\", keep_last_n=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Next: [Part 3](deploy.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "c1f394f9662881005685eeb18d8f9f77079b1b8b9a5ece1f825bfa01fcb7f52f" } } }, "nbformat": 4, "nbformat_minor": 2 }