{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n", "\n", "> Part 2: train a model\n", "\n", "![position of this step in the lifecycle](diagrams/scope-train.svg)\n", "> The blue boxes show the steps implemented in this notebook.\n", "\n", "In [Part 1](data.ipynb), we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3. \n", "\n", "In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load data that has been extracted in [part 1](data.ipynb)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m2022-06-25 11:25:08,430 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,431 | INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising MongodbDriver\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,432 | INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising LargeFileMongo\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,432 | INFO | Settings: configured ✅\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,433 | INFO | 🔩 tracing_database: MongodbDriver\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,433 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,434 | INFO | 🔩 is_production: False\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,434 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:08,434 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n", "\u001b[38;5;226m2022-06-25 11:25:08,435 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n", "\u001b[38;5;226m2022-06-25 11:25:08,435 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'filter': {'$and': [{'tags': 'train'}, {'feedback': {'$ne': None}}]}, 'sort': []}\n" ] } ], "source": [ "from great_ai import query_ground_truth\n", "\n", "data = query_ground_truth(\"train\")\n", "X = [d.input for d in data for domain in d.feedback]\n", "y = [domain for d in data for domain in d.feedback]" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "alignmentgroup": "True", "hovertemplate": "domain=%{x}
count=%{y}", "legendgroup": "", "marker": { "color": "#636efa", "pattern": { "shape": "" } }, "name": "", "offsetgroup": "", "orientation": "v", "showlegend": false, "textposition": "auto", "type": "bar", "x": [ "Medicine", "Computer Science", "Biology", "Chemistry", "Materials Science", "Engineering", "Physics", "Psychology", "Mathematics", "Political Science", "Business", "Sociology", "Economics", "Geography", "Geology", "Environmental Science", "History", "Art", "Philosophy" ], "xaxis": "x", "y": [ 4409, 1422, 1369, 1352, 1026, 887, 638, 619, 582, 340, 333, 303, 256, 254, 229, 217, 113, 112, 65 ], "yaxis": "y" } ], "layout": { "barmode": "relative", "height": 400, "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "width": 1200, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "domain" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "count" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "from collections import Counter\n", "import plotly.express as px\n", "\n", "df = pd.DataFrame(Counter(y).most_common(), columns=[\"domain\", \"count\"])\n", "px.bar(df, \"domain\", \"count\", width=1200, height=400).show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimise and train Multinomial Naive Bayes classifier" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import MultinomialNB\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "\n", "\n", "def create_pipeline() -> Pipeline:\n", " return Pipeline(\n", " steps=[\n", " (\"vectorizer\", TfidfVectorizer(sublinear_tf=True)),\n", " (\"classifier\", MultinomialNB()),\n", " ]\n", " )" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 3 folds for each of 24 candidates, totalling 72 fits\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_classifier__alphaparam_classifier__fit_priorparam_vectorizer__max_dfparam_vectorizer__min_dfparamssplit0_test_scoresplit1_test_scoresplit2_test_scoremean_test_scorestd_test_scorerank_test_score
71.9865880.0508961.0902510.1355080.5False0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4591650.4730240.4754620.4692170.0071771
102.0703330.0383960.9763150.0337420.5False0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4575240.4635750.4580070.4597020.0027452
192.0491660.1931131.0645760.0870341False0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.4416570.4529330.4512860.4486250.0049733
62.2881450.1313381.1334450.0995830.5False0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4369760.4496930.4379110.4415270.0057874
221.8722340.0729970.7480700.0858821False0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.4323220.4388050.4282090.4331120.0043625
112.0676910.1269230.9109470.0787480.5False0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4264360.4291820.4374100.4310090.0046626
231.8473300.1475040.4953540.0185891False0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.4221300.4308750.4308290.4279450.0041127
92.0714890.2560861.0559360.0371980.5False0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4167460.4259380.4173810.4200220.0041928
201.7765460.0646770.8884850.0933021False0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.4134410.4171220.4271960.4192530.0058149
82.0159920.0625830.9744340.0825820.5False0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.4125220.4104090.4250470.4159930.00646010
182.0320500.1043130.9692100.0709971False0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.3923930.3961430.3864390.3916590.00399611
12.3076110.1429021.1021780.0787550.5True0.0520{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3858340.3922250.3665490.3815360.01091412
41.8769580.0506481.0319750.0293410.5True0.120{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3773090.3842240.3666320.3760550.00723613
211.8974790.0949030.7405160.1438211False0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.3605510.3663110.3504390.3591010.00656114
51.8249820.0381130.9608140.0413300.5True0.1100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3478310.3507080.3499630.3495010.00121915
22.1514460.1204400.8876200.0379000.5True0.05100{'classifier__alpha': 0.5, 'classifier__fit_pr...0.3350930.3400200.3415020.3388720.00273916
172.0049610.1966400.9534820.1139141True0.1100{'classifier__alpha': 1, 'classifier__fit_prio...0.3000930.3060200.3050470.3037200.00259517
131.8907090.0562341.0062190.0426281True0.0520{'classifier__alpha': 1, 'classifier__fit_prio...0.2971190.3020610.2989750.2993850.00203818
142.0224320.2017450.9318370.0474671True0.05100{'classifier__alpha': 1, 'classifier__fit_prio...0.3007460.2914920.2974560.2965650.00383019
02.0815720.1238751.0586920.0340840.5True0.055{'classifier__alpha': 0.5, 'classifier__fit_pr...0.2966670.2947150.2921440.2945090.00185220
161.9617000.1089770.9648160.0897351True0.120{'classifier__alpha': 1, 'classifier__fit_prio...0.2783890.2871340.2863200.2839480.00394521
32.2727790.1558970.9825280.0457440.5True0.15{'classifier__alpha': 0.5, 'classifier__fit_pr...0.2758330.2726930.2784270.2756510.00234422
121.9798260.1065240.9388290.0509751True0.055{'classifier__alpha': 1, 'classifier__fit_prio...0.1813090.1867070.1934060.1871410.00494823
151.9528170.1079000.9388510.0482951True0.15{'classifier__alpha': 1, 'classifier__fit_prio...0.1604020.1720250.1714550.1679600.00535024
\n", "
" ], "text/plain": [ " mean_fit_time std_fit_time mean_score_time std_score_time \\\n", "7 1.986588 0.050896 1.090251 0.135508 \n", "10 2.070333 0.038396 0.976315 0.033742 \n", "19 2.049166 0.193113 1.064576 0.087034 \n", "6 2.288145 0.131338 1.133445 0.099583 \n", "22 1.872234 0.072997 0.748070 0.085882 \n", "11 2.067691 0.126923 0.910947 0.078748 \n", "23 1.847330 0.147504 0.495354 0.018589 \n", "9 2.071489 0.256086 1.055936 0.037198 \n", "20 1.776546 0.064677 0.888485 0.093302 \n", "8 2.015992 0.062583 0.974434 0.082582 \n", "18 2.032050 0.104313 0.969210 0.070997 \n", "1 2.307611 0.142902 1.102178 0.078755 \n", "4 1.876958 0.050648 1.031975 0.029341 \n", "21 1.897479 0.094903 0.740516 0.143821 \n", "5 1.824982 0.038113 0.960814 0.041330 \n", "2 2.151446 0.120440 0.887620 0.037900 \n", "17 2.004961 0.196640 0.953482 0.113914 \n", "13 1.890709 0.056234 1.006219 0.042628 \n", "14 2.022432 0.201745 0.931837 0.047467 \n", "0 2.081572 0.123875 1.058692 0.034084 \n", "16 1.961700 0.108977 0.964816 0.089735 \n", "3 2.272779 0.155897 0.982528 0.045744 \n", "12 1.979826 0.106524 0.938829 0.050975 \n", "15 1.952817 0.107900 0.938851 0.048295 \n", "\n", " param_classifier__alpha param_classifier__fit_prior \\\n", "7 0.5 False \n", "10 0.5 False \n", "19 1 False \n", "6 0.5 False \n", "22 1 False \n", "11 0.5 False \n", "23 1 False \n", "9 0.5 False \n", "20 1 False \n", "8 0.5 False \n", "18 1 False \n", "1 0.5 True \n", "4 0.5 True \n", "21 1 False \n", "5 0.5 True \n", "2 0.5 True \n", "17 1 True \n", "13 1 True \n", "14 1 True \n", "0 0.5 True \n", "16 1 True \n", "3 0.5 True \n", "12 1 True \n", "15 1 True \n", "\n", " param_vectorizer__max_df param_vectorizer__min_df \\\n", "7 0.05 20 \n", "10 0.1 20 \n", "19 0.05 20 \n", "6 0.05 5 \n", "22 0.1 20 \n", "11 0.1 100 \n", "23 0.1 100 \n", "9 0.1 5 \n", "20 0.05 100 \n", "8 0.05 100 \n", "18 0.05 5 \n", "1 0.05 20 \n", "4 0.1 20 \n", "21 0.1 5 \n", "5 0.1 100 \n", "2 0.05 100 \n", "17 0.1 100 \n", "13 0.05 20 \n", "14 0.05 100 \n", "0 0.05 5 \n", "16 0.1 20 \n", "3 0.1 5 \n", "12 0.05 5 \n", "15 0.1 5 \n", "\n", " params split0_test_score \\\n", "7 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.459165 \n", "10 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.457524 \n", "19 {'classifier__alpha': 1, 'classifier__fit_prio... 0.441657 \n", "6 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.436976 \n", "22 {'classifier__alpha': 1, 'classifier__fit_prio... 0.432322 \n", "11 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.426436 \n", "23 {'classifier__alpha': 1, 'classifier__fit_prio... 0.422130 \n", "9 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.416746 \n", "20 {'classifier__alpha': 1, 'classifier__fit_prio... 0.413441 \n", "8 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.412522 \n", "18 {'classifier__alpha': 1, 'classifier__fit_prio... 0.392393 \n", "1 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.385834 \n", "4 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.377309 \n", "21 {'classifier__alpha': 1, 'classifier__fit_prio... 0.360551 \n", "5 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.347831 \n", "2 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.335093 \n", "17 {'classifier__alpha': 1, 'classifier__fit_prio... 0.300093 \n", "13 {'classifier__alpha': 1, 'classifier__fit_prio... 0.297119 \n", "14 {'classifier__alpha': 1, 'classifier__fit_prio... 0.300746 \n", "0 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.296667 \n", "16 {'classifier__alpha': 1, 'classifier__fit_prio... 0.278389 \n", "3 {'classifier__alpha': 0.5, 'classifier__fit_pr... 0.275833 \n", "12 {'classifier__alpha': 1, 'classifier__fit_prio... 0.181309 \n", "15 {'classifier__alpha': 1, 'classifier__fit_prio... 0.160402 \n", "\n", " split1_test_score split2_test_score mean_test_score std_test_score \\\n", "7 0.473024 0.475462 0.469217 0.007177 \n", "10 0.463575 0.458007 0.459702 0.002745 \n", "19 0.452933 0.451286 0.448625 0.004973 \n", "6 0.449693 0.437911 0.441527 0.005787 \n", "22 0.438805 0.428209 0.433112 0.004362 \n", "11 0.429182 0.437410 0.431009 0.004662 \n", "23 0.430875 0.430829 0.427945 0.004112 \n", "9 0.425938 0.417381 0.420022 0.004192 \n", "20 0.417122 0.427196 0.419253 0.005814 \n", "8 0.410409 0.425047 0.415993 0.006460 \n", "18 0.396143 0.386439 0.391659 0.003996 \n", "1 0.392225 0.366549 0.381536 0.010914 \n", "4 0.384224 0.366632 0.376055 0.007236 \n", "21 0.366311 0.350439 0.359101 0.006561 \n", "5 0.350708 0.349963 0.349501 0.001219 \n", "2 0.340020 0.341502 0.338872 0.002739 \n", "17 0.306020 0.305047 0.303720 0.002595 \n", "13 0.302061 0.298975 0.299385 0.002038 \n", "14 0.291492 0.297456 0.296565 0.003830 \n", "0 0.294715 0.292144 0.294509 0.001852 \n", "16 0.287134 0.286320 0.283948 0.003945 \n", "3 0.272693 0.278427 0.275651 0.002344 \n", "12 0.186707 0.193406 0.187141 0.004948 \n", "15 0.172025 0.171455 0.167960 0.005350 \n", "\n", " rank_test_score \n", "7 1 \n", "10 2 \n", "19 3 \n", "6 4 \n", "22 5 \n", "11 6 \n", "23 7 \n", "9 8 \n", "20 9 \n", "8 10 \n", "18 11 \n", "1 12 \n", "4 13 \n", "21 14 \n", "5 15 \n", "2 16 \n", "17 17 \n", "13 18 \n", "14 19 \n", "0 20 \n", "16 21 \n", "3 22 \n", "12 23 \n", "15 24 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "optimisation_pipeline = GridSearchCV(\n", " create_pipeline(),\n", " {\n", " \"vectorizer__min_df\": [5, 20, 100],\n", " \"vectorizer__max_df\": [0.05, 0.1],\n", " \"classifier__alpha\": [0.5, 1],\n", " \"classifier__fit_prior\": [True, False],\n", " },\n", " scoring=\"f1_macro\",\n", " cv=3,\n", " n_jobs=-1,\n", " verbose=1,\n", ")\n", "optimisation_pipeline.fit(X, y)\n", "\n", "results = pd.DataFrame(optimisation_pipeline.cv_results_)\n", "results.sort_values(\"rank_test_score\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('vectorizer',\n",
       "                 TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n",
       "                ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('vectorizer',\n", " TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),\n", " ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import set_config\n", "\n", "set_config(display=\"diagram\")\n", "\n", "classifier = create_pipeline()\n", "classifier.set_params(**optimisation_pipeline.best_params_)\n", "classifier.fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export the model using GreatAI" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39m2022-06-25 11:25:32,714 | INFO | Copying file for small-domain-prediction-0\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:32,737 | INFO | Compressing small-domain-prediction-0\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,050 | INFO | Uploading /tmp/tmpgerx8x95/small-domain-prediction-0.tar.gz to Mongo (GridFS)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,107 | INFO | Uploading small-domain-prediction-0.tar.gz 0.26/1.85 MB (14.2%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,109 | INFO | Uploading small-domain-prediction-0.tar.gz 0.52/1.85 MB (28.3%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,112 | INFO | Uploading small-domain-prediction-0.tar.gz 0.78/1.85 MB (42.5%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,114 | INFO | Uploading small-domain-prediction-0.tar.gz 1.04/1.85 MB (56.6%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,116 | INFO | Uploading small-domain-prediction-0.tar.gz 1.31/1.85 MB (70.8%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,117 | INFO | Uploading small-domain-prediction-0.tar.gz 1.57/1.85 MB (84.9%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,120 | INFO | Uploading small-domain-prediction-0.tar.gz 1.83/1.85 MB (99.1%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,120 | INFO | Uploading small-domain-prediction-0.tar.gz 1.85/1.85 MB (100.0%)\u001b[0m\n", "\u001b[38;5;39m2022-06-25 11:25:33,124 | INFO | Model small-domain-prediction uploaded with version 0\u001b[0m\n" ] }, { "data": { "text/plain": [ "'small-domain-prediction:0'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from great_ai import save_model\n", "\n", "\n", "save_model(classifier, key=\"small-domain-prediction\", keep_last_n=5)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad" } } }, "nbformat": 4, "nbformat_minor": 2 }