Train a domain classifier on the semantic scholar dataset¶

Part 2: train a model

The blue boxes show the steps implemented in this notebook.

In Part 1, we have cleaned and transformed our training data. We can now access this data using great_ai.LargeFile. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3.

In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using great_ai.save_model.

Load data that has been extracted in part 1¶

In [1]:

from great_ai import query_ground_truth

data = query_ground_truth("train")
X = [d.input for d in data for domain in d.feedback]
y = [domain for d in data for domain in d.feedback]

from great_ai import query_ground_truth data = query_ground_truth("train") X = [d.input for d in data for domain in d.feedback] y = [domain for d in data for domain in d.feedback]

2022-06-25 14:50:29,879 |  WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️
2022-06-25 14:50:29,880 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongodbDriver
2022-06-25 14:50:29,881 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo
2022-06-25 14:50:29,881 |     INFO | Settings: configured ✅
2022-06-25 14:50:29,882 |     INFO | 🔩 tracing_database: MongodbDriver
2022-06-25 14:50:29,883 |     INFO | 🔩 large_file_implementation: LargeFileMongo
2022-06-25 14:50:29,883 |     INFO | 🔩 is_production: False
2022-06-25 14:50:29,884 |     INFO | 🔩 should_log_exception_stack: True
2022-06-25 14:50:29,884 |     INFO | 🔩 prediction_cache_size: 512
2022-06-25 14:50:29,885 |     INFO | 🔩 dashboard_table_size: 50
2022-06-25 14:50:29,885 |  WARNING | You still need to check whether you follow all best practices before trusting your deployment.
2022-06-25 14:50:29,885 |  WARNING | > Find out more at https://se-ml.github.io/practices/

In [2]:

import pandas as pd
from collections import Counter
import plotly.express as px

df = pd.DataFrame(Counter(y).most_common(), columns=["domain", "count"])
px.bar(df, "domain", "count", width=1200, height=400).show()

import pandas as pd from collections import Counter import plotly.express as px df = pd.DataFrame(Counter(y).most_common(), columns=["domain", "count"]) px.bar(df, "domain", "count", width=1200, height=400).show()

Optimise and train Multinomial Naive Bayes classifier¶

In [3]:

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer


def create_pipeline() -> Pipeline:
    return Pipeline(
        steps=[
            ("vectorizer", TfidfVectorizer(sublinear_tf=True)),
            ("classifier", MultinomialNB()),
        ]
    )

from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import Pipeline from sklearn.feature_extraction.text import TfidfVectorizer def create_pipeline() -> Pipeline: return Pipeline( steps=[ ("vectorizer", TfidfVectorizer(sublinear_tf=True)), ("classifier", MultinomialNB()), ] )

In [4]:

from sklearn.model_selection import GridSearchCV

optimisation_pipeline = GridSearchCV(
    create_pipeline(),
    {
        "vectorizer__min_df": [5, 20, 100],
        "vectorizer__max_df": [0.05, 0.1],
        "classifier__alpha": [0.5, 1],
        "classifier__fit_prior": [True, False],
    },
    scoring="f1_macro",
    cv=3,
    n_jobs=-1,
    verbose=1,
)
optimisation_pipeline.fit(X, y)

results = pd.DataFrame(optimisation_pipeline.cv_results_)
results.sort_values("rank_test_score")

from sklearn.model_selection import GridSearchCV optimisation_pipeline = GridSearchCV( create_pipeline(), { "vectorizer__min_df": [5, 20, 100], "vectorizer__max_df": [0.05, 0.1], "classifier__alpha": [0.5, 1], "classifier__fit_prior": [True, False], }, scoring="f1_macro", cv=3, n_jobs=-1, verbose=1, ) optimisation_pipeline.fit(X, y) results = pd.DataFrame(optimisation_pipeline.cv_results_) results.sort_values("rank_test_score")

Fitting 3 folds for each of 24 candidates, totalling 72 fits

Out[4]:

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_classifier__alpha	param_classifier__fit_prior	param_vectorizer__max_df	param_vectorizer__min_df	params	split0_test_score	split1_test_score	split2_test_score	mean_test_score	std_test_score	rank_test_score
7	1.962260	0.147449	0.935357	0.063659	0.5	False	0.05	20	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.485030	0.463849	0.481840	0.476906	0.009324	1
10	1.942605	0.111027	0.952361	0.066812	0.5	False	0.1	20	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.482890	0.459556	0.479362	0.473936	0.010270	2
19	2.145152	0.068978	1.002291	0.047358	1	False	0.05	20	{'classifier__alpha': 1, 'classifier__fit_prio...	0.467330	0.442994	0.464302	0.458208	0.010829	3
22	1.971888	0.126950	0.739795	0.071551	1	False	0.1	20	{'classifier__alpha': 1, 'classifier__fit_prio...	0.454830	0.422902	0.450677	0.442803	0.014174	4
6	1.861275	0.013389	1.058907	0.111122	0.5	False	0.05	5	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.456127	0.422456	0.443827	0.440803	0.013912	5
11	1.825397	0.105754	0.892227	0.057003	0.5	False	0.1	100	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.438232	0.440464	0.422667	0.433788	0.007916	6
23	1.693333	0.009667	0.501491	0.006545	1	False	0.1	100	{'classifier__alpha': 1, 'classifier__fit_prio...	0.433915	0.439470	0.416031	0.429805	0.010001	7
8	2.008045	0.145330	0.944559	0.155925	0.5	False	0.05	100	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.436178	0.425724	0.418396	0.426766	0.007297	8
20	1.749200	0.022959	0.889532	0.047517	1	False	0.05	100	{'classifier__alpha': 1, 'classifier__fit_prio...	0.428215	0.425398	0.411051	0.421555	0.007516	9
9	1.960889	0.098004	0.985957	0.080925	0.5	False	0.1	5	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.430638	0.406619	0.420213	0.419157	0.009834	10
18	1.807799	0.064891	0.881872	0.030810	1	False	0.05	5	{'classifier__alpha': 1, 'classifier__fit_prio...	0.402402	0.372353	0.386189	0.386981	0.012280	11
1	2.009232	0.043125	0.899676	0.036977	0.5	True	0.05	20	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.389797	0.372619	0.388358	0.383591	0.007781	12
4	1.868087	0.094739	1.005353	0.101466	0.5	True	0.1	20	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.379323	0.364667	0.379060	0.374350	0.006848	13
21	1.958430	0.039639	0.890963	0.012546	1	False	0.1	5	{'classifier__alpha': 1, 'classifier__fit_prio...	0.366936	0.343361	0.363883	0.358060	0.010468	14
5	1.940692	0.018320	0.898865	0.030651	0.5	True	0.1	100	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.354850	0.349237	0.342737	0.348941	0.004950	15
2	1.855691	0.029506	0.866492	0.038048	0.5	True	0.05	100	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.349005	0.341832	0.328617	0.339818	0.008444	16
17	1.798559	0.103497	0.888273	0.069050	1	True	0.1	100	{'classifier__alpha': 1, 'classifier__fit_prio...	0.312332	0.297655	0.307471	0.305819	0.006104	17
14	2.016041	0.232138	0.967630	0.146144	1	True	0.05	100	{'classifier__alpha': 1, 'classifier__fit_prio...	0.304942	0.296921	0.297121	0.299661	0.003735	18
13	1.829513	0.112645	0.885848	0.027726	1	True	0.05	20	{'classifier__alpha': 1, 'classifier__fit_prio...	0.301539	0.285396	0.297272	0.294736	0.006830	19
0	1.905362	0.018052	0.885552	0.023985	0.5	True	0.05	5	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.295635	0.276759	0.296270	0.289555	0.009052	20
16	1.793688	0.049995	0.921301	0.060980	1	True	0.1	20	{'classifier__alpha': 1, 'classifier__fit_prio...	0.286746	0.272260	0.277084	0.278696	0.006023	21
3	2.078568	0.045549	0.963691	0.048281	0.5	True	0.1	5	{'classifier__alpha': 0.5, 'classifier__fit_pr...	0.276452	0.265509	0.268949	0.270303	0.004569	22
12	1.839506	0.048910	0.921812	0.010374	1	True	0.05	5	{'classifier__alpha': 1, 'classifier__fit_prio...	0.183196	0.186144	0.180323	0.183221	0.002376	23
15	1.909279	0.110639	1.056087	0.129738	1	True	0.1	5	{'classifier__alpha': 1, 'classifier__fit_prio...	0.165277	0.165840	0.167088	0.166068	0.000757	24

In [5]:

from sklearn import set_config

set_config(display="diagram")

classifier = create_pipeline()
classifier.set_params(**optimisation_pipeline.best_params_)
classifier.fit(X, y)

from sklearn import set_config set_config(display="diagram") classifier = create_pipeline() classifier.set_params(**optimisation_pipeline.best_params_) classifier.fit(X, y)

Out[5]:

Pipeline(steps=[('vectorizer',
                 TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),
                ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Export the model using GreatAI¶

In [6]:

from great_ai import save_model


save_model(classifier, key="small-domain-prediction", keep_last_n=5)

from great_ai import save_model save_model(classifier, key="small-domain-prediction", keep_last_n=5)

2022-06-25 14:50:53,592 |     INFO | Copying file for small-domain-prediction-0
2022-06-25 14:50:53,613 |     INFO | Compressing small-domain-prediction-0
2022-06-25 14:50:53,917 |     INFO | Uploading /tmp/tmpvxez8op8/small-domain-prediction-0.tar.gz to Mongo (GridFS)
2022-06-25 14:50:53,972 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.26/1.85 MB (14.1%)
2022-06-25 14:50:53,974 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.52/1.85 MB (28.2%)
2022-06-25 14:50:53,975 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.78/1.85 MB (42.3%)
2022-06-25 14:50:53,977 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.04/1.85 MB (56.4%)
2022-06-25 14:50:53,979 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.31/1.85 MB (70.5%)
2022-06-25 14:50:53,980 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.57/1.85 MB (84.7%)
2022-06-25 14:50:53,982 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.83/1.85 MB (98.8%)
2022-06-25 14:50:53,982 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.85/1.85 MB (100.0%)
2022-06-25 14:50:53,985 |     INFO | Model small-domain-prediction uploaded with version 0

Out[6]:

'small-domain-prediction:0'

Next: Part 3¶

---

Last update: July 9, 2022