Skip to content

# Train a domain classifier on the semantic scholar dataset¶

> Part 2: train a model

> The blue boxes show the steps implemented in this notebook.

In Part 1, we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3.

In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`.

## Load data that has been extracted in part 1¶

In [1]:
Copied!

    from great_ai import query_ground_truth

    data = query_ground_truth("train")
    X = [d.input for d in data for domain in d.feedback]
    y = [domain for d in data for domain in d.feedback]

from great_ai import query_ground_truth data = query_ground_truth("train") X = [d.input for d in data for domain in d.feedback] y = [domain for d in data for domain in d.feedback]

    2022-06-25 14:50:29,879 |  WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️
    2022-06-25 14:50:29,880 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongodbDriver
    2022-06-25 14:50:29,881 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo
    2022-06-25 14:50:29,881 |     INFO | Settings: configured ✅
    2022-06-25 14:50:29,882 |     INFO | 🔩 tracing_database: MongodbDriver
    2022-06-25 14:50:29,883 |     INFO | 🔩 large_file_implementation: LargeFileMongo
    2022-06-25 14:50:29,883 |     INFO | 🔩 is_production: False
    2022-06-25 14:50:29,884 |     INFO | 🔩 should_log_exception_stack: True
    2022-06-25 14:50:29,884 |     INFO | 🔩 prediction_cache_size: 512
    2022-06-25 14:50:29,885 |     INFO | 🔩 dashboard_table_size: 50
    2022-06-25 14:50:29,885 |  WARNING | You still need to check whether you follow all best practices before trusting your deployment.
    2022-06-25 14:50:29,885 |  WARNING | > Find out more at https://se-ml.github.io/practices/

In [2]:
Copied!

    import pandas as pd
    from collections import Counter
    import plotly.express as px

    df = pd.DataFrame(Counter(y).most_common(), columns=["domain", "count"])
    px.bar(df, "domain", "count", width=1200, height=400).show()

import pandas as pd from collections import Counter import plotly.express as px df = pd.DataFrame(Counter(y).most_common(), columns=["domain", "count"]) px.bar(df, "domain", "count", width=1200, height=400).show()

## Optimise and train Multinomial Naive Bayes classifier¶

In [3]:
Copied!

    from sklearn.naive_bayes import MultinomialNB
    from sklearn.pipeline import Pipeline
    from sklearn.feature_extraction.text import TfidfVectorizer

    def create_pipeline() -> Pipeline:
        return Pipeline(
            steps=[
                ("vectorizer", TfidfVectorizer(sublinear_tf=True)),
                ("classifier", MultinomialNB()),
            ]
        )

from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import Pipeline from sklearn.feature_extraction.text import TfidfVectorizer def create_pipeline() -> Pipeline: return Pipeline( steps=[ ("vectorizer", TfidfVectorizer(sublinear_tf=True)), ("classifier", MultinomialNB()), ] )
In [4]:
Copied!

    from sklearn.model_selection import GridSearchCV

    optimisation_pipeline = GridSearchCV(
        create_pipeline(),
        {
            "vectorizer__min_df": [5, 20, 100],
            "vectorizer__max_df": [0.05, 0.1],
            "classifier__alpha": [0.5, 1],
            "classifier__fit_prior": [True, False],
        },
        scoring="f1_macro",
        cv=3,
        n_jobs=-1,
        verbose=1,
    )
    optimisation_pipeline.fit(X, y)

    results = pd.DataFrame(optimisation_pipeline.cv_results_)
    results.sort_values("rank_test_score")

from sklearn.model_selection import GridSearchCV optimisation_pipeline = GridSearchCV( create_pipeline(), { "vectorizer__min_df": [5, 20, 100], "vectorizer__max_df": [0.05, 0.1], "classifier__alpha": [0.5, 1], "classifier__fit_prior": [True, False], }, scoring="f1_macro", cv=3, n_jobs=-1, verbose=1, ) optimisation_pipeline.fit(X, y) results = pd.DataFrame(optimisation_pipeline.cv_results_) results.sort_values("rank_test_score")

    Fitting 3 folds for each of 24 candidates, totalling 72 fits

Out[4]:

| mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_classifier__alpha | param_classifier__fit_prior | param_vectorizer__max_df | param_vectorizer__min_df | params | split0_test_score | split1_test_score | split2_test_score | mean_test_score | std_test_score | rank_test_score
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
7 | 1.962260 | 0.147449 | 0.935357 | 0.063659 | 0.5 | False | 0.05 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.485030 | 0.463849 | 0.481840 | 0.476906 | 0.009324 | 1
10 | 1.942605 | 0.111027 | 0.952361 | 0.066812 | 0.5 | False | 0.1 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.482890 | 0.459556 | 0.479362 | 0.473936 | 0.010270 | 2
19 | 2.145152 | 0.068978 | 1.002291 | 0.047358 | 1 | False | 0.05 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.467330 | 0.442994 | 0.464302 | 0.458208 | 0.010829 | 3
22 | 1.971888 | 0.126950 | 0.739795 | 0.071551 | 1 | False | 0.1 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.454830 | 0.422902 | 0.450677 | 0.442803 | 0.014174 | 4
6 | 1.861275 | 0.013389 | 1.058907 | 0.111122 | 0.5 | False | 0.05 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.456127 | 0.422456 | 0.443827 | 0.440803 | 0.013912 | 5
11 | 1.825397 | 0.105754 | 0.892227 | 0.057003 | 0.5 | False | 0.1 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.438232 | 0.440464 | 0.422667 | 0.433788 | 0.007916 | 6
23 | 1.693333 | 0.009667 | 0.501491 | 0.006545 | 1 | False | 0.1 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.433915 | 0.439470 | 0.416031 | 0.429805 | 0.010001 | 7
8 | 2.008045 | 0.145330 | 0.944559 | 0.155925 | 0.5 | False | 0.05 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.436178 | 0.425724 | 0.418396 | 0.426766 | 0.007297 | 8
20 | 1.749200 | 0.022959 | 0.889532 | 0.047517 | 1 | False | 0.05 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.428215 | 0.425398 | 0.411051 | 0.421555 | 0.007516 | 9
9 | 1.960889 | 0.098004 | 0.985957 | 0.080925 | 0.5 | False | 0.1 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.430638 | 0.406619 | 0.420213 | 0.419157 | 0.009834 | 10
18 | 1.807799 | 0.064891 | 0.881872 | 0.030810 | 1 | False | 0.05 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.402402 | 0.372353 | 0.386189 | 0.386981 | 0.012280 | 11
1 | 2.009232 | 0.043125 | 0.899676 | 0.036977 | 0.5 | True | 0.05 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.389797 | 0.372619 | 0.388358 | 0.383591 | 0.007781 | 12
4 | 1.868087 | 0.094739 | 1.005353 | 0.101466 | 0.5 | True | 0.1 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.379323 | 0.364667 | 0.379060 | 0.374350 | 0.006848 | 13
21 | 1.958430 | 0.039639 | 0.890963 | 0.012546 | 1 | False | 0.1 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.366936 | 0.343361 | 0.363883 | 0.358060 | 0.010468 | 14
5 | 1.940692 | 0.018320 | 0.898865 | 0.030651 | 0.5 | True | 0.1 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.354850 | 0.349237 | 0.342737 | 0.348941 | 0.004950 | 15
2 | 1.855691 | 0.029506 | 0.866492 | 0.038048 | 0.5 | True | 0.05 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.349005 | 0.341832 | 0.328617 | 0.339818 | 0.008444 | 16
17 | 1.798559 | 0.103497 | 0.888273 | 0.069050 | 1 | True | 0.1 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.312332 | 0.297655 | 0.307471 | 0.305819 | 0.006104 | 17
14 | 2.016041 | 0.232138 | 0.967630 | 0.146144 | 1 | True | 0.05 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.304942 | 0.296921 | 0.297121 | 0.299661 | 0.003735 | 18
13 | 1.829513 | 0.112645 | 0.885848 | 0.027726 | 1 | True | 0.05 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.301539 | 0.285396 | 0.297272 | 0.294736 | 0.006830 | 19
0 | 1.905362 | 0.018052 | 0.885552 | 0.023985 | 0.5 | True | 0.05 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.295635 | 0.276759 | 0.296270 | 0.289555 | 0.009052 | 20
16 | 1.793688 | 0.049995 | 0.921301 | 0.060980 | 1 | True | 0.1 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.286746 | 0.272260 | 0.277084 | 0.278696 | 0.006023 | 21
3 | 2.078568 | 0.045549 | 0.963691 | 0.048281 | 0.5 | True | 0.1 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.276452 | 0.265509 | 0.268949 | 0.270303 | 0.004569 | 22
12 | 1.839506 | 0.048910 | 0.921812 | 0.010374 | 1 | True | 0.05 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.183196 | 0.186144 | 0.180323 | 0.183221 | 0.002376 | 23
15 | 1.909279 | 0.110639 | 1.056087 | 0.129738 | 1 | True | 0.1 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.165277 | 0.165840 | 0.167088 | 0.166068 | 0.000757 | 24

In [5]:
Copied!

    from sklearn import set_config

    set_config(display="diagram")

    classifier = create_pipeline()
    classifier.set_params(**optimisation_pipeline.best_params_)
    classifier.fit(X, y)

from sklearn import set_config set_config(display="diagram") classifier = create_pipeline() classifier.set_params(**optimisation_pipeline.best_params_) classifier.fit(X, y)
Out[5]:

    Pipeline(steps=[('vectorizer',
                     TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),
                    ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline

    Pipeline(steps=[('vectorizer',
                     TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),
                    ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])

TfidfVectorizer

    TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)

MultinomialNB

    MultinomialNB(alpha=0.5, fit_prior=False)

## Export the model using GreatAI¶

In [6]:
Copied!

    from great_ai import save_model

    save_model(classifier, key="small-domain-prediction", keep_last_n=5)

from great_ai import save_model save_model(classifier, key="small-domain-prediction", keep_last_n=5)

    2022-06-25 14:50:53,592 |     INFO | Copying file for small-domain-prediction-0
    2022-06-25 14:50:53,613 |     INFO | Compressing small-domain-prediction-0
    2022-06-25 14:50:53,917 |     INFO | Uploading /tmp/tmpvxez8op8/small-domain-prediction-0.tar.gz to Mongo (GridFS)
    2022-06-25 14:50:53,972 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.26/1.85 MB (14.1%)
    2022-06-25 14:50:53,974 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.52/1.85 MB (28.2%)
    2022-06-25 14:50:53,975 |     INFO | Uploading small-domain-prediction-0.tar.gz 0.78/1.85 MB (42.3%)
    2022-06-25 14:50:53,977 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.04/1.85 MB (56.4%)
    2022-06-25 14:50:53,979 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.31/1.85 MB (70.5%)
    2022-06-25 14:50:53,980 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.57/1.85 MB (84.7%)
    2022-06-25 14:50:53,982 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.83/1.85 MB (98.8%)
    2022-06-25 14:50:53,982 |     INFO | Uploading small-domain-prediction-0.tar.gz 1.85/1.85 MB (100.0%)
    2022-06-25 14:50:53,985 |     INFO | Model small-domain-prediction uploaded with version 0

Out[6]:

    'small-domain-prediction:0'

### Next: Part 3¶

Last update: July 8, 2022