Skip to content

# Optimise and train a model¶

> The blue boxes show the steps implemented in this notebook.

In the first part, we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3 or GridFS.

In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`.

## Load data that has been extracted in part 1¶

In [1]:
Copied!

    from great_ai import query_ground_truth

    data = query_ground_truth("train")
    X = [d.input for d in data for domain in d.feedback]
    y = [domain for d in data for domain in d.feedback]

from great_ai import query_ground_truth data = query_ground_truth("train") X = [d.input for d in data for domain in d.feedback] y = [domain for d in data for domain in d.feedback]

    Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️
    Cannot find credentials files, defaulting to using ParallelTinyDbDriver
    The selected tracing database (ParallelTinyDbDriver) is not recommended for production
    Cannot find credentials files, defaulting to using LargeFileLocal
    GreatAI (v0.1.4): configured ✅
      🔩 tracing_database: ParallelTinyDbDriver
      🔩 large_file_implementation: LargeFileLocal
      🔩 is_production: False
      🔩 should_log_exception_stack: True
      🔩 prediction_cache_size: 512
      🔩 dashboard_table_size: 50
    You still need to check whether you follow all best practices before trusting your deployment.
    > Find out more at https://se-ml.github.io/practices

In [23]:
Copied!

    from collections import Counter
    import matplotlib.pyplot as plt

    domains, counts = zip(*Counter(y).most_common())

    # Configure matplotlib to have nice, high-resolution charts
    %matplotlib inline
    plt.rcParams["figure.figsize"] = (20, 5)
    plt.rcParams["figure.facecolor"] = "white"
    plt.rcParams["font.size"] = 12

    plt.xticks(rotation=90)
    plt.bar(domains, counts)
    None

from collections import Counter import matplotlib.pyplot as plt domains, counts = zip(*Counter(y).most_common()) # Configure matplotlib to have nice, high-resolution charts %matplotlib inline plt.rcParams["figure.figsize"] = (20, 5) plt.rcParams["figure.facecolor"] = "white" plt.rcParams["font.size"] = 12 plt.xticks(rotation=90) plt.bar(domains, counts) None

## Optimise and train Multinomial Naive Bayes classifier¶

In [3]:
Copied!

    from sklearn.naive_bayes import MultinomialNB
    from sklearn.pipeline import Pipeline
    from sklearn.feature_extraction.text import TfidfVectorizer

    def create_pipeline() -> Pipeline:
        return Pipeline(
            steps=[
                ("vectorizer", TfidfVectorizer(sublinear_tf=True)),
                ("classifier", MultinomialNB()),
            ]
        )

from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import Pipeline from sklearn.feature_extraction.text import TfidfVectorizer def create_pipeline() -> Pipeline: return Pipeline( steps=[ ("vectorizer", TfidfVectorizer(sublinear_tf=True)), ("classifier", MultinomialNB()), ] )
In [4]:
Copied!

    from sklearn.model_selection import GridSearchCV

    optimisation_pipeline = GridSearchCV(
        create_pipeline(),
        {
            "vectorizer__min_df": [5, 20, 100],
            "vectorizer__max_df": [0.05, 0.1],
            "classifier__alpha": [0.5, 1],
            "classifier__fit_prior": [True, False],
        },
        scoring="f1_macro",
        cv=3,
        n_jobs=-1,
        verbose=1,
    )
    optimisation_pipeline.fit(X, y)

    results = pd.DataFrame(optimisation_pipeline.cv_results_)
    results.sort_values("rank_test_score")

from sklearn.model_selection import GridSearchCV optimisation_pipeline = GridSearchCV( create_pipeline(), { "vectorizer__min_df": [5, 20, 100], "vectorizer__max_df": [0.05, 0.1], "classifier__alpha": [0.5, 1], "classifier__fit_prior": [True, False], }, scoring="f1_macro", cv=3, n_jobs=-1, verbose=1, ) optimisation_pipeline.fit(X, y) results = pd.DataFrame(optimisation_pipeline.cv_results_) results.sort_values("rank_test_score")

    Fitting 3 folds for each of 24 candidates, totalling 72 fits

Out[4]:

| mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_classifier__alpha | param_classifier__fit_prior | param_vectorizer__max_df | param_vectorizer__min_df | params | split0_test_score | split1_test_score | split2_test_score | mean_test_score | std_test_score | rank_test_score
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
7 | 9.447799 | 0.785535 | 4.260490 | 0.180851 | 0.5 | False | 0.05 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.517618 | 0.518688 | 0.513803 | 0.516703 | 0.002096 | 1
10 | 9.750011 | 0.671769 | 4.791153 | 0.544862 | 0.5 | False | 0.1 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.513254 | 0.517940 | 0.509116 | 0.513437 | 0.003605 | 2
11 | 9.398448 | 0.656403 | 5.194527 | 0.303382 | 0.5 | False | 0.1 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.508306 | 0.502244 | 0.501839 | 0.504129 | 0.002958 | 3
8 | 9.285461 | 0.869226 | 5.030563 | 0.305472 | 0.5 | False | 0.05 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.504070 | 0.502116 | 0.500603 | 0.502263 | 0.001419 | 4
19 | 9.662259 | 0.234167 | 4.854762 | 0.126622 | 1 | False | 0.05 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.492205 | 0.504466 | 0.495489 | 0.497387 | 0.005182 | 5
23 | 9.024540 | 0.469833 | 3.235823 | 0.173482 | 1 | False | 0.1 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.501203 | 0.493465 | 0.492361 | 0.495676 | 0.003934 | 6
20 | 9.304917 | 0.192977 | 4.370798 | 0.255480 | 1 | False | 0.05 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.498869 | 0.494357 | 0.492431 | 0.495219 | 0.002698 | 7
22 | 9.642570 | 0.438559 | 3.332855 | 0.435930 | 1 | False | 0.1 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.479585 | 0.496964 | 0.485209 | 0.487253 | 0.007241 | 8
6 | 10.047471 | 0.473215 | 4.358744 | 0.051892 | 0.5 | False | 0.05 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.472578 | 0.486362 | 0.481865 | 0.480268 | 0.005739 | 9
5 | 10.287111 | 0.349778 | 4.908435 | 0.099558 | 0.5 | True | 0.1 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.474821 | 0.474690 | 0.478096 | 0.475869 | 0.001576 | 10
2 | 10.997282 | 0.851917 | 6.131619 | 0.274400 | 0.5 | True | 0.05 | 100 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.476218 | 0.471377 | 0.478548 | 0.475381 | 0.002987 | 11
9 | 9.147621 | 0.290876 | 5.392678 | 0.390403 | 0.5 | False | 0.1 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.458994 | 0.475008 | 0.469998 | 0.468000 | 0.006689 | 12
1 | 10.299674 | 0.486001 | 5.776907 | 0.462777 | 0.5 | True | 0.05 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.461036 | 0.466296 | 0.467280 | 0.464871 | 0.002741 | 13
4 | 10.771393 | 0.330218 | 4.961415 | 0.454726 | 0.5 | True | 0.1 | 20 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.454689 | 0.463340 | 0.457743 | 0.458591 | 0.003582 | 14
14 | 9.969123 | 0.715883 | 4.382175 | 0.216170 | 1 | True | 0.05 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.444039 | 0.449822 | 0.450524 | 0.448128 | 0.002906 | 15
17 | 9.439775 | 1.333323 | 4.914944 | 0.292653 | 1 | True | 0.1 | 100 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.440663 | 0.448419 | 0.443441 | 0.444174 | 0.003209 | 16
18 | 10.025776 | 0.198347 | 5.375965 | 0.220059 | 1 | False | 0.05 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.418284 | 0.428063 | 0.428696 | 0.425014 | 0.004766 | 17
13 | 9.251913 | 0.441015 | 5.085037 | 0.224299 | 1 | True | 0.05 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.400471 | 0.406275 | 0.406521 | 0.404422 | 0.002796 | 18
21 | 9.173751 | 0.112242 | 4.271786 | 0.293579 | 1 | False | 0.1 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.393559 | 0.402869 | 0.406431 | 0.400953 | 0.005427 | 19
16 | 9.683013 | 0.600089 | 4.924559 | 0.321389 | 1 | True | 0.1 | 20 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.387535 | 0.392661 | 0.393992 | 0.391396 | 0.002784 | 20
0 | 9.835260 | 0.220524 | 6.034349 | 0.586497 | 0.5 | True | 0.05 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.379471 | 0.381348 | 0.386078 | 0.382299 | 0.002780 | 21
3 | 11.905621 | 0.329256 | 5.571229 | 0.541937 | 0.5 | True | 0.1 | 5 | {'classifier__alpha': 0.5, 'classifier__fit_pr... | 0.368234 | 0.367599 | 0.372849 | 0.369561 | 0.002339 | 22
12 | 10.378068 | 0.227200 | 5.247946 | 0.126687 | 1 | True | 0.05 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.278087 | 0.278157 | 0.278143 | 0.278129 | 0.000030 | 23
15 | 10.375661 | 0.139826 | 5.239350 | 0.298795 | 1 | True | 0.1 | 5 | {'classifier__alpha': 1, 'classifier__fit_prio... | 0.259347 | 0.262720 | 0.258716 | 0.260261 | 0.001758 | 24

In [5]:
Copied!

    from sklearn import set_config

    set_config(display="diagram")

    classifier = create_pipeline()
    classifier.set_params(**optimisation_pipeline.best_params_)
    classifier.fit(X, y)

from sklearn import set_config set_config(display="diagram") classifier = create_pipeline() classifier.set_params(**optimisation_pipeline.best_params_) classifier.fit(X, y)
Out[5]:

    Pipeline(steps=[('vectorizer',
                     TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),
                    ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline

    Pipeline(steps=[('vectorizer',
                     TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)),
                    ('classifier', MultinomialNB(alpha=0.5, fit_prior=False))])

TfidfVectorizer

    TfidfVectorizer(max_df=0.05, min_df=20, sublinear_tf=True)

MultinomialNB

    MultinomialNB(alpha=0.5, fit_prior=False)

## Export the model using GreatAI¶

In [6]:
Copied!

    from great_ai import save_model

    save_model(classifier, key="small-domain-prediction", keep_last_n=5)

from great_ai import save_model save_model(classifier, key="small-domain-prediction", keep_last_n=5)

    Fetching cached versions of small-domain-prediction
    Copying file for small-domain-prediction-1
    Compressing small-domain-prediction-1
    Model small-domain-prediction uploaded with version 1

Out[6]:

    'small-domain-prediction:1'

## Next: Part 3¶

Last update: July 12, 2022