Deployed b97b20b with MkDocs version: 1.3.0

This commit is contained in:
2022-07-10 17:45:41 +00:00
parent 38c201a13f
commit 9182ba84ed
35 changed files with 11722 additions and 1478 deletions

View file

@ -1,16 +1,19 @@
# Train and deploy a SOTA model
Let's see GreatAI in action by going over the life-cycle of a simple service.
## Objectives
1. You will see how the [great_ai.utilities][] can integrate into your Data Science workflow.
2. You will use [great_ai.large_file][] to version and store your trained model.
3. You will use [GreatAI][great_ai.GreatAI] to prepare your model for a robust and responsible deployment.
## Overview
You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676) in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field). We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html).
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/). The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag).
## Objectives
1. You will see how the `great_ai.utilities` can integrate into your Data Science workflow.
2. You will use `great_ai.large_file` to version and store your trained model
3. You will use `GreatAI` to prepare your model for a robust and responsible deployment
!!! success
You are ready to start the tutorial. Feel free to come back to the [summary](#summary) section once you're finished.
@ -18,7 +21,7 @@ We use the same synthetic dataset derived from the [Microsoft Academic Graph](ht
<div style="display: flex; justify-content: space-evenly;" markdown>
[:fontawesome-solid-chart-simple: Train it](train.ipynb){ .md-button .md-button--primary }
[:material-cloud-tags: Deploy it](deploy.ipynb){ .md-button .md-button--secondary }
[:material-cloud-tags: Deploy it](deploy.ipynb){ .md-button .md-button--primary }
</div>
@ -26,11 +29,11 @@ We use the same synthetic dataset derived from the [Microsoft Academic Graph](ht
### The [training notebook](train.ipynb)
We load and preprocess the dataset while relying on `great_ai.utilities.clean` for the heavy-lifting. Additionally, the preprocessing is parallelised using `great_ai.utilities.simple_parallel_map`.
We load and preprocess the dataset while relying on [great_ai.utilities.clean][] for the heavy-lifting. Additionally, the preprocessing is parallelised using [great_ai.utilities.simple_parallel_map][]
After training and evaluating a model, it is exported using `great_ai.save_model`.
After training and evaluating a model, it is exported using [great_ai.save_model][].
??? tip "Using the cloud"
??? tip "Remote storage"
To store your model remotely, you need to set your credentials before calling `save_model`.
For example, to use [AWS S3](https://aws.amazon.com/s3/):
@ -51,7 +54,7 @@ After training and evaluating a model, it is exported using `great_ai.save_model
### The [deployment notebook](deploy.ipynb)
We create an inference function that can be hardened by wrapping it in a `great_ai.GreatAI` instance.
We create an inference function that can be hardened by wrapping it in a [GreatAI][great_ai.GreatAI] instance.
```python
from great_ai import GreatAI, use_model
@ -64,7 +67,7 @@ def predict_domain(sentence, model):
return str(model.predict(inputs)[0])
```
1. `@use_model` loads and injects your model into the `predict_domain` function's `model` argument.
1. [@use_model][great_ai.use_model] loads and injects your model into the `predict_domain` function's `model` argument.
You can freely reference it knowing that it is always given to the function.
Finally, we deploy the model, inference function, and the GreatAI wrapping all of these. For that we either use: `great-ai deploy.ipynb` or build a Docker image.