andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed aa31d2b with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-15 15:52:07 +00:00
parent 5a8662103a
commit 087e049514
17 changed files with 58 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

2
tutorial/deploy/deploy.ipynb
View file

@ -6,7 +6,7 @@
"source": [
"# Harden and deploy your app\n",
"\n",
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.\n",
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.\n",
"\n",
"With `GreatAI`, it has become a matter of 4 lines of code."
]

4
tutorial/deploy/index.html
View file

@ -1829,7 +1829,7 @@ Licensed under the Apache License, Version 2.0.
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">&#182;</a></h1><p>Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.</p>
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">&#182;</a></h1><p>Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.</p>
<p>With <code>GreatAI</code>, it has become a matter of 4 lines of code.</p>
</div>
@ -2108,7 +2108,7 @@ def predict_domain(sentence, model):
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

10
tutorial/index.html
View file

@ -862,12 +862,12 @@
<h2 id="objectives">Objectives<a class="headerlink" href="#objectives" title="Permanent link">#</a></h2>
<ol>
<li>You will see how the <a href="/reference/utilities">great_ai.utilities</a> can integrate into your Data Science workflow.</li>
<li>You will use <a href="/reference/large_file">great_ai.large_file</a> to version and store your trained model.</li>
<li>You will use <a href="/reference/large-file">great_ai.large_file</a> to version and store your trained model.</li>
<li>You will use <a class="autorefs autorefs-internal" href="../reference/#great_ai.GreatAI">GreatAI</a> to prepare your model for a robust and responsible deployment.</li>
</ol>
<h2 id="overview">Overview<a class="headerlink" href="#overview" title="Permanent link">#</a></h2>
<p>You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the <a href="https://arxiv.org/abs/1903.10676">SciBERT paper</a> in which SciBERT <a href="https://paperswithcode.com/sota/sentence-classification-on-paper-field">achieved an F1-score of 0.6571</a>. We are going to outperform it using a trivial text classification model: a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html">Linear SVM</a>.</p>
<p>We use the same synthetic dataset derived from the <a href="https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/">Microsoft Academic Graph</a>. The dataset is <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">available here</a>.</p>
<p>You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the <a href="https://arxiv.org/abs/1903.10676" target="_blank">SciBERT paper</a> in which SciBERT <a href="https://paperswithcode.com/sota/sentence-classification-on-paper-field" target="_blank">achieved an F1-score of 0.6571</a>. We are going to outperform it using a trivial text classification model: a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html" target="_blank">Linear SVM</a>.</p>
<p>We use the same synthetic dataset derived from the <a href="https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/" target="_blank">Microsoft Academic Graph</a>. The dataset is <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag" target="_blank">available here</a>.</p>
<div class="admonition success">
<p class="admonition-title">Success</p>
<p>You are ready to start the tutorial. Feel free to come back to the <a href="#summary">summary</a> section once you're finished.</p>
@ -914,7 +914,7 @@
<li><a class="autorefs autorefs-internal" href="../reference/#great_ai.use_model">@use_model</a> loads and injects your model into the <code>predict_domain</code> function's <code>model</code> argument.
You can freely reference it knowing that the function is always provided with it.</li>
</ol>
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened-service properly.</a></p>
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened service properly.</a></p>
<div style="display: flex; justify-content: space-evenly;">
<p><a class="md-button md-button--primary" href="/how-to-guides/create-service"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 22a2 2 0 0 0 2-2V4a2 2 0 0 0-2-2h-6v7L9.5 7.5 7 9V2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12Z"/></svg></span> Learn about all the features</a></p>
<p><a class="md-button md-button--secondary" href="/examples/simple/data"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M7 2v2h1v14a4 4 0 0 0 4 4 4 4 0 0 0 4-4V4h1V2H7m4 14c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m2-4c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m1-5h-4V4h4v3Z"/></svg></span> Look at more examples</a></p>
@ -925,7 +925,7 @@
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

8
tutorial/index.md
View file

@ -5,14 +5,14 @@ Let's see `great-ai` in action by going over the life-cycle of a simple service.
## Objectives
1. You will see how the [great_ai.utilities](/reference/utilities) can integrate into your Data Science workflow.
2. You will use [great_ai.large_file](/reference/large_file) to version and store your trained model.
2. You will use [great_ai.large_file](/reference/large-file) to version and store your trained model.
3. You will use [GreatAI][great_ai.GreatAI] to prepare your model for a robust and responsible deployment.
## Overview
You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676) in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field). We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html).
You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676){ target=_blank } in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field){ target=_blank }. We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }.
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/). The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag).
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/){ target=_blank }. The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag){ target=_blank }.
!!! success
You are ready to start the tutorial. Feel free to come back to the [summary](#summary) section once you're finished.
@ -70,7 +70,7 @@ def predict_domain(sentence, model):
1. [@use_model][great_ai.use_model] loads and injects your model into the `predict_domain` function's `model` argument.
You can freely reference it knowing that the function is always provided with it.
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened-service properly.](/how-to-guides/use-service)
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened service properly.](/how-to-guides/use-service)
<div style="display: flex; justify-content: space-evenly;" markdown>
[:material-book: Learn about all the features](/how-to-guides/create-service){ .md-button .md-button--primary }

8
tutorial/train/index.html
View file

@ -1896,8 +1896,8 @@ Licensed under the Apache License, Version 2.0.
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<p>First, we have to get some data. After downloading it <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">from here</a>, we might notice that the dataset is in <a href="https://jsonlines.org/">JSON Lines</a> format (each line is a seperate JSON document).</p>
<p>Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also <a href="/reference/utilities/#great_ai.utilities.clean.clean">cleaned</a> to remove any LaTeX, XML, unicode, PDF-extraction artifacts.</p>
<p>First, we have to get some data. After downloading it <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">from here</a>, we might notice that the dataset is in <a href="https://jsonlines.org/">JSON Lines</a> format (each line is a separate JSON document).</p>
<p>Let's write a function which takes a single line and returns the sentence and the corresponding label from it. Before returning, the sentence is also <a href="/reference/utilities/#great_ai.utilities.clean.clean">cleaned</a> to remove any LaTeX, XML, unicode, PDF-extraction artifacts.</p>
</div>
</div>
@ -2367,7 +2367,7 @@ class="
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<p>Great work, we can be rightfully satisfied with our model. Seeing the results, we achieved an F1-score of 0.69 which is about <strong>5% better than SciBERT's</strong> 0.6571!</p>
<p>You might wonder that <em>"this is great, but besides some utility functions (<code>clean</code>, <code>simple_parallel_map</code>, ...) what more value does GreatAI add?"</em>. This would be a valid argument because the scope of GreatAI actually only starts here.</p>
<blockquote><p>Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey-zone for software engineering.</p>
<blockquote><p>Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey zone for software engineering.</p>
</blockquote>
<p>In order to use this model in production, we have to make it available on some possibly shared infrastructure.</p>
@ -2484,7 +2484,7 @@ save_model(model, key="my-domain-predictor")</div>
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

6
tutorial/train/train.ipynb
View file

@ -30,9 +30,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a seperate JSON document). \n",
"First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a separate JSON document). \n",
"\n",
"Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also [cleaned](/reference/utilities/#great_ai.utilities.clean.clean) to remove any LaTeX, XML, unicode, PDF-extraction artifacts."
"Let's write a function which takes a single line and returns the sentence and the corresponding label from it. Before returning, the sentence is also [cleaned](/reference/utilities/#great_ai.utilities.clean.clean) to remove any LaTeX, XML, unicode, PDF-extraction artifacts."
]
},
{
@ -238,7 +238,7 @@
"\n",
"You might wonder that *\"this is great, but besides some utility functions (`clean`, `simple_parallel_map`, ...) what more value does GreatAI add?\"*. This would be a valid argument because the scope of GreatAI actually only starts here.\n",
"\n",
"> Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey-zone for software engineering.\n",
"> Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey zone for software engineering.\n",
"\n",
"In order to use this model in production, we have to make it available on some possibly shared infrastructure."
]