andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed aa31d2b with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-15 15:52:07 +00:00
parent 5a8662103a
commit 087e049514
17 changed files with 58 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

2
examples/simple/data/data.ipynb
View file

@ -103,9 +103,7 @@
" unchunk,\n",
")\n",
"\n",
"\n",
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
" # Extract\n",
" response = urllib.request.urlopen(\n",
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/\"\n",
" \"open-corpus/2022-02-01/{chunk_key}\"\n",

6
examples/simple/data/index.html
View file

@ -2083,9 +2083,7 @@ f"""Processing {len(chunks)} out of the {
<span class="n">unchunk</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">preprocess_chunk</span><span class="p">(</span><span class="n">chunk_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]]:</span>
<span class="c1"># Extract</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/&quot;</span>
<span class="s2">&quot;open-corpus/2022-02-01/</span><span class="si">{chunk_key}</span><span class="s2">&quot;</span>
@ -2125,9 +2123,7 @@ from great_ai.utilities import (
unchunk,
)
def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
# Extract
response = urllib.request.urlopen(
f"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/"
"open-corpus/2022-02-01/{chunk_key}"
@ -2294,7 +2290,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

4
examples/simple/train/index.html
View file

@ -1877,7 +1877,7 @@ Licensed under the Apache License, Version 2.0.
<blockquote><p>The blue boxes show the steps implemented in this notebook.</p>
</blockquote>
<p>In the first part, we have cleaned and transformed our training data. We can now access this data using <code>great_ai.LargeFile</code>. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3 or GridFS.</p>
<p>In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using <code>great_ai.save_model</code>.</p>
<p>In this part, we hyperparameter-optimise and train a simple Naive Bayes classifier which we then export for deployment using <code>great_ai.save_model</code>.</p>
<h2 id="load-data-that-has-been-extracted-in-part-1">Load data that has been extracted in <a href="/examples/simple/data">part 1</a><a class="anchor-link" href="#load-data-that-has-been-extracted-in-part-1">&#182;</a></h2>
</div>
</div>
@ -2864,7 +2864,7 @@ save_model(classifier, key="small-domain-prediction", keep_last_n=5)</div>
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

2
examples/simple/train/train.ipynb
View file

@ -11,7 +11,7 @@
"\n",
"In the first part, we have cleaned and transformed our training data. We can now access this data using `great_ai.LargeFile`. Locally, it will gives us the cached version, otherwise, the latest version is downloaded from S3 or GridFS. \n",
"\n",
"In this part, we hyperparameter-optimise and train a simple, Naive Bayes classifier which we then export for deployment using `great_ai.save_model`.\n",
"In this part, we hyperparameter-optimise and train a simple Naive Bayes classifier which we then export for deployment using `great_ai.save_model`.\n",
"\n",
"## Load data that has been extracted in [part 1](/examples/simple/data)"
]

8
how-to-guides/use-service/index.html
View file

@ -936,7 +936,7 @@
<p>As you can see, the original return value is wrapped in a <a class="autorefs autorefs-internal" href="../../reference/views/#great_ai.Trace">Trace</a> object (which is also persisted in your database of choice). You can access the original value under the <code>output</code> property.</p>
<h2 id="online-prediction">Online prediction<a class="headerlink" href="#online-prediction" title="Permanent link">#</a></h2>
<p>Likely, the main way you would like to expose your model is through an HTTP API. <a class="autorefs autorefs-internal" href="../../reference/#great_ai.deploy.great_ai.GreatAI.create">@GreatAI.create</a> scaffolds many REST API endpoints for your model and creates a <a href="https://fastapi.tiangolo.com/" target="_blank">FastAPI</a> app available under <a class="autorefs autorefs-internal" href="../../reference/#great_ai.GreatAI">GreatAI.app</a>. This can be served using <a href="https://www.uvicorn.org/" target="_blank">uvicorn</a> or any other <a href="https://asgi.readthedocs.io/en/latest/" target="_blank">ASGI server</a>.</p>
<p>Since most ML code lives in <a href="https://jupyter.org/" target="_blank">Jupyter</a> notebooks, therefore, deploying a notebook containing the inference function is supported. To this end, <code>uvicorn</code> is wrapped by the <code>great-ai</code> command-line utility which, among others, takes care of feeding a notebook into <code>uvicorn</code>. It also supports auto-reloading.</p>
<p>Since most ML code lives in <a href="https://jupyter.org/" target="_blank">Jupyter</a> notebooks, therefore, deploying a notebook containing the inference function is supported. To achieve this, <code>uvicorn</code> is wrapped by the <code>great-ai</code> command-line utility which, among others, takes care of feeding a notebook into <code>uvicorn</code>. It also supports auto-reloading.</p>
<h3 id="in-development">In development<a class="headerlink" href="#in-development" title="Permanent link">#</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a>great-ai greeter.py
</code></pre></div>
@ -970,7 +970,7 @@
<p>You can replace <code>pwd</code> with the path to your code's folder.</p>
</blockquote>
<h4 id="use-a-platform-as-a-service">Use a Platform-as-a-Service<a class="headerlink" href="#use-a-platform-as-a-service" title="Permanent link">#</a></h4>
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://aws.amazon.com/ecs/" target="_blank">AWS ECS</a>, <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a>, <a href="https://mlem.ai/" target="_blank">MLEM</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://aws.amazon.com/ecs/" target="_blank">AWS ECS</a>, <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a>, <a href="https://mlem.ai/" target="_blank">MLEM</a>, <a href="https://streamlit.io/" target="_blank">Streamlit</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
<p>To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as <a href="https://pytorch.org/" target="_blank">PyTorch</a> or <a href="https://www.tensorflow.org/" target="_blank">TensorFlow</a>.</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="k">FROM</span><span class="w"> </span><span class="s">schmelczera/great-ai:latest</span>
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>
@ -986,7 +986,7 @@
<a id="__codelineno-6-12" name="__codelineno-6-12" href="#__codelineno-6-12"></a><span class="c"># Add you application code to the image</span>
<a id="__codelineno-6-13" name="__codelineno-6-13" href="#__codelineno-6-13"></a><span class="k">COPY</span><span class="w"> </span>. .
<a id="__codelineno-6-14" name="__codelineno-6-14" href="#__codelineno-6-14"></a>
<a id="__codelineno-6-15" name="__codelineno-6-15" href="#__codelineno-6-15"></a><span class="c"># The default ENTRYPOINT is great-ai, specify it&#39;s argument using CMD</span>
<a id="__codelineno-6-15" name="__codelineno-6-15" href="#__codelineno-6-15"></a><span class="c"># The default ENTRYPOINT is great-ai, specify its argument using CMD</span>
<a id="__codelineno-6-16" name="__codelineno-6-16" href="#__codelineno-6-16"></a><span class="k">CMD</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;deploy.ipynb&quot;</span><span class="p">]</span>
</code></pre></div>
<h2 id="batch-prediction">Batch prediction<a class="headerlink" href="#batch-prediction" title="Permanent link">#</a></h2>
@ -1017,7 +1017,7 @@
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

6
how-to-guides/use-service/use-service.md
View file

@ -35,7 +35,7 @@ As you can see, the original return value is wrapped in a [Trace][great_ai.Trace
Likely, the main way you would like to expose your model is through an HTTP API. [@GreatAI.create][great_ai.GreatAI.create] scaffolds many REST API endpoints for your model and creates a [FastAPI](https://fastapi.tiangolo.com/){ target=_blank } app available under [GreatAI.app][great_ai.GreatAI]. This can be served using [uvicorn](https://www.uvicorn.org/){ target=_blank } or any other [ASGI server](https://asgi.readthedocs.io/en/latest/){ target=_blank }.
Since most ML code lives in [Jupyter](https://jupyter.org/){ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To this end, `uvicorn` is wrapped by the `great-ai` command-line utility which, among others, takes care of feeding a notebook into `uvicorn`. It also supports auto-reloading.
Since most ML code lives in [Jupyter](https://jupyter.org/){ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To achieve this, `uvicorn` is wrapped by the `great-ai` command-line utility which, among others, takes care of feeding a notebook into `uvicorn`. It also supports auto-reloading.
### In development
@ -81,7 +81,7 @@ docker run -p 6060:6060 --volume `pwd`:/app --rm \
#### Use a Platform-as-a-Service
Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [AWS ECS](https://aws.amazon.com/ecs/){ target=_blank }, [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank }, [MLEM](https://mlem.ai/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.
Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [AWS ECS](https://aws.amazon.com/ecs/){ target=_blank }, [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank }, [MLEM](https://mlem.ai/){ target=_blank }, [Streamlit](https://streamlit.io/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.
To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as [PyTorch](https://pytorch.org/){ target=_blank } or [TensorFlow](https://www.tensorflow.org/){ target=_blank }.
@ -100,7 +100,7 @@ RUN large-file --backend s3 --secrets s3.ini --cache my-domain-predictor
# Add you application code to the image
COPY . .
# The default ENTRYPOINT is great-ai, specify it's argument using CMD
# The default ENTRYPOINT is great-ai, specify its argument using CMD
CMD ["deploy.ipynb"]
```

4
index.html
View file

@ -935,7 +935,7 @@
</div>
<h2 id="why-is-this-great">Why is this GREAT?<a class="headerlink" href="#why-is-this-great" title="Permanent link">#</a></h2>
<p><img alt="scope of GreatAI" src="media/scope-simple.drawio.svg" /></p>
<p>GreatAI fits between the prototype and deployment phases of your (or your organisation's) AI development lifecycle. This is highlighted with blue in the diagram. Here, a number of best practices can be automatically implemented aiming to achieve the following attributes:</p>
<p>GreatAI fits between the prototype and deployment phases of your (or your organisation's) AI development lifecycle. This is highlighted with blue in the diagram. Here, several best practices can be automatically implemented aiming to achieve the following attributes:</p>
<ul>
<li><strong>G</strong>eneral: use any Python library without restriction</li>
<li><strong>R</strong>obust: have error-handling and well-tested utilities out-of-the-box </li>
@ -961,7 +961,7 @@
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

2
index.md
View file

@ -86,7 +86,7 @@ great-ai demo.py
![scope of GreatAI](media/scope-simple.drawio.svg)
GreatAI fits between the prototype and deployment phases of your (or your organisation's) AI development lifecycle. This is highlighted with blue in the diagram. Here, a number of best practices can be automatically implemented aiming to achieve the following attributes:
GreatAI fits between the prototype and deployment phases of your (or your organisation's) AI development lifecycle. This is highlighted with blue in the diagram. Here, several best practices can be automatically implemented aiming to achieve the following attributes:
- **G**eneral: use any Python library without restriction
- **R**obust: have error-handling and well-tested utilities out-of-the-box

2
search/search_index.json
View file

File diff suppressed because one or more lines are too long

48
sitemap.xml
View file

@ -2,122 +2,122 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
<lastmod>2022-07-13</lastmod>
<lastmod>2022-07-15</lastmod>
<changefreq>daily</changefreq>
</url>
</urlset>

BIN
sitemap.xml.gz
View file

Binary file not shown.

2
tutorial/deploy/deploy.ipynb
View file

@ -6,7 +6,7 @@
"source": [
"# Harden and deploy your app\n",
"\n",
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.\n",
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.\n",
"\n",
"With `GreatAI`, it has become a matter of 4 lines of code."
]

4
tutorial/deploy/index.html
View file

@ -1829,7 +1829,7 @@ Licensed under the Apache License, Version 2.0.
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">&#182;</a></h1><p>Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.</p>
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">&#182;</a></h1><p>Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.</p>
<p>With <code>GreatAI</code>, it has become a matter of 4 lines of code.</p>
</div>
@ -2108,7 +2108,7 @@ def predict_domain(sentence, model):
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

10
tutorial/index.html
View file

@ -862,12 +862,12 @@
<h2 id="objectives">Objectives<a class="headerlink" href="#objectives" title="Permanent link">#</a></h2>
<ol>
<li>You will see how the <a href="/reference/utilities">great_ai.utilities</a> can integrate into your Data Science workflow.</li>
<li>You will use <a href="/reference/large_file">great_ai.large_file</a> to version and store your trained model.</li>
<li>You will use <a href="/reference/large-file">great_ai.large_file</a> to version and store your trained model.</li>
<li>You will use <a class="autorefs autorefs-internal" href="../reference/#great_ai.GreatAI">GreatAI</a> to prepare your model for a robust and responsible deployment.</li>
</ol>
<h2 id="overview">Overview<a class="headerlink" href="#overview" title="Permanent link">#</a></h2>
<p>You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the <a href="https://arxiv.org/abs/1903.10676">SciBERT paper</a> in which SciBERT <a href="https://paperswithcode.com/sota/sentence-classification-on-paper-field">achieved an F1-score of 0.6571</a>. We are going to outperform it using a trivial text classification model: a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html">Linear SVM</a>.</p>
<p>We use the same synthetic dataset derived from the <a href="https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/">Microsoft Academic Graph</a>. The dataset is <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">available here</a>.</p>
<p>You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the <a href="https://arxiv.org/abs/1903.10676" target="_blank">SciBERT paper</a> in which SciBERT <a href="https://paperswithcode.com/sota/sentence-classification-on-paper-field" target="_blank">achieved an F1-score of 0.6571</a>. We are going to outperform it using a trivial text classification model: a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html" target="_blank">Linear SVM</a>.</p>
<p>We use the same synthetic dataset derived from the <a href="https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/" target="_blank">Microsoft Academic Graph</a>. The dataset is <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag" target="_blank">available here</a>.</p>
<div class="admonition success">
<p class="admonition-title">Success</p>
<p>You are ready to start the tutorial. Feel free to come back to the <a href="#summary">summary</a> section once you're finished.</p>
@ -914,7 +914,7 @@
<li><a class="autorefs autorefs-internal" href="../reference/#great_ai.use_model">@use_model</a> loads and injects your model into the <code>predict_domain</code> function's <code>model</code> argument.
You can freely reference it knowing that the function is always provided with it.</li>
</ol>
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened-service properly.</a></p>
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened service properly.</a></p>
<div style="display: flex; justify-content: space-evenly;">
<p><a class="md-button md-button--primary" href="/how-to-guides/create-service"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 22a2 2 0 0 0 2-2V4a2 2 0 0 0-2-2h-6v7L9.5 7.5 7 9V2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12Z"/></svg></span> Learn about all the features</a></p>
<p><a class="md-button md-button--secondary" href="/examples/simple/data"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M7 2v2h1v14a4 4 0 0 0 4 4 4 4 0 0 0 4-4V4h1V2H7m4 14c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m2-4c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m1-5h-4V4h4v3Z"/></svg></span> Look at more examples</a></p>
@ -925,7 +925,7 @@
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

8
tutorial/index.md
View file

@ -5,14 +5,14 @@ Let's see `great-ai` in action by going over the life-cycle of a simple service.
## Objectives
1. You will see how the [great_ai.utilities](/reference/utilities) can integrate into your Data Science workflow.
2. You will use [great_ai.large_file](/reference/large_file) to version and store your trained model.
2. You will use [great_ai.large_file](/reference/large-file) to version and store your trained model.
3. You will use [GreatAI][great_ai.GreatAI] to prepare your model for a robust and responsible deployment.
## Overview
You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676) in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field). We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html).
You are going to train a field of study (domain) classifier for scientific sentences. The exact task was proposed by the [SciBERT paper](https://arxiv.org/abs/1903.10676){ target=_blank } in which SciBERT [achieved an F1-score of 0.6571](https://paperswithcode.com/sota/sentence-classification-on-paper-field){ target=_blank }. We are going to outperform it using a trivial text classification model: a [Linear SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html){ target=_blank }.
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/). The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag).
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/){ target=_blank }. The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag){ target=_blank }.
!!! success
You are ready to start the tutorial. Feel free to come back to the [summary](#summary) section once you're finished.
@ -70,7 +70,7 @@ def predict_domain(sentence, model):
1. [@use_model][great_ai.use_model] loads and injects your model into the `predict_domain` function's `model` argument.
You can freely reference it knowing that the function is always provided with it.
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened-service properly.](/how-to-guides/use-service)
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened service properly.](/how-to-guides/use-service)
<div style="display: flex; justify-content: space-evenly;" markdown>
[:material-book: Learn about all the features](/how-to-guides/create-service){ .md-button .md-button--primary }

8
tutorial/train/index.html
View file

@ -1896,8 +1896,8 @@ Licensed under the Apache License, Version 2.0.
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<p>First, we have to get some data. After downloading it <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">from here</a>, we might notice that the dataset is in <a href="https://jsonlines.org/">JSON Lines</a> format (each line is a seperate JSON document).</p>
<p>Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also <a href="/reference/utilities/#great_ai.utilities.clean.clean">cleaned</a> to remove any LaTeX, XML, unicode, PDF-extraction artifacts.</p>
<p>First, we have to get some data. After downloading it <a href="https://github.com/allenai/scibert/tree/master/data/text_classification/mag">from here</a>, we might notice that the dataset is in <a href="https://jsonlines.org/">JSON Lines</a> format (each line is a separate JSON document).</p>
<p>Let's write a function which takes a single line and returns the sentence and the corresponding label from it. Before returning, the sentence is also <a href="/reference/utilities/#great_ai.utilities.clean.clean">cleaned</a> to remove any LaTeX, XML, unicode, PDF-extraction artifacts.</p>
</div>
</div>
@ -2367,7 +2367,7 @@ class="
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
<p>Great work, we can be rightfully satisfied with our model. Seeing the results, we achieved an F1-score of 0.69 which is about <strong>5% better than SciBERT's</strong> 0.6571!</p>
<p>You might wonder that <em>"this is great, but besides some utility functions (<code>clean</code>, <code>simple_parallel_map</code>, ...) what more value does GreatAI add?"</em>. This would be a valid argument because the scope of GreatAI actually only starts here.</p>
<blockquote><p>Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey-zone for software engineering.</p>
<blockquote><p>Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey zone for software engineering.</p>
</blockquote>
<p>In order to use this model in production, we have to make it available on some possibly shared infrastructure.</p>
@ -2484,7 +2484,7 @@ save_model(model, key="my-domain-predictor")</div>
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

6
tutorial/train/train.ipynb
View file

@ -30,9 +30,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a seperate JSON document). \n",
"First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a separate JSON document). \n",
"\n",
"Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also [cleaned](/reference/utilities/#great_ai.utilities.clean.clean) to remove any LaTeX, XML, unicode, PDF-extraction artifacts."
"Let's write a function which takes a single line and returns the sentence and the corresponding label from it. Before returning, the sentence is also [cleaned](/reference/utilities/#great_ai.utilities.clean.clean) to remove any LaTeX, XML, unicode, PDF-extraction artifacts."
]
},
{
@ -238,7 +238,7 @@
"\n",
"You might wonder that *\"this is great, but besides some utility functions (`clean`, `simple_parallel_map`, ...) what more value does GreatAI add?\"*. This would be a valid argument because the scope of GreatAI actually only starts here.\n",
"\n",
"> Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey-zone for software engineering.\n",
"> Not coincidentally, this is the point where the scope of Data Science ends but it's still a grey zone for software engineering.\n",
"\n",
"In order to use this model in production, we have to make it available on some possibly shared infrastructure."
]