Deployed 69f2f6d with MkDocs version: 1.3.1

thesis/chapters/5_cases/scibert.tex

@@ -5,7 +5,7 @@ Let us now turn our attention towards a more complex component. The ScoutinScien
 The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended by the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but not adequate for providing an accurate overview. Thus, this is the baseline that I attempt to improve on in this section.
 
 \begin{displayquote}
-Compared with Section \ref{section:simple-case}, this time around, the toolset of GreatAI is available at our disposal. Hopefully, this will streamline the development and --- especially --- the deployment. Given its arguably higher complexity, the experiment falls closer to industrial use-cases, and hence, can give more accurate feedback on how to further improve the API.
+Compared with Section \ref{section:simple-case}, this time around, the toolset of \textit{GreatAI} is available at our disposal. Hopefully, this will streamline the development and --- especially --- the deployment. Given its arguably higher complexity, the experiment falls closer to industrial use-cases, and hence, can give more accurate feedback on how to further improve the API.
 \end{displayquote}
 
 \subsection{Background}
@@ -43,11 +43,11 @@ For the first iteration, 1500 sentences were selected for 2 experts to annotate
 The next step is finetuning SciBERT with the help of HuggingFace transformers \cite{wolf2019huggingface}. The data are divided into training and test splits with a ratio of 4:1. From the train split, a validation split is also derived which is used for early stopping. The objective function is the positive class' F1-score and the early stopping patience is 5 epochs. The learning rate is $5 \times 10^{-5}$ and AdamW \cite{loshchilov2017decoupled} is used for optimisation with a weight decay of 0.05. The code can be found in the documentation\footnote{\href{https://great-ai.scoutinscience.com/examples/scibert/train/}{great-ai.scoutinscience.com/examples/scibert/train/}}, it is surprisingly slightly shorter than the code of Section \ref{section:simple-case}.
 
 \begin{displayquote}
-\textbf{Reproducability} Reproducible experiments are generally preferred. It is easy to forget to set some seeds values and, for example, end up with different datapoints in the test-train splits during training and validation in a Continuous Integration (CI) pipeline. To facilitate reproducability, it would be useful to reset the seeds of each imported library's random number generators (RNG) when GreatAI is configured. Thus, a feature has been added to detect and reset RNG-s of installed and imported libraries. This certainly will not solve the reproducibility crisis \cite{hutson2018artificial} on its own, however, in some cases, it can result in one fewer step to miss.
+\textbf{Reproducability} Reproducible experiments are generally preferred. It is easy to forget to set some seeds values and, for example, end up with different datapoints in the test-train splits during training and validation in a Continuous Integration (CI) pipeline. To facilitate reproducability, it would be useful to reset the seeds of each imported library's random number generators (RNGs) when \textit{GreatAI} is configured. Thus, a feature has been added to detect and reset RNGs of installed and imported libraries. This certainly will not solve the reproducibility crisis \cite{hutson2018artificial} on its own, however, in some cases, it can result in one fewer step to miss.
 \end{displayquote}
 
 \begin{displayquote}
-\textbf{Utility of LargeFiles-s} For the purposes of the documentation, the finetuning was conducted in the Google Colab online environment which is excellent for providing anyone with GPU-time for free. However, notebook environments are ephemeral, resulting in the need to manually upload and download all relevant data whenever a new virtual machine (VM) instance is granted. The \texttt{LargeFile} implementation alleviated this problem by handling the uploads and downloads automatically. Of course, first, backwards compatibility had to be solved for Python 3.7 which is the only available environment in Colab.
+\textbf{Utility of LargeFiles} For the purposes of the documentation, the finetuning was conducted in the Google Colab online environment which is excellent for providing anyone with GPU-time for free. However, notebook environments are ephemeral, resulting in the need to manually upload and download all relevant data whenever a new virtual machine (VM) instance is granted. The \texttt{LargeFile} implementation alleviated this problem by handling the uploads and downloads automatically. Of course, first, backwards compatibility had to be solved for Python 3.7 which is the only available environment in Colab.
 \end{displayquote}
 
 The best validation results were achieved after 8 epochs which was slightly more than expected but is presumably due to the weight decay. The confusion matrix on the test split can be seen in Figure \ref{fig:scibert-confusion}: regardless the subjective definition of the task, SciBERT manages to achieve good quality which is indicated by an F1-score of \textbf{0.89}.
@@ -96,7 +96,7 @@ Even though the operation is conceptually simple, because of the opaque datastru
 
 \section{Improving GreatAI}
 
-After having solved two problems by implementing two standalone services and integrating them into an existing ecosystem while relying on GreatAI as a primary tool, a wide variety of insights have been gained. In the next couple of subsections, the extra features and design decisions are presented that have been motivated by the \textit{Highlights service}. After which, the final surface of the API is described and evaluated by its relation to the SE4ML \cite{serban2020adoption,serban2021practices} and AI engineering \cite{john2020architecting,john2020ai} best practices.
+After having solved two problems by implementing two standalone services and integrating them into an existing ecosystem while relying on \textit{GreatAI} as a primary tool, a wide variety of insights have been gained. In the next couple of subsections, the extra features and design decisions are presented that have been motivated by the \textit{Highlights service}. After which, the final surface of the API is described and evaluated by its relation to the SE4ML \cite{serban2020adoption,serban2021practices} and AI engineering \cite{john2020architecting,john2020ai} best practices.
 
 \subsection{Caching}
 
@@ -104,13 +104,13 @@ Sustainability is an increasingly crucial concern of ethical AI \cite{van2021sus
 
 \subsection{Revisiting \texttt{parallel\_map}}
 
-Even though most inference functions are CPU-bound, turns out, sometimes they involve IO, especially, when relying on the results of other, remote models. This means that a significant performance improvement can be achieved by implementing some inference functions asynchronously \cite{tilkov2010node}. Thus, GreatAI also has to support decorating both regular (synchronous) and asynchronous functions. There is one notable consequence of this: the batch processing feature also has to be compatible with \texttt{async} inference functions. Batch processing is still a useful feature since it is likely that async inference functions are both IO (remote calls) and CPU (local evaluation) constrained at the same time, thus, they can benefit from multi-core parallelisation. 
+Even though most inference functions are CPU-bound, turns out, sometimes they involve IO, especially, when relying on the results of other, remote models. This means that a significant performance improvement can be achieved by implementing some inference functions asynchronously \cite{tilkov2010node}. Thus, \textit{GreatAI} also has to support decorating both regular (synchronous) and asynchronous functions. There is one notable consequence of this: the batch processing feature also has to be compatible with \texttt{async} inference functions. Batch processing is still a useful feature since it is likely that async inference functions are both IO (remote calls) and CPU (local evaluation) constrained at the same time, thus, they can benefit from multi-core parallelisation. 
 
 However, the standard library's \texttt{multiprocessing}, the third party \texttt{multiprocess} \cite{mckerns2012building}, and, another popular library, \texttt{joblib}\footnote{\href{https://joblib.readthedocs.io/en/latest/}{joblib.readthedocs.io/en/latest}} all lack the support for efficiently parallelising async functions. For this reason, \texttt{parallel\_map} is reimplemented to create an event-loop in each worker process to keep the efficiency of non-blocking IO while also providing parallelisation for the CPU-bound sections of code.
 
 \subsection{Programmatic integration}
 
-Apart from supporting \texttt{async} calls, there are a couple of more step that can be taken to help integrating any service with a GreatAI deployment. This is implemented by the \texttt{call\_remote\_great\_ai} function which hides the networking required to call a GreatAI instance's REST API. It takes care of validation, automatic retries, serialisation, and deserialisation. This comes with the added benefit of encouraging decoupled services because the friction of integrating them is no longer noticeable which is beneficial for human collaboration \cite{hasselbring2002component}.
+Apart from supporting \texttt{async} calls, there are a couple of more step that can be taken to help integrating any service with a \textit{GreatAI} deployment. This is implemented by the \texttt{call\_remote\_great\_ai} function which hides the networking required to call a \textit{GreatAI} instance's REST API. It takes care of validation, automatic retries, serialisation, and deserialisation. This comes with the added benefit of encouraging decoupled services because the friction of integrating them is no longer noticeable which is beneficial for human collaboration \cite{hasselbring2002component}.
 
 Additionally, a REST API is generated with its accompanying OpenAPI schema\footnote{\href{https://swagger.io/specification}{swagger.io/specification}} and served with a \href{https://swagger.io/}{Swagger} template. It also contains metadata about the function, for instance, its docstring, version, and version of its registered models concatenated in order to be SemVer\footnote{\href{https://semver.org/}{semver.org}} compatible. These can be seen in Figure \ref{fig:greatai-api}. This, combined with a \texttt{/version} HTTP endpoint for programmatic access and validation of the service's metadata proved to be key features when integrating the \textit{Highlights service} into ScoutinScience's service-based architecture.
 
@@ -118,13 +118,13 @@ Additionally, a REST API is generated with its accompanying OpenAPI schema\footn
     \centering
     \includegraphics[width=0.85\linewidth]{figures/greatai-api.png}
     \captionsetup{width=.9\linewidth}
-    \caption{Documentation of the automatically scaffolded REST API of a GreatAI service. Notice, how its version string includes its registered models in a SemVer compliant way: \texttt{0.0.1+small-domain-prediction-v11}.}
+    \caption{Documentation of the automatically scaffolded REST API of a \textit{GreatAI} service. Notice, how its version string includes its registered models in a SemVer compliant way: \texttt{0.0.1+small-domain-prediction-v11}.}
     \label{fig:greatai-api}
 \end{figure}
 
 \subsection{\textit{Human} integration}
 
-Even though the REST API of GreatAI services exposes all necessary features\footnote{Such as providing feedback per prediction, complexly filtering and sorting traces, create-read-update-delete (CRUD) operations for the feedback and traces, accessing live monitoring info (current configuration, versions, cache statistics), etc.} which are great for programmatic access, these are not ideal for direct human comprehension. In order to ease the introduction of GreatAI services, a rudimentary dashboard is --- optionally --- generated next to the REST API. The dashboard's main features can be observed in Figures \ref{fig:greatai-header}, \ref{fig:greatai-table}, and \ref{fig:greatai-parallel}. The diagrams and filterable/sortable table are interconnected and are automatically updated, the reactive behaviour is provided by the Dash framework \cite{shammamah_hossain-proc-scipy-2019}.
+Even though the REST API of \textit{GreatAI} services exposes all necessary features\footnote{Such as providing feedback per prediction, complexly filtering and sorting traces, create-read-update-delete (CRUD) operations for the feedback and traces, accessing live monitoring info (current configuration, versions, cache statistics), etc.} which are great for programmatic access, these are not ideal for direct human comprehension. In order to ease the introduction of \textit{GreatAI} services, a rudimentary dashboard is --- optionally --- generated next to the REST API. The dashboard's main features can be observed in Figures \ref{fig:greatai-header}, \ref{fig:greatai-table}, and \ref{fig:greatai-parallel}. The diagrams and filterable/sortable table are interconnected and are automatically updated, the reactive behaviour is provided by the Dash framework \cite{shammamah_hossain-proc-scipy-2019}.
 
 \begin{figure}
     \centering