Update thesis

thesis/chapters/5_case/2-stage.tex

@@ -0,0 +1,35 @@
+\section{A complex case}
+
+Let us now turn our attention towards a more complex component. The ScoutinScience Dashboard contains a full-page evaluation view for each academic publication. On this, the known metadata, historical data about the paper's topics, social media mentions, a PDF viewer showing the document, and other augmentation tools are displayed. One of these is the \textit{interesting sentences} section, which aims to summarise the paper from a technology-transfer perspective.
+
+The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended with the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but not adequate for providing an accurate overview. Thus, this is the baseline that I attempt to improve on in this section.
+
+\begin{displayquote}
+Compared with Section \ref{section:simple-case}, this time around, the toolset of GreatAI is at our disposal. Hopefully, this will streamline the development and --- especially --- the deployment. Given its arguably higher complexity, this experiment falls closer to industrial use-cases, and hence, can give a more accurate feedback on how to further improve the API.
+\end{displayquote}
+
+\subsection{Background}
+
+Automatic text summarisation (ATS) is one of earliest established problems of text analysis and boasts numerous promising results \cite{el2021automatic}. However, our problem requires generating a special type of summary: it must only concern a single aspect (tech-transfer) of the document. Aspect-based text summarisation has also seen some progress over the last decades \cite{berkovsky2008aspect,hayashi2021wikiasp}, but these approaches require concretely defined topics. Unfortunately, \textit{tech-transfer potential} is anything but a clear topic definition.
+
+Our numerous discussions and interviews with business developers over the last years made it clear that there is no universally agreed on definition for it. At least, all of them agrees that they know it when they see it. Additionally, most of them agree that they can confidently make a decision at the granularity of sentences. This gives rise to an obvious idea: show the experts something that they can annotate. Because the time of experts is valuable, and relevant sentences are few and far between, extra care needs to be taken to improve the ratio of positive examples in the dataset. The research of Iwatsuki Kenichi on formulaic expressions (FE) \cite{iwatsuki2020evaluation,iwatsuki2021extraction,iwatsuki2021communicative,iwatsuki2022extraction} provides a promising direction to do so. 
+
+A formulaic expression is a phrase with zero or more slots that expresses a certain intent. In the context of scientific texts, an example\footnote{Taken from the ground-truth data at \href{https://github.com/Alab-NII/FECFevalDataset/blob/master/human_evaluation/background.tsv}{github.com/Alab-NII/FECFevalDataset}} could be: \texttt{it was not until * that}. The asterisk can be substituted with multiple terms and the intention of this expression is (likely) to describe the \textit{History of the related topics}. Iwatsuki et al. identified a set of 39 intentions, compiled a manually labelled dataset \cite{iwatsuki2020evaluation}, and developed multiple approaches for automatically extracting and classifying formulaic expressions in large corpora \cite{iwatsuki2021communicative,iwatsuki2022extraction}.
+
+\subsection{Methods}
+
+In the following, we explore a 2-stage retrieval approach \cite{schutze2008introduction} commonly used in the field of information retrieval. The first stage is expected to filter out sentences that are certainly not relevant from a technology-transfer perspective using Iwatsuki's formulaic expression intention labels. Subsequently, the second stage utilises a fine-tuned SciBERT model to rank the remaining sentence based on a model learned from expert annotations.
+
+This approach has multiple shortcoming, for the first stage, we must assume the independence of sentences and that the FE intentions are strongly correlated with the sought after aspect. Additionally, the reranking only considers the individual relevance of the sentences instead of the overall relevance (utility) of the summary. It is expected, that stemming from the length of the documents and the sparseness of the selected sentences, that any combination of them is likely to have low redundancy.
+
+TODO
+
+Finetuning SciBERT \cite{jurafsky2019speech}.
+
+\subsection{Results}
+
+For measuring the interrater agreement, Cohen's kappa \cite{cohen1960coefficient} is calculated as shown in Equation \ref{equation:kappa}.
+
+\begin{equation} \label{equation:kappa}
+\kappa_{agreement} \equiv \frac{p_{observed} - p_{expected}}{1 - p_{expected}} = 1 - \frac{1 - p_{observed}}{1 - p_{expected}}
+\end{equation}

thesis/chapters/5_case/features.tex

@@ -0,0 +1,40 @@
+\section{Bridging \textbf{the gap} with GreatAI}
+
+This section briefly explores how the problems raised can be solved using GreatAI, and the API it provides to best fit the needs of its users. We first focus on the aspects of data, then, the automated wrapping of service, lastly we discuss the utility of helper functions.
+
+First, let us revisit the scope. As concluded in Section \ref{section:scope}, GreatAI should ease the \textit{transition} step between prototypes and production-ready deployments. However, this leaves open the question of what constitutes to this step? There are cross-cutting concerns such as the feature extraction code: for example, feature extraction is implemented and used in the training phase but it is also deployed alongside the model. The robustness criterion has to be met by this procedure after deployment even though its implementation is only in focus at the earlier stage of the project. Since having an untested function deployed into production can have severe repercussions, I believe, assuring its correctness lies within the scope of GreatAI.
+
+\subsection{Data}
+
+There are two kinds of data storage need we need to address: training data and trained models. Because our code is probably already tracked under Git (and likely synced with GitHub), using the Git Large File Storage (LFS)\footnote{\href{https://git-lfs.github.com/}{https://git-lfs.github.com/}} might seem intriguing. However, it is a paid (and surprisingly expensive) service of GitHub especially when we factor in the expected sizes of the models and train data with the fact that the only way remove files counting towards our quota is to \href{https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository}{recreate the repository}.
+
+The Data Version Control (DVC)\footnote{\href{https://dvc.org/}{https://dvc.org/}} open-source project provides a nearly perfect solution. It comes with a command-line interface (CLI) inspired by git's, and it can be integrated with several backend storage servers. Its only downside is of course that it is one more tool that increases the complexity of the project and the initial setup time. If this is an acceptable price to pay, then I personally recommend opting for DVC. Nevertheless, if this may prohibit a team from properly handling data according to the best practices, I present a simpler solution in the following. 
+
+The complexity of an API can be decreased by relying on its users preexisting knowledge. Therefore, we can reuse familiar API-s, such as the \texttt{open()} method from Python. A method is proposed which provides the same interface, however, the backing storage for it is a mixture of local disk space, S3-compatible storage, MongoDB, or any other storage backend. It provides a superset of \texttt{open()}'s interface; the same parameters can be used with it.
+
+Easing development isn't just about automating everything but also making the code easy to change (which is the \textit{Viscosity} dimension of CDCB). Going from opening a local file on the disk with the built-in open method, to opening  a file from S3 is as easy as changing \texttt{with open('file.txt', 'w' as f: ...} to \texttt{with LargeFileS3('file.txt', 'w' as f: ...}. In the case of the latter, an additional \texttt{version} keyword argument can also be given to lock ourselves in using as certain version which is very much desired in the case of models.
+
+The obstacles coming from the intertwined nature of different models is widely recognised \cite{sculley2015hidden,haakman2021ai,amershi2019software}. This can lead to non-monotonic error propagation, meaning that improvements in one part of the system might decrease the overall system quality \cite{amershi2019software}. The importance of schema versioning in an environment of rapidly changing models and transformations is highlighted and solved for a specific use-case in \cite{van2017versioning}.
+
+The expected features: progress bar, caching, automatically purging the cache, automatically deleting old remote version if requested are all present and come with recommended --- but easy to see and change --- configuration.
+
+\subsection{Utilities}
+
+
+utilities: clean, language, parallel map \textit{Enable Parallel Training Experiments}
+
+traces
+
+\textit{Deployment approach}
+
+% Should the order of the decorators matter? all except in one case, they're written in a way that the order doesn't matter even with the original semantics of decorators. In that one case, it cannot be written in that way. Instead of correcting a user's error, there's a mechanism looking for this error and the user is notified. Guessing the unspecified is cool, but correcting the wrong is not
+
+to do
+
+% During development, I wanted to check out which types of request fail -> log errors in traces
+% Even production systems are not perfect, saving and letting the users filter on the errors is useful. e.g. they can correlate it with the input
+
+% I use a toy example when quickly experimenting, it's important not to overfit on it ( moving it into the library would result in a online for it, so I have to consciously avoid that), but having a very simple 
+
+% Argumetn/parameter names were confusing 
+% offlinemode -> cacheonly mode 

thesis/chapters/5_case/introduction.tex

@@ -0,0 +1,11 @@
+\chapter{The ScoutinScience platform} \label{chapter:case}
+
+The core product of ScoutinScience B.V. is its platform. The clients are technology-transfer offices of Dutch and German universities, government organisations (e.g.: Wetsus), and corporates (e.g.: Heraeus Group, Ruma Rubber B.V.) who wish to extend the scope of their R\&D activities. ScoutinScience connects to multiple data sources of academic publications and integrates them into a single database. Each new publication is evaluated with a suite of AI components that ultimately determine its technology transfer potential. Other features are also extracted that help the users get a quick overview of the authors, topics, and contributions of a given piece of research.
+
+Each client organisation gets to see a different filtered view of this database ranked by the predicted probability of technology transfer opportunities being present. The main motivation is to make these business developers' and other professionals work more efficient by showing them which papers have the largest likelihood of being considered interesting by them. 
+
+To achieve this, we have a service-based architecture \cite{kleppmann2017designing} on the backend, apart from the data integration, communication, and business logic, it is made up of services wrapping simpler (phrase-matching, Naive Bayes) and more sophisticated (conditional random fields, transformer) models. As we will soon see, these can also depend on each other, for instance, based on the predicted scientific domain, a different model may be applied for scoring the paper's certain aspects.
+
+I was the first software engineer on the team which has grown considerably in the past two years. While architecting, designing, and integrating more and better models into our software solution, we noticed the same difficulties as described in Chapter \ref{chapter:background}. The gap between prototypes and production-ready services is larger than it seems. It is also larger than it should be. This motivated me to investigate the state-of-the-art and had found that it is insufficient in many cases. Since the ScoutinScience platform is a quite typical example of applying AI in the industry, it will serve as the real-life case, problem context, and testbed for attempting to design a solution which can advance the state-of-the-art.
+
+In this chapter, the process of designing GreatAI is described along with how it fits into real-life use cases. First, a simple experiment is presented which leads to the implementation of a service, then, as the featureset of the library grows and matures, a more complex service is developed. Subsequently, the close to final library version is used to refactor existing ScoutinScinece services in order to further refine the API of GreatAI. Lastly, the final version of the design is presented and qualitatively evaluated to verify how well it satisfies the requirements described in Section \ref{section:requirements}.

thesis/chapters/5_case/main.tex

@@ -0,0 +1,10 @@
+\input{chapters/5_case/introduction}
+
+\input{chapters/5_case/naive-bayes}
+\input{chapters/5_case/features}
+
+\input{chapters/5_case/2-stage}
+
+\input{chapters/5_case/refactoring}
+
+\input{chapters/5_case/results}

thesis/chapters/5_case/naive-bayes.tex

@@ -0,0 +1,97 @@
+\section{A simple case} \label{section:simple-case}
+
+Using different models for slight variations of the same problem is quite commonplace in the industry. For instance, UberEats has a vast, hierarchical set of models for every country, region, and city for calculating the estimated time of delivery \cite{li2017scaling}. We have also found, that in order to best process an academic publication, knowing its domain is essential. The reason for this can be (among others) the wildly different vocabularies of different domains. For example, the term \textit{framework} in computer science almost always refers to a software artifact (usually implying high tech-transfer potential), while in every other domain, \textit{framework} is used to describe theoretical models that are less central to practical applications. Of course, it is not merely the meaning of the terms but more importantly, their distribution that varies significantly. Therefore, the topic of this section is to design and develop a domain prediction model for academic papers.
+
+\subsection{Background}
+
+Fortunately, this is one of the oldest subjects of text classification. In fact, Maron introduced the Naive Bayes classifier in 1961 for exactly this purpose: classifying documents' subjects. To look at a more recent approach, SciBERT \cite{beltagy2019scibert} --- a BERT \cite{devlin2018bert} model pretrained on academic publications --- was also used for a similar task in which the domains of sentences have to be decided\footnote{\href{https://paperswithcode.com/sota/sentence-classification-on-paper-field}{paperswithcode.com/sota/sentence-classification-on-paper-field}}. It achieved an F1-score of $0.6571$ after being pretrained on the Semantic Scholar Corpus (SSC) \cite{Lo2020S2ORCTS} and finetuned on the train split of the Microsoft Academic Graph (MAG) dataset \cite{wang2019review}\footnote{SciBERT was applied to a preprocessed version of this dataset available at \href{https://github.com/allenai/scibert/tree/master/data/text_classification/mag}{github.com/allenai/scibert/tree/master/data/text\_classification/mag}}.
+
+\begin{displayquote}
+\textbf{Design note} After getting familiar with the context, it is time to focus on experimenting and developing our domain prediction service. At the same time, the difficulties encountered should be noted and integrated into GreatAI's design.
+\end{displayquote}
+
+\subsection{Data}
+
+Two datasets will be considered for the experiments. SciBERT's MAG and the SSC. The former is used to compare the results with SciBERT's, while the latter is utilised for training a model for production purposes because it has 19 labels compared with MAG's 7 and it also contains abstracts instead of just sentences, thus, it is more fitting for our use-case.
+
+SciBERT's version of the MAG dataset has 84 thousand and 22.3 thousand sentences in its train and test splits respectively. These are mostly in English and have all punctuation and casing removed. Each sentence is classified as belonging to one of seven fields. Figure \ref{fig:mag-distribtion} shows that the classes have a uniform distribution. 
+
+\begin{figure}
+    \centering
+    \includegraphics[width=\linewidth]{figures/mag-distribution.png}
+    \caption{Class distribution of the MAG  \cite{wang2019review} dataset's 84000 sentences in its \textit{train} split.}
+    \label{fig:mag-distribtion}
+\end{figure}
+
+SSC is much larger: it contains over 80 million abstracts. Having more data certainly helps in sampling the term distribution more accurately, the law of diminishing returns apply, especially when using simple models. Therefore, the data will be randomly downsampled to leave us with a more manageable couple of hundred megabytes of abstracts. We can see the distribution of class labels in Figure \ref{fig:ss-distribution}. The dataset is considerably less balanced: \textit{medicine} is by far the most popular field.
+
+\begin{figure}
+    \centering
+    \includegraphics[width=\linewidth]{figures/ss-distribution.png}
+    \caption{Label distribution of the Semantic Scholar dataset \cite{Lo2020S2ORCTS}. The \textit{variable} refers to the position of the domain in the list of domains assigned to a paper.}
+    \label{fig:ss-distribution}
+\end{figure}
+
+\begin{displayquote}
+\textbf{Where should we store this data?} "On my machine" seems like an easy answer. However, if we have a team working with the data or it has intrinsic value, it must be stored in an easy-to-access, potentially redundant way. Serban et al. \cite{serban2020adoption} expressed this need in the following best practice: \textit{Make Data Sets Available on Shared Infrastructure (private or public)}. Meanwhile, wherever data is stored, it should also be versioned to satisfy the next best practice: \textit{Use Versioning for Data, Model, Configurations and Training Scripts}.
+\end{displayquote}
+
+MAG needs no further preprocessing if we aim to match SciBERT's setup \cite{beltagy2019scibert}. But since SSC contains a heap of metadata, the relevant parts have to be extracted and preprocessed. In this case, these are the the concatenation of the abstract's text, paper's title and the journal's name along with the paper's domains (there can be multiple domains for a single paper, it is a mulitlabel classification task). Lastly, the non-English entries are discarded because we only expect to process papers in  English.
+
+\begin{displayquote}
+\textbf{How should we preprocess the data?} These simple processing steps (filter, map, project) are almost always present in the data science life-cycle. For example, cleaning the input text from various HTML, OCR, PDF, or \LaTeX \hskip 0.12cm extraction artifacts is almost always necessary for text analysis. This is captured in the AI best-practices collection under the following category: \textit{Write Reusable Scripts for Data Cleaning and Merging}. Also, the best practice of \textit{Test all Feature Extraction Code} is somewhat applicable: the applied processing steps must not introduce unwanted artifacts.
+\end{displayquote}
+
+\subsection{Methods}
+
+Since the aim is to classify papers to allow the ScoutinScience platform to select models which have been trained on a matching vocabulary (and domain), it seems reasonable that only considering the distribution (frequencies) of individual terms may be sufficient. To test this hypothesis, a unigram language model (Multinomial Naive Bayes) is constructed and its accuracy is compared with SciBERT's. The former definitely aligns with the advice to \textit{Use The Most Efficient Models}.
+
+Using the MNB implementation of scikit-learn \cite{pedregosa2011scikit}, it only took a couple of lines to create, hyperparameter-optimise, and test a text classifier. Including data loading and visualisations, it takes 71 LOC to be more precise. \footnote{The code is available at \href{https://github.com/ScoutinScience/great-ai/blob/main/examples/simple-mag/train.ipynb}{github.com/ScoutinScience/great-ai/blob/main/examples/simple-mag/train.ipynb}} This further proves relatively how simple it is to apply existing algorithms. The code can be considered for satisfying the \textit{Automate Hyper-Parameter Optimisation} best-practice, since it also implements an automated hyperparameter sweep. 
+
+The sentences are tokenised into words and vectorised with TF-IDF (with logarithmic term frequency) \cite{buckley1985implementation}, the hyperparameters found via 3-fold cross-validation on the \textit{train} split lead to filtering out tokens which occur in fewer than 5 documents or more than 5\% of the documents.
+
+\begin{displayquote}
+\textbf{What could be automated here?} As discussed in Section \ref{section:accessible-ai}, libraries exposing algorithms and state-of-the-art models can already be considered mature and accessible. In this case, only scikit-learn was utilised, but subjectively, most popular libraries have a similarly easy to use use API. Therefore, I see no urgent need for further action regarding the \textit{experimentation} step of the life-cycle in connection with the AI best practices.
+\end{displayquote}
+
+\subsection{Results \& Discussion}
+
+\begin{figure}
+    \centering
+    \includegraphics[width=0.8\linewidth]{figures/confusion-matrix.png}
+    \caption{Confusion matrix of a Naive Bayes classifier on the MAG dataset's sentences. The matrix is normalised column-wise. Notice, how most mistakes happen between semantically similar classes, for instance: \textit{politics} -- \textit{sociology} or \textit{business} -- \textit{economics}.}
+    \label{fig:mag-confusion}
+\end{figure}
+
+\begin{figure}
+    \centering
+    \includegraphics[width=\linewidth]{figures/ss-confusion.png}
+    \caption{Confusion matrix of a Naive Bayes classifier on the SSC dataset's sentences. The matrix is normalised column-wise. Notice, how most mistakes happen between semantically similar classes, for instance:  \textit{philosohpy} -- \textit{sociology} or \textit{history} -- \textit{art}.}
+    \label{fig:ss-confusion}
+\end{figure}
+
+When this model is applied to the \textit{test} split of MAG, we get the confusion matrix of Figure \ref{fig:mag-confusion}. This Naive Bayes classifier achieves a whopping $0.6795$ F1-score. This is $3.4\%$ better than SciBERT's on the same dataset. Thus, it seems, MNB clearly outperforms SciBERT for this particular use-case: it is not only more accurate, its model is magnitudes smaller, while it is also considerably faster to train (or finetune in the case of SciBERT) and use.
+
+It is, of course, not entirely surprising that the sophisticated transformer architecture of SciBERT is not necessary for a plain task like this. Apart from phrases, the relation between separate words of a sentence do not carry nearly as much discriminative power as the identity of the terms\footnote{On a similar note, the independence assumption of Naive Bayes is often less wrong than it might seem \cite{hand2001idiot}.}, hence there is little reason for using an attention mechanism. The fact that SciBERT even works in any way on this task is already a testament to its general applicability. Nevertheless, this short experiment has proved that we can safely opt for using MNB for production.
+
+Since Multinomial Naive Bayes is best at returning a single label and SSC is has multiple labels per datapoint: for evaluation purposes, it is checked whether the returned label is contained in the labels of the ground truth. On this dataset, MNB achieves a significantly lower macro-average F1-score which is 0.49. The weighted-average F1 is 0.61 and the overall accuracy is 62\%. The large difference between the macro and weighted averages come from the unbalanced distribution of the labels, better performance could be achieved by uniformly sampling from each class.
+
+The lower F1-score is not surprising because there are more than twice as many classes in this dataset,  Additionally, the mistakes made are defendable when we look at Figure \ref{fig:ss-confusion}: most of them are between close or related classes.
+
+\begin{displayquote}
+This is the usual point where papers conclude: a proof-of-concept/prototype has been built and its performance demonstrated, measured --- and usually --- explained. Nonetheless, in an industrial setting, our problem is far from being solved: it has yet to be deployed.
+\end{displayquote}
+
+\subsection{Deployment}
+
+First, an inference function needs to be written that can take an input on the fly and calculate a corresponding prediction. Since we aim to follow the best practices, namely: \textit{Explain Results and Decisions to Users} and \textit{Employ Interpretable Models When Possible}, giving an explanation of the results is expected. Fortunately, with our simple model it's easy to determine the most influential weights, thus, words. The last deployment step may be to provide access to our model for others.
+
+\begin{displayquote}
+\textbf{How do we provide an interface for the inference function?} We either have an offline or online inference workflow (or both). For the former, we have to provide a way to use it in batch processing; a simple Python function may be adequate for this purpose, though, allowing it to be easily (or automatically) parallelised would make its consumers' DX better. If it is an online-workflow, we must have a service running continuously and accepting input at any time. This can be achieved by a remote procedure call (RPC) interface, or more commonly, a web API. Developers usually refer to these as REST API-s, sometimes, they even follow the conventions of REST. Either ways, we must develop a wrapper over the service in order to make it available for other internal/external consumers.
+\end{displayquote}
+
+According to the research on the adoption of best practices, this is where many real-world projects conclude. This happens to be \textbf{the gap}. Believing that solely focusing on the research and experiments is good enough is a fallacy: when following this approach, the deployment step ends up being a rushed attempt of wrapping the \textit{AI} and putting it in the production environment. This is inarguably a deployment. However, it follows very few of the best practices. This can lead to suboptimal real-life performance, lack of accountability, lack of opportunity to improve, and can overall lead to negative societal impact \cite{o2016weapons}.
+
+\begin{displayquote}
+\textbf{How could we implement more best practices?} The most notable missing best practices are the lack of automated deployment, automated regression testing, online monitoring, persisting the traces, graceful error-handling, taking feedback on the results (if possible in the use-case), calculating the online accuracy based on the feedback, and retraining the model if necessary using novel data. These all correspond to a best practice.
+\end{displayquote}

thesis/chapters/5_case/refactoring.tex

@@ -0,0 +1 @@
+\section{Refactoring with GreatAI}

thesis/chapters/5_case/results.tex

@@ -0,0 +1,38 @@
+\section{Results}
+
+\begin{table}
+\centering
+\caption{A subset of the AI lifecycle \href{https://se-ml.github.io/practices/}{best practices identified by Serban et al.} \cite{serban2020adoption,serban2021practices} and the level of support GreatAI provides for them. \textit{Full} requires no action from the user, \textit{Partial} requires at least some involvement, while \textit{Slight} provides some useful features but the client is still expected to make a significant effort.}
+\label{table:best-practices}
+\begin{tabular}{p{7cm}@{\hskip 0.5cm}c@{\hskip 0.5cm}c}
+\hline
+\textbf{Best practice} & \textbf{Implementation} & \textbf{Level of support} \\\hline
+
+Use Sanity Checks for All External Data Sources  & \texttt{great\_ai.parameter} & Partial \\\hline
+Check that Input Data is Complete, Balanced and Well Distributed  & Type-checked input & Slight \\\hline
+Write Reusable Scripts for Data Cleaning and Merging & \texttt{great\_ai.utilities} & Partial \\\hline
+Make Data Sets Available on Shared Infrastructure (private or public) & \texttt{great\_ai.large\_file} & Full \\\hline
+
+Test all Feature Extraction Code & \texttt{great\_ai.utilities} & Partial \\\hline
+Employ Interpretable Models When Possible & \texttt{great\_ai} & Slight \\\hline
+Enable Parallel Training Experiments & \texttt{great\_ai.parallel\_map} & Partial \\\hline
+Continuously Measure Model Quality and Performance & \texttt{great\_ai} & Full \\\hline
+Use Versioning for Data, Model, Configurations and Training Scripts & \texttt{great\_ai.large\_file} & Full \\\hline
+
+Run Automated Regression Tests & \texttt{great\_ai} & Full \\\hline
+Use Continuous Integration & Docker Images \& scripts & Partial \\\hline
+Use Static Analysis to Check Code Quality & Typed API & Partial \\\hline
+Assure Application Security & GreatAI is audited & Partial \\\hline
+
+Automate Model Deployment & Docker Images \& scripts & Partial \\\hline
+TODO: Enable Shadow Deployment & GreatAI & Full \\\hline
+Continuously Monitor the Behaviour of Deployed Models & \texttt{great\_ai} & Full \\\hline
+Enable Automatic Roll Backs for Production Models &  Docker Images & Partial \\\hline
+Log Production Predictions with the Model's Version and Input Data & GreatAI & Full \\\hline
+
+Explain Results and Decisions to Users & GreatAI & Slight \\\hline
+\end{tabular}
+\end{table}
+
+
+Table \ref{table:best-practices} summarises the implemented best practices.