Work on thesis
This commit is contained in:
parent
5a143b4026
commit
573ada2d63
12 changed files with 265 additions and 124 deletions
|
|
@ -17,7 +17,7 @@ The utility of \textit{GreatAI} is validated using the principles of design scie
|
|||
I hypothesise that facilitating the adoption of AI deployment best practices is viable by finding less complex framework designs which are easier to adopt in order to decrease the negative externality of misused AI. This paper is set out to investigate this hypothesis by answering the following research questions.
|
||||
|
||||
\begin{rqlist}
|
||||
\item To what extent does the complexity of AI deployment frameworks hinder industrial applications?
|
||||
\item To what extent does the complexity of AI deployment hinder industrial applications?
|
||||
\item What API design techniques can be effectively applied in order to decrease the complexity of correctly deploying AI services?
|
||||
\item To what extent can \textit{GreatAI} automatically implement AI deployment best practices?
|
||||
\item How adequate is the design of \textit{GreatAI} for helping to apply best practices in other contexts?
|
||||
|
|
|
|||
|
|
@ -26,7 +26,9 @@ Similarly, Thiée \cite{thiee2021systematic} describes the slow but ever-growing
|
|||
|
||||
Serban et al. \cite{serban2020adoption,serban2021practices} describe the results of their global surveys aiming to ascertain the SOTA in how teams develop, deploy, and maintain ML systems. In \cite{serban2020adoption}, they compiled a set of 29 actionable best practices. These were analysed and validated with a survey of 313 participants to discover the adoption rate and relative importance of each best practice. For example, they determined the most important best practice to be \textit{logging production prediction traces}, however, the adoption was measured to be below 40\%. In more than three quarters of the cases, newcomers to AI reported that they \textit{partially} or \textit{not at all} follow best practices. This tendency decreases with more years of experience, reaching a maximum adoption rate of just above 60\%. In a similar fashion, Serban et al. in \cite{serban2021practices}, identify another 14 best practices that concern trustworthy AI mainly through data governance. They strive to complement high-level checklists with actionable best practices. Analysing 42 survey responses reveals a familiar pattern. Most best practices have less than 50\% adoption.
|
||||
|
||||
Finally, Bosch et al. \cite{bosch2021engineering} organise and structure the problem space of AI engineering research based on their 16 primary case studies. The authors note the increasing and broad adoption of ML in the industry, while also emphasising that \textit{transition from prototype to production-quality deployment} proves to be challenging for many companies. Large amounts of software engineering expertise is required to create additional facilities for the application such as data pipelines, monitoring, and logging. They define \textit{deployment \& compliance} to be one of the four main categories of problems and describe it as highly underestimated and the source of ample struggle.
|
||||
John et al. \cite{john2020architecting} compare and contrast the recent scientific and grey literature of AI deployments from which they extract concrete challenges and practices. They also observe that most companies are placing a lot more models into production compared with previous years. Additionally, they point out that many deployment techniques are absent in contemporary literature which is speculated to be caused by the immaturity of deployment processes employed in academia. Because for instance, most models in scientific literature experience only initial deployment and are not constantly replaced or refreshed as performance degrades over time.
|
||||
|
||||
Finally, in a follow-up study to \cite{john2020architecting}, Bosch et al. \cite{bosch2021engineering} organise and structure the problem space of AI engineering research based on their 16 primary case studies. The authors note the increasing and broad adoption of ML in the industry, while also emphasising that the \textit{transition from prototype to production-quality deployment} proves to be challenging for many companies. Large amounts of software engineering expertise is required to create additional facilities for the application such as data pipelines, monitoring, and logging. They define \textit{deployment \& compliance} to be one of the four main categories of problems and describe it as highly underestimated and the source of ample struggle.
|
||||
|
||||
\section{Existing solutions} \label{section:existing}
|
||||
|
||||
|
|
@ -54,21 +56,21 @@ Additionally, increasing attention is given to ML deployments in embedded system
|
|||
\begin{tabular}{|l|c|c|c|c|c|c|c|}
|
||||
\hline
|
||||
& AutoAI & Azure ML & SageMaker & TFX & TorchX & MLflow & Seldon Core \\ \hline
|
||||
Open-source\textsuperscript{1}& & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Self-hosted\textsuperscript{1}& & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Vendor-agnostic\textsuperscript{2}& & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
AI-agnostic\textsuperscript{2}& & \checkmark & \checkmark & & & \checkmark & \checkmark \\ \hline
|
||||
E2E feedback\textsuperscript{3}& & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
||||
Distributed monitoring\textsuperscript{3}& & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\textsuperscript{*} & \checkmark \\ \hline
|
||||
Online model selection\textsuperscript{3}& \checkmark\textsuperscript{*} & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
||||
Versioning\textsuperscript{3}& \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Quick setup\textsuperscript{4}& \checkmark & \checkmark & & & & & \\ \hline
|
||||
Open-source\textsuperscript{1} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Self-hosted\textsuperscript{1} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Vendor-agnostic\textsuperscript{2} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
AI-agnostic\textsuperscript{2} & & \checkmark & \checkmark & & & \checkmark & \checkmark \\ \hline
|
||||
E2E feedback\textsuperscript{3} & & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
||||
Distributed monitoring\textsuperscript{3} & & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\textsuperscript{*} & \checkmark \\ \hline
|
||||
Online model selection\textsuperscript{3} & \checkmark\textsuperscript{*} & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
||||
Versioning\textsuperscript{3} & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
||||
Quick setup\textsuperscript{4} & \checkmark & \checkmark & & & & & \\ \hline
|
||||
No DevOps dependencies\textsuperscript{4}& & & & & \checkmark & & \\ \hline
|
||||
\end{tabular}}
|
||||
\begin{tablenotes}
|
||||
\item[1] For privacy and accountability reasons. \cite{bosch2021engineering}
|
||||
\item[2] Minimising required glue code. \cite{sculley2015hidden}
|
||||
\item[3] Implementing best practices. \cite{serban2020adoption,serban2021practices}
|
||||
\item[3] Implementing best practices. \cite{serban2020adoption,serban2021practices,john2020architecting}
|
||||
\item[4] Easy integration into existing processes. \cite{haakman2021ai,thiee2021systematic}
|
||||
\item[*] Only partial support.
|
||||
\end{tablenotes}
|
||||
|
|
@ -81,4 +83,4 @@ In summary, the problems expressed in Section \ref{section:industry} can be unde
|
|||
|
||||
The surveys and case studies have shown the industry's continuous struggle to evolve their prototypes into robust and responsible production-ready deployments. Simultaneously, platforms aiming to help overcome this challenge already exist but lack widespread adoption. The frequently recurring explanations for not adopting existing solutions surfaced in Section \ref{section:industry} revolve around their complexity and rigidity. These complaints are validated when looking at the available frameworks in Section \ref{section:existing}. While using AI has become more accessible than ever, deploying remains challenging owing to the lack of any \textit{easy-to-use framework for robust end-to-end AI deployments}.
|
||||
|
||||
The coexistence of multiple major obstacles along with their promised solutions and the lack of their wide-spread adoption leads us to believe that current frameworks are inadequate for many contexts. There is an unmet need for accessible AI deployment methods. The revolution brought by FLAIR, HuggingFace, and similar libraries for the domain of ML remains unmatched in the domain of AI Engineering.
|
||||
The coexistence of multiple major obstacles along with their promised solutions and the lack of their wide-spread adoption leads us to believe that current frameworks are inadequate for many contexts. Thus, the answer to \textbf{RQ1} is that the complexity of deploying AI can severely hinder industrial applications even in the presence of existing frameworks. There is an unmet need for accessible AI deployment methods. The revolution brought by FLAIR, HuggingFace, and similar libraries for the domain of ML remains unmatched in the domain of AI Engineering.
|
||||
|
|
|
|||
|
|
@ -1,11 +0,0 @@
|
|||
\chapter{The ScoutinScience platform} \label{chapter:case}
|
||||
|
||||
The core product of \href{https://scoutinscience.com/}{ScoutinScience B.V.} is its platform. The clients are technology-transfer offices of Dutch and German universities, government organisations (e.g.: Wetsus), and corporates (e.g.: Heraeus Group, Ruma Rubber B.V.) who wish to extend the scope of their R\&D activities. ScoutinScience connects to multiple data sources of academic publications and integrates them into a single database. Each new publication is evaluated with a suite of AI components that ultimately determine its technology transfer potential. Other features are also extracted that help the users get a quick overview of the authors, topics, and contributions of a given piece of research.
|
||||
|
||||
Each client organisation gets to see a different filtered view of this database ranked by the predicted probability of technology transfer opportunities being present. The main motivation is to make these business developers' and other professionals work more efficient by showing them which papers have the largest likelihood of being considered interesting by them.
|
||||
|
||||
To achieve this, we have a service-based architecture \cite{kleppmann2017designing} on the backend, apart from the data integration, communication, and business logic, it is made up of services wrapping simpler (phrase-matching, Naïve Bayes) and more sophisticated (conditional random fields, transformer) models. As we will soon see, these can also depend on each other, for instance, based on the predicted scientific domain, a different model can be chosen for scoring certain aspects of papers.
|
||||
|
||||
I was among the first engineers on the team which has grown considerably in the past two years. While architecting, designing, and integrating more and better models into our software solution, I experienced the same difficulties as described in Chapter \ref{chapter:background}. The gap between prototypes and production-ready services is larger than it seems. It is also larger than it should be. This motivated me to investigate the state-of-the-art and had found that it is insufficient in many cases. Since the ScoutinScience platform is a quite typical example of applying AI in the industry, it will serve as the real-life case, problem context, and testbed for attempting to design a solution which can advance the state-of-the-art.
|
||||
|
||||
In this chapter, the process of designing \textit{GreatAI} is described along with how it fits into real-life use cases. First, a simple experiment is presented which leads to the implementation of a service, subsequently, as the feature-set of the library grows and matures, a more complex software service is developed. Lastly, the final version of the design is presented and qualitatively evaluated to verify how well it satisfies the requirements described in Section \ref{section:requirements}.
|
||||
|
|
@ -1,4 +1,14 @@
|
|||
\input{chapters/5_cases/introduction}
|
||||
\chapter{The ScoutinScience platform} \label{chapter:case}
|
||||
|
||||
The core product of \href{https://scoutinscience.com/}{ScoutinScience B.V.} is its platform. The clients are technology-transfer offices of Dutch and German universities, government organisations (e.g.: Wetsus), and corporates (e.g.: Heraeus Group, Ruma Rubber B.V.) who wish to extend the scope of their R\&D activities. ScoutinScience connects to multiple data sources of academic publications and integrates them into a single database. Each new publication is evaluated with a suite of AI components that ultimately determine its technology transfer potential. Other features are also extracted that help the users get a quick overview of the authors, topics, and contributions of a given piece of research.
|
||||
|
||||
Each client organisation gets to see a different filtered view of this database ranked by the predicted probability of technology transfer opportunities being present. The main motivation is to make these business developers' and other professionals work more efficient by showing them which papers have the largest likelihood of being considered interesting by them.
|
||||
|
||||
To achieve this, we have a service-based architecture \cite{kleppmann2017designing} on the backend, apart from the data integration, communication, and business logic, it is made up of services wrapping simpler (phrase-matching, Naïve Bayes) and more sophisticated (conditional random fields, transformer) models. As we will soon see, these can also depend on each other, for instance, based on the predicted scientific domain, a different model can be chosen for scoring certain aspects of papers.
|
||||
|
||||
I was among the first engineers on the team which has grown considerably in the past two years. While architecting, designing, and integrating more and better models into our software solution, I experienced the same difficulties as described in Chapter \ref{chapter:background}. The gap between prototypes and production-ready services is larger than it seems. It is also larger than it should be. This motivated me to investigate the state-of-the-art and had found that it is insufficient in many cases. Since the ScoutinScience platform is a quite typical example of applying AI in the industry, it will serve as the real-life case, problem context, and testbed for attempting to design a solution which can advance the state-of-the-art.
|
||||
|
||||
In this chapter, the process of designing \textit{GreatAI} is described along with how it fits into real-life use cases. First, a simple experiment is presented which leads to the implementation of a service, subsequently, as the feature-set of the library grows and matures, a more complex software service is developed. Lastly, the final version of the design is presented and qualitatively evaluated to verify how well it satisfies the requirements described in Section \ref{section:requirements}.
|
||||
|
||||
\input{chapters/5_cases/naive-bayes}
|
||||
\input{chapters/5_cases/scibert}
|
||||
\input{chapters/5_cases/results}
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ SciBERT's version of the MAG dataset has 84 thousand and 22.3 thousand sentences
|
|||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.5\linewidth]{figures/mag-distribution.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Class distribution of the MAG \cite{wang2019review} dataset's 84000 sentences in its \textit{train} split.}
|
||||
\label{fig:mag-distribtion}
|
||||
\end{figure}
|
||||
|
|
@ -30,6 +31,7 @@ SSC is much larger: it contains over 80 million abstracts. Having more data cert
|
|||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.8\linewidth]{figures/ss-distribution.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Label distribution of the Semantic Scholar dataset \cite{Lo2020S2ORCTS}. Each publication may be assigned at most 3 domains.}
|
||||
\label{fig:ss-distribution}
|
||||
\end{figure}
|
||||
|
|
@ -63,6 +65,7 @@ The sentences are tokenised into words and vectorised with TF-IDF (with logarith
|
|||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.8\linewidth]{figures/mag-confusion.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Confusion matrix of a Naïve Bayes classifier on the MAG dataset's sentences. The matrix is normalised column-wise. Notice, how most mistakes happen between semantically similar classes, for instance: \textit{politics} -- \textit{sociology} or \textit{business} -- \textit{economics}.}
|
||||
\label{fig:mag-confusion}
|
||||
\end{figure}
|
||||
|
|
@ -70,6 +73,7 @@ The sentences are tokenised into words and vectorised with TF-IDF (with logarith
|
|||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=\linewidth]{figures/ss-confusion.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Confusion matrix of a Naïve Bayes classifier on the SSC dataset's sentences. The matrix is normalised column-wise. Notice, how most mistakes happen between semantically similar classes, for instance: \textit{philosohpy} -- \textit{sociology} or \textit{history} -- \textit{art}.}
|
||||
\label{fig:ss-confusion}
|
||||
\end{figure}
|
||||
|
|
@ -100,7 +104,7 @@ According to the body of research on the adoption of best practices, this is whe
|
|||
\textbf{How could we implement more best practices?} The most notable missing best practices are the lack of automated deployment, automated regression testing, online monitoring, persisting the traces, graceful error-handling, taking feedback on the results (if possible in the use-case), calculating the online accuracy based on the feedback, and retraining the model if necessary using novel data. These all correspond to best practices.
|
||||
\end{displayquote}
|
||||
|
||||
\section{Bridging \textbf{the gap} with GreatAI}
|
||||
\section{Bridging the gap with GreatAI}
|
||||
|
||||
First, let us revisit the library's scope for clarification. As concluded in Section \ref{section:scope}, GreatAI should ease the \textit{transition} step between prototypes and production-ready deployments. However, this leaves open the question of what constitutes to this step? There are cross-cutting concerns, for example, feature extraction is implemented and used in the training phase but it is also deployed alongside the model. The robustness criterion has to be met by this procedure even though its implementation is only in focus in the earlier stages of the project. Since having an untested function deployed into production can have severe repercussions, I conclude, assuring its correctness lies within the scope of GreatAI.
|
||||
|
||||
|
|
@ -193,6 +197,7 @@ After implementing some features of the library it can be already used for deplo
|
|||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.7\linewidth]{figures/dashboard-domains.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Screenshot of the domain prediction integrated into the ScoutinScience platform where it is used as a filtering option.}
|
||||
\label{fig:dashboard-domains}
|
||||
\end{figure}
|
||||
|
|
|
|||
|
|
@ -1,62 +0,0 @@
|
|||
\section{Results}
|
||||
|
||||
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.9\linewidth]{figures/greatai-header.png}
|
||||
\caption{}
|
||||
\label{fig:greatai-header}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\textwidth]{figures/greatai-table.png}
|
||||
\caption{}
|
||||
\label{fig:greatai-table}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\textwidth]{figures/greatai-parallel.png}
|
||||
\caption{}
|
||||
\label{fig:greatai-parallel}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\begin{table}
|
||||
\centering
|
||||
\caption{A subset of the AI lifecycle \href{https://se-ml.github.io/practices/}{best practices identified by Serban et al.} \cite{serban2020adoption,serban2021practices} and the level of support GreatAI provides for them. \textit{Full} requires no action from the user, \textit{Partial} requires at least some involvement, while \textit{Slight} provides some useful features but the client is still expected to make a significant effort.}
|
||||
\label{table:best-practices}
|
||||
\begin{tabular}{p{7cm}@{\hskip 0.5cm}c@{\hskip 0.5cm}c}
|
||||
\hline
|
||||
\textbf{Best practice} & \textbf{Implementation} & \textbf{Level of support} \\\hline
|
||||
|
||||
Use Sanity Checks for All External Data Sources & \texttt{great\_ai.parameter} & Partial \\\hline
|
||||
Check that Input Data is Complete, Balanced and Well Distributed & Type-checked input & Slight \\\hline
|
||||
Write Reusable Scripts for Data Cleaning and Merging & \texttt{great\_ai.utilities} & Partial \\\hline
|
||||
Make Data Sets Available on Shared Infrastructure (private or public) & \texttt{great\_ai.large\_file} & Full \\\hline
|
||||
|
||||
Test all Feature Extraction Code & \texttt{great\_ai.utilities} & Partial \\\hline
|
||||
Employ Interpretable Models When Possible & \texttt{great\_ai} & Slight \\\hline
|
||||
Enable Parallel Training Experiments & \texttt{great\_ai.parallel\_map} & Partial \\\hline
|
||||
Continuously Measure Model Quality and Performance & \texttt{great\_ai} & Full \\\hline
|
||||
Use Versioning for Data, Model, Configurations and Training Scripts & \texttt{great\_ai.large\_file} & Full \\\hline
|
||||
|
||||
Run Automated Regression Tests & \texttt{great\_ai} & Full \\\hline
|
||||
Use Continuous Integration & Docker Images \& scripts & Partial \\\hline
|
||||
Use Static Analysis to Check Code Quality & Typed API & Partial \\\hline
|
||||
Assure Application Security & GreatAI is audited & Partial \\\hline
|
||||
|
||||
Automate Model Deployment & Docker Images \& scripts & Partial \\\hline
|
||||
TODO: Enable Shadow Deployment & GreatAI & Full \\\hline
|
||||
Continuously Monitor the Behaviour of Deployed Models & \texttt{great\_ai} & Full \\\hline
|
||||
Enable Automatic Roll Backs for Production Models & Docker Images & Partial \\\hline
|
||||
Log Production Predictions with the Model's Version and Input Data & GreatAI & Full \\\hline
|
||||
|
||||
Explain Results and Decisions to Users & GreatAI & Slight \\\hline
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
|
||||
Table \ref{table:best-practices} summarises the implemented best practices.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
\section{Text summarisation with SciBERT}
|
||||
\section{Text summarisation with SciBERT} \label{section:complex-case}
|
||||
|
||||
Let us now turn our attention towards a more complex component. The ScoutinScience Dashboard\footnote{\href{https://dashboard.scoutinscience.com/}{dashboard.scoutinscience.com}} contains a full-page evaluation view for each academic publication. On this, the known metadata, historical trends about the paper's topics, social media mentions, a PDF viewer showing the document, and other augmentation tools are displayed. One of these is the \textit{Highlights} section, which aims to summarise the paper from a technology-transfer perspective.
|
||||
Let us now turn our attention towards a more complex component. The ScoutinScience Dashboard\footnote{\href{https://dashboard.scoutinscience.com/}{dashboard.scoutinscience.com}} contains a full-page evaluation view for academic publications. On this, the known metadata, historical trends about the paper's topics, social media mentions, a PDF viewer showing the document, and other augmentation tools are displayed. One of these is the \textit{Highlights} section, which aims to summarise the paper from a technology-transfer perspective.
|
||||
|
||||
The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended by the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but not adequate for providing an accurate overview. Thus, this is the baseline that I attempt to improve on in this section.
|
||||
|
||||
|
|
@ -20,17 +20,18 @@ A formulaic expression is a phrase with zero or more ``slots'' which when filled
|
|||
|
||||
\subsection{Methods}
|
||||
|
||||
In order to compile our dataset, experts are asked to judge sentences that passed an \textit{intention check}. This pooling approach is commonly used in the field of information retrieval \cite{schutze2008introduction}. The filtering stage is expected to sieve out sentences that are probably not relevant from a technology-transfer perspective using Iwatsuki's formulaic expression intention labels. Subsequently, relevance judgements --- in the form of \textit{interesting} or \textit{not interesting} labels --- are gathered for these sentences. This method turns the extractive summarisation into a binary classification task for which a SciBERT model \cite{beltagy2019scibert} can be finetuned. Ultimately, the summaries are derived from sentences that are selected by the classifier trained on the experts' annotations.
|
||||
In order to compile a new dataset, experts are asked to judge sentences that passed an \textit{intention check}. This pooling approach is commonly used in the field of information retrieval \cite{schutze2008introduction}. The filtering is expected to sieve out sentences that are probably not relevant from a technology-transfer perspective using Iwatsuki's formulaic expression intention labels. Subsequently, relevance judgements --- in the form of \textit{interesting} or \textit{not interesting} labels --- are gathered for these sentences. This method turns the extractive summarisation into a binary classification task for which a SciBERT model \cite{beltagy2019scibert} can be finetuned. Ultimately, the summaries are derived from sentences that are selected by the classifier trained on the experts' annotations.
|
||||
|
||||
We have to note two possible shortcomings of this setup: firstly, we assume the FE intentions are strongly correlated with the sought-after aspect, this may or may not be true. secondly, only the individual relevance of the sentences is considered instead of the overall relevance (utility) of the summary. Nonetheless, it is expected that stemming from the length of the documents and the sparseness of the selected sentences, that any combination of them is likely to have low redundancy.
|
||||
|
||||
\subsection{Results}
|
||||
|
||||
For the first iteration, 1500 sentences were selected for 2 experts to annotate in a binary fashion according to strict guidelines. An example is shown in Figure \ref{fig:annotator}. Afterwards, for measuring the interrater agreement, Cohen's kappa \cite{cohen1960coefficient} is calculated as shown in Equation \ref{equation:kappa}. Which turns out to be \textbf{0.4310} for the two annotators. This happens to be just above the lower end of \textit{moderate agreement}. However, we have to note that the original quality ranges are often criticised for being too relaxed \cite{mchugh2012interrater}. However, in the case of summarisation, Verberne et al. \cite{verberne2018creating} argue that reasonable end-quality can be reached even when the interrater agreement is low. The ground truth is determined by taking the logical disjunction of the annotations.
|
||||
For the first iteration, 1500 sentences were selected for 2 experts to annotate in a binary fashion according to strict guidelines. An example is shown in Figure \ref{fig:annotator}. Afterwards, for measuring the interrater agreement, Cohen's kappa \cite{cohen1960coefficient} is calculated as shown in Equation \ref{equation:kappa}. Which turns out to be \textbf{0.4310} for the two annotators. This happens to be just above the lower end of \textit{moderate agreement}. However, we have to note that the original quality ranges are often criticised for being too relaxed \cite{mchugh2012interrater}. However, in the case of summarisation, Verberne et al. \cite{verberne2018creating} argue that reasonable end-quality can be reached even when the interrater agreement is relatively low. The ground truth is determined by taking the logical disjunction of the annotations.
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.75\linewidth]{figures/annotator.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Annotator UI showing a single sentence and the two possible labels that can be assigned to it based on its relevance to technology transfer.}
|
||||
\label{fig:annotator}
|
||||
\end{figure}
|
||||
|
|
@ -39,40 +40,42 @@ For the first iteration, 1500 sentences were selected for 2 experts to annotate
|
|||
\kappa_{agreement} \equiv \frac{p_{observed} - p_{expected}}{1 - p_{expected}} = 1 - \frac{1 - p_{observed}}{1 - p_{expected}}
|
||||
\end{equation}
|
||||
|
||||
The next step is finetuning SciBERT with the help of HuggingFace transformers \cite{wolf2019huggingface}. The data are divided into training and test split with a ratio of 4:1. From the train split, a validation split is also derived which is used for early stopping. The objective function is the macro-averaged F1-score and the early stopping patience is 5 epochs. The learning rate is $5 \times 10^{-5}$ and AdamW \cite{loshchilov2017decoupled} is used for optimisation with a weight decay of 0.05. The code can be found in the documentation\footnote{\href{https://great-ai.scoutinscience.com/examples/scibert/train/}{great-ai.scoutinscience.com/examples/scibert/train/}}, it is surprisingly slightly shorter than the code of Section \ref{section:simple-case}.
|
||||
The next step is finetuning SciBERT with the help of HuggingFace transformers \cite{wolf2019huggingface}. The data are divided into training and test splits with a ratio of 4:1. From the train split, a validation split is also derived which is used for early stopping. The objective function is the positive class' F1-score and the early stopping patience is 5 epochs. The learning rate is $5 \times 10^{-5}$ and AdamW \cite{loshchilov2017decoupled} is used for optimisation with a weight decay of 0.05. The code can be found in the documentation\footnote{\href{https://great-ai.scoutinscience.com/examples/scibert/train/}{great-ai.scoutinscience.com/examples/scibert/train/}}, it is surprisingly slightly shorter than the code of Section \ref{section:simple-case}.
|
||||
|
||||
\begin{displayquote}
|
||||
\textbf{Reproducability} Reproducible experiments are generally preferred. It is easy to forget to set some seeds values and, for example, end up with different datapoints in the test-train splits. To facilitate reproducability, it would be useful to reset the seeds of each imported library's random number generators (RNG) when GreatAI is configured. Thus, a feature has been added to detect and reset RNG-s of installed and imported libraries. This certainly will not solve the reproducibility crisis \cite{hutson2018artificial} on its own, however, in some cases, it can result in one fewer step to miss.
|
||||
\textbf{Reproducability} Reproducible experiments are generally preferred. It is easy to forget to set some seeds values and, for example, end up with different datapoints in the test-train splits during training and validation in a Continuous Integration (CI) pipeline. To facilitate reproducability, it would be useful to reset the seeds of each imported library's random number generators (RNG) when GreatAI is configured. Thus, a feature has been added to detect and reset RNG-s of installed and imported libraries. This certainly will not solve the reproducibility crisis \cite{hutson2018artificial} on its own, however, in some cases, it can result in one fewer step to miss.
|
||||
\end{displayquote}
|
||||
|
||||
\begin{displayquote}
|
||||
\textbf{Utility of LargeFiles-s} For the purposes of the documentation, the finetuning was conducted in the Google Colab online environment which is excellent for providing anyone with GPU-time for free. However, notebook environments are ephemeral, resulting in the need to manually upload and download all relevant data whenever a new virtual machine (VM) instance is granted. The \texttt{LargeFile} implementation alleviated this problem by handling the uploads and downloads automatically.
|
||||
\textbf{Utility of LargeFiles-s} For the purposes of the documentation, the finetuning was conducted in the Google Colab online environment which is excellent for providing anyone with GPU-time for free. However, notebook environments are ephemeral, resulting in the need to manually upload and download all relevant data whenever a new virtual machine (VM) instance is granted. The \texttt{LargeFile} implementation alleviated this problem by handling the uploads and downloads automatically. Of course, first, backwards compatibility had to be solved for Python 3.7 which is the only available environment in Colab.
|
||||
\end{displayquote}
|
||||
|
||||
The best validation results were achieved after 8 epoch which was slightly more than expected but is presumably due to the weight decay. The confusion matrix on the test split can be seen in Figure \ref{fig:scibert-confusion}: regardless the subjectiveness of the task, SciBERT manages to achieve good quality which is indicated by a macro-averaged F1-score of \textbf{0.89}.
|
||||
The best validation results were achieved after 8 epochs which was slightly more than expected but is presumably due to the weight decay. The confusion matrix on the test split can be seen in Figure \ref{fig:scibert-confusion}: regardless the subjective definition of the task, SciBERT manages to achieve good quality which is indicated by an F1-score of \textbf{0.89}.
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.4\linewidth]{figures/scibert-confusion.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Confusion matrix of the fine-tuned SciBERT model on the \textit{summary candidate sentences} dataset. The values are globally normalised and represent percentages.}
|
||||
\label{fig:scibert-confusion}
|
||||
\end{figure}
|
||||
|
||||
Let us check how well the selected sentences correspond with the tech-transfer potential. Users and in-house experts can rate publications (from a tech-transfer perspective) by assigning them to one of four categories: \texttt{A}, \texttt{B}, \texttt{C}, and \texttt{D} with \texttt{A} being the most and \texttt{D} the least promising. This feedback is stored and used for analytic and training purposes. Since both the feedback grade and the ``highlights'' are supposed to reflect the same aspect of papers, therefore, we can reasonably expect some correlation between them.
|
||||
|
||||
Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as predicted by the finetuned model in 4 categories (grades) of papers. The two datasets come from non-overlapping sets of papers, hence, the results come solely from the model's ability to generalise. It is interesting to see that the Spearman's rank correlation coefficient \cite{spearman1961proof} between the normalised ``highlights'' counts and the ratings of papers is \textbf{0.4584} and is statistically significant ($P = 5.4 \times 10^{-74}$). This proves the presence of a monotonic association. For context, the correlation between the grades and the number of sentences found by the baseline approach is 0.05597 ($P = 0.03$).
|
||||
Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as predicted by the finetuned model in 4 categories (grades) of papers. The two datasets come from non-overlapping sets of papers, hence, the results come solely from the model's ability to generalise. It is interesting to see that the Spearman's rank correlation coefficient \cite{spearman1961proof} between the normalised ``highlights'' counts and the ratings of papers is \textbf{0.4784} and is statistically significant ($P = 5.4 \times 10^{-74}$). This proves the presence of a monotonic association. For context, the correlation between the grades and the number of sentences found by the baseline approach is 0.06597 ($P = 0.03$). We can conclude that the classifier's output is indicative of the publications' tech-transfer potential.
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.85\linewidth]{figures/highlights-histograms.png}
|
||||
\caption{Distribution of mean predicted summary candidate sentence counts in 4 categories. Category \texttt{A} correspond to the most, while \texttt{D} to the least interesting papers based on median user feedback. The sample size is 1406 (\texttt{D}=715, \texttt{C}=309, \texttt{B}=198, \texttt{A}=184). The histograms are on the same scale.}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Distribution of mean predicted summary candidate sentence counts in 4 categories. Category \texttt{A} correspond to the most, while \texttt{D} to the least interesting papers based on mean user feedback. The sample size is 1406 (\texttt{D}=715, \texttt{C}=309, \texttt{B}=198, \texttt{A}=184). The histograms are on the same scale.}
|
||||
\label{fig:histograms}
|
||||
\end{figure}
|
||||
|
||||
We can conclude that the classifier's output is indicative of the publications' tech-transfer potential. To implement the summarisation, at most the top 7 selected sentences are chosen as ranked by their log-probabilities. They are subsequently reordered according to their position in the text. As a quasi-explanation, the tokens' attention scores are visualised and overlaid on the highlighted sentences. The \textit{i}-th token's visualised attention comes from summing up the attention weights of each of the last layer's heads between the \texttt{[CLS]} and the \textit{i}-th token. To improve the end-user experience, a high-pass filter and a stop-word list is applied to the scores in order not to highlight the syntax-related tokens (punctuation, determiners). The service --- after being integrated into the dashboard --- can be seen in Figure \ref{fig:dashboard-highlights}.
|
||||
|
||||
\subsection{Deployment}
|
||||
|
||||
To implement the summarisation, at most the top 7 selected sentences are chosen as ranked by their log-probabilities. They are subsequently reordered according to their position in the text. As a quasi-explanation, the tokens' attention scores are visualised and overlaid on the highlighted sentences. The \textit{i}-th token's visualised attention comes from summing up the attention weights of each of the last layer's heads between the \texttt{[CLS]} and the \textit{i}-th token. To improve the end-user experience, a high-pass filter and a stop-word list is applied to the scores in order not to highlight the syntax-related tokens (punctuation, determiners). The service --- after being integrated into the dashboard --- can be seen in Figure \ref{fig:dashboard-highlights}.
|
||||
|
||||
\begin{displayquote}
|
||||
In order to get insights into their inner workings, HuggingFace models can be given \texttt{output\_attentions=True} in their constructor which results in a new property becoming accessible on the results for querying the attentions. The only issue with it is that it is a 5-dimensional matrix which makes exploring and understanding it non-obvious. In short, it has very low \textit{Discoveribility}. For example, the attentions for the UI are calculated with this expression:
|
||||
\begin{minted}[
|
||||
|
|
@ -85,32 +88,64 @@ Even though the operation is conceptually simple, because of the opaque datastru
|
|||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.9\linewidth]{figures/dashboard-highlights.png}
|
||||
\caption{The tech-transfer summary of an academic publication (\cite{bruns2022deep}). The titles and sentences can be clicked for navigating the paper on the right, while some explanation is provided by the highlighted words the opacity of which corresponds to their attention weights.}
|
||||
\includegraphics[width=1\linewidth]{figures/dashboard-highlights.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{The tech-transfer summary of an academic publication (\cite{bruns2022deep}). The titles and sentences can be clicked for navigating the paper on the right, meanwhile some explanation is provided by the highlighted words, the opacity of which corresponds to their attention weights.}
|
||||
\label{fig:dashboard-highlights}
|
||||
\end{figure}
|
||||
|
||||
\section{Improving GreatAI}
|
||||
|
||||
After having solved two problems by implementing two standalone services and integrating them into an existing ecosystem while relying on GreatAI as a primary tool, a wide variety of insights have been gained. In the next couple of subsections, the extra features and design decisions are presented that have been motivated by the \textit{Highlights service}. After which, the final surface of the API is described and evaluated by its relation to the SE4ML \cite{serban2020adoption,serban2021practices} and AI engineering \cite{john2020architecting,john2020ai} best practices.
|
||||
|
||||
\subsection{Caching}
|
||||
|
||||
Sustainability is an increasingly important concern of ethical AI \cite{van2021sustainable}. Without discussing the pros and cons of the green computing movement \cite{10.1145/1400181.1400186}, we can still agree that computing time should not be wasted. To this end, caching the results of expensive operations has to be considered in an AI deployment. The highlights service is often called multiple times with the same parameters from different other services. With each operation taking up to a couple of seconds, implementing caching can lead to vastly faster response times and an overall more socially conscious deployment.
|
||||
Sustainability is an increasingly crucial concern of ethical AI \cite{van2021sustainable}. Without discussing the pros and cons of the green computing movement \cite{10.1145/1400181.1400186}, we can still agree that computing time should not be wasted. To this end, caching the results of expensive operations has to be considered in any AI deployment. In this case, the \textit{Highlights service} is often called multiple times from different other services with the same parameters. With each operation taking up to a couple of seconds, implementing caching can lead to vastly faster response times and an overall more socially conscious deployment.
|
||||
|
||||
\subsection{Revisiting \texttt{parallel\_map}}
|
||||
|
||||
Even though most inference functions are CPU-bound, turns out, sometimes they involve IO, especially, when relying on the results of other, remote models. This means that a significant performance improvement can be achieved by implementing some inference functions asynchronously \cite{tilkov2010node}. Thus, GreatAI also has to support decorating both regular (synchronous) and asynchronous functions. There is one notable consequence of this: the batch processing feature also has to be compatible with \texttt{async} inference functions. Batch processing is still a useful feature since it is likely that async inference functions are both IO (remote calls) and CPU (local evaluation) heavy at the same time, thus, they can benefit from multi-core parallelisation.
|
||||
Even though most inference functions are CPU-bound, turns out, sometimes they involve IO, especially, when relying on the results of other, remote models. This means that a significant performance improvement can be achieved by implementing some inference functions asynchronously \cite{tilkov2010node}. Thus, GreatAI also has to support decorating both regular (synchronous) and asynchronous functions. There is one notable consequence of this: the batch processing feature also has to be compatible with \texttt{async} inference functions. Batch processing is still a useful feature since it is likely that async inference functions are both IO (remote calls) and CPU (local evaluation) constrained at the same time, thus, they can benefit from multi-core parallelisation.
|
||||
|
||||
However, the standard library's \texttt{multiprocessing}, the third party \texttt{multiprocess} \cite{mckerns2012building}, and, another popular library, \texttt{joblib}\footnote{\href{https://joblib.readthedocs.io/en/latest/}{joblib.readthedocs.io/en/latest}} all lack the support for efficiently parallelising async functions. For this reason, \texttt{parallel\_map} is reimplemented to create an event-loop in each worker process to keep the efficiency of non-blocking IO while also providing parallelisation for the CPU-bound sections of code.
|
||||
|
||||
\subsection{Integration}
|
||||
\subsection{Programmatic integration}
|
||||
|
||||
Apart from supporting \texttt{async} calls, there are a couple of more step that can be taken to help integrating any service with a GreatAI deployment. This is implemented by the \texttt{call\_remote\_great\_ai()} function which hides the networking required to call a GreatAI instance's REST API. It takes care of validation, automatic retries, serialisation, and deserialisation.
|
||||
Apart from supporting \texttt{async} calls, there are a couple of more step that can be taken to help integrating any service with a GreatAI deployment. This is implemented by the \texttt{call\_remote\_great\_ai} function which hides the networking required to call a GreatAI instance's REST API. It takes care of validation, automatic retries, serialisation, and deserialisation. This comes with the added benefit of encouraging decoupled services because the friction of integrating them is no longer noticeable which is beneficial for human collaboration \cite{hasselbring2002component}.
|
||||
|
||||
Additionally, a REST API is generated with its accompanying \href{https://swagger.io/specification/}{OpenAPI schema} and served with a \href{https://swagger.io/}{Swagger} template. It also contains metadata about the function, for instance, its docstring, version, and version of its registered models concatenated in order to be SemVer\footnote{\href{https://semver.org/}{semver.org}} compatible. These can be seen in Figure \ref{fig:greatai-api}. This, combined with a \texttt{/version} endpoint for programmatic access and validation of the service's metadata proved to be key features when integrating the \textit{Highlights service} into ScoutinScience's service-based architecture.
|
||||
Additionally, a REST API is generated with its accompanying OpenAPI schema\footnote{\href{https://swagger.io/specification}{swagger.io/specification}} and served with a \href{https://swagger.io/}{Swagger} template. It also contains metadata about the function, for instance, its docstring, version, and version of its registered models concatenated in order to be SemVer\footnote{\href{https://semver.org/}{semver.org}} compatible. These can be seen in Figure \ref{fig:greatai-api}. This, combined with a \texttt{/version} HTTP endpoint for programmatic access and validation of the service's metadata proved to be key features when integrating the \textit{Highlights service} into ScoutinScience's service-based architecture.
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=0.75\linewidth]{figures/greatai-api.png}
|
||||
\includegraphics[width=0.85\linewidth]{figures/greatai-api.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{Documentation of the automatically scaffolded REST API of a GreatAI service. Notice, how its version string includes its registered models in a SemVer compliant way: \texttt{0.0.1+small-domain-prediction-v11}.}
|
||||
\label{fig:greatai-api}
|
||||
\end{figure}
|
||||
|
||||
\subsection{\textit{Human} integration}
|
||||
|
||||
Even though the REST API of GreatAI services exposes all necessary features\footnote{Such as providing feedback per prediction, complexly filtering and sorting traces, create-read-update-delete (CRUD) operations for the feedback and traces, accessing live monitoring info (current configuration, versions, cache statistics), etc.} which are great for programmatic access, these are not ideal for direct human comprehension. In order to ease the introduction of GreatAI services, a rudimentary dashboard is --- optionally --- generated next to the REST API. The dashboard's main features can be observed in Figures \ref{fig:greatai-header}, \ref{fig:greatai-table}, and \ref{fig:greatai-parallel}. The diagrams and filterable/sortable table are interconnected and are automatically updated, the reactive behaviour is provided by the Dash framework \cite{shammamah_hossain-proc-scipy-2019}.
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\linewidth]{figures/greatai-header.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{The header of the automatically generated dashboard of the service from Section \ref{section:simple-case}. A documentation is generated on the left, while the histogram of response times is shown on the right. The current configuration is prominently displayed on the bottom.}
|
||||
\label{fig:greatai-header}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\textwidth]{figures/greatai-table.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{A dynamically updating, tabular view of traces matching a user-defined filter. Useful for exploring historical predictions or debugging the cause of exceptions (which are also searchable). The filters set in the table affect the other diagrams of the dashboard.}
|
||||
\label{fig:greatai-table}
|
||||
\end{figure}
|
||||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\textwidth]{figures/greatai-parallel.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{A parallel coordinates view of the traces displayed in the table above. Adding new axes is as easy as calling \texttt{log\_metric} inside the inference function.}
|
||||
\label{fig:greatai-parallel}
|
||||
\end{figure}
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
\chapter{Interviews} \label{chapter:interviews}
|
||||
|
||||
\section{Threats to validity}
|
||||
114
docs/thesis/chapters/6_results.tex
Normal file
114
docs/thesis/chapters/6_results.tex
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
\chapter{Results} \label{chapter:interviews}
|
||||
|
||||
It should not be surprising that neither data scientists nor software engineers can be replaced by software libraries. However, a non-negligible subset of their processes can be partially or fully automated, especially when it comes to packaging and deploying AI/ML services. My goal was to design a library with an API that finds the balance between being simple enough to adopt without friction, yet useful/powerful enough to be adopted. Simplicity is subjective and it will be discussed separately in Section \ref{section:interviews}. For now, let us look at the utility of GreatAI.
|
||||
|
||||
For answering \textbf{RQ3} --- To what extent can GreatAI automatically implement AI deployment best practices? --- a comparison is presented in the following that illustrates which best-practices can be implemented/scaffolded/configured with little user input; hence, through a simple and streamlined API. Tables \ref{table:best-practices-1} and \ref{table:best-practices-2} summarise the implemented best practices in the context of practices found by prior surveys of scientific and grey literature \cite{serban2020adoption,serban2021practices,john2020architecting}.
|
||||
|
||||
In order to show an accurately nuanced representation, a \textit{Level of support} is determined for each best practice on a scale of \textit{Fully automated}, \textit{Supported}, and \textit{Partially supported}. For instance, \textit{Use static analysis to check code quality} from Table \ref{table:best-practices-1} is \textit{Supported} because the entire public interface of GreatAI is correctly typed (including generics and asynchronous coroutines) and compatible with \href{https://mypy.readthedocs.io/en/stable/index.html#}{\texttt{mypy}} and \href{https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance}{\texttt{Pylance}}. This means that when GreatAI is used in any Python project, these tools can be applied to statically check the soundness of the projects' integration with GreatAI. However, if the library's user does not use typehints in their code and it contains more complex control flow, it can only be partially typechecked. In short, this best practice is supported, and a considerable part of it is already implemented by GreatAI, but clients should still keep in mind that they might also need to make effort to fully implement it.
|
||||
|
||||
This is not the case for \textit{Log production predictions with the model's version and input data} because by default, it is automatically implemented when calling \texttt{@GreatAI.create}. Users can still specify the exact expected behaviour, e.g.: where to store traces, additional metrics to log, or disabling the logging of sensitive input. Nevertheless, without input from the library's user, the best practice is already reasonably well implemented.
|
||||
|
||||
In Table \ref{table:best-practices-2}, six additional best practices have been added which are generally well-known software engineering considerations that are also applicable to AI/ML deployments. These have not explicitly made it into the aforementioned surveys, however according to the insights gained from Sections \ref{section:simple-case} and \ref{section:complex-case}, implementing them has a positive effect on deployment quality. In future research, attention could be given to their level of industry-wide adoption and quantitative utility.
|
||||
|
||||
\begin{table}
|
||||
\centering
|
||||
\begin{threeparttable}
|
||||
\caption{A subset of AI lifecycle best practices and the level of support GreatAI provides for them. The level of support is one of \textit{Fully automated} (\checkmark\checkmark) which means that no action is required from the user, \textit{Supported} (\checkmark) only automates the reasonably automatable aspects, while \textit{Partially supported} ($\sim$) provides some useful features but the client is expected to build on top of these.}
|
||||
|
||||
\label{table:best-practices-1}
|
||||
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
|
||||
\begin{tabular}{p{7cm}@{\hskip 0.5cm}l@{\hskip 0cm}c} \hline
|
||||
|
||||
\textbf{Best practice} & \textbf{Implementation} & \textbf{Support} \\\hline
|
||||
Use sanity checks for all external data sources\textsuperscript{1} & \texttt{@parameter} & \checkmark \\\hline
|
||||
Check that input data is complete, balanced, and well distributed\textsuperscript{1} & \texttt{@parameter} & $\sim$ \\\hline
|
||||
Write reusable scripts for data cleaning and merging (for NLP)\textsuperscript{1} & \texttt{utilities} & \checkmark\checkmark \\\hline
|
||||
Make data sets available on shared infrastructure\textsuperscript{1} & \texttt{large\_file} & \checkmark\checkmark \\\hline
|
||||
Test all feature extraction code (for NLP)\textsuperscript{1} & \texttt{utilities} & \checkmark\checkmark \\\hline
|
||||
Employ interpretable models when possible\textsuperscript{1} & \texttt{views} & $\sim$ \\\hline
|
||||
Continuously measure model quality and performance\textsuperscript{1, 2} & Feedback API & \checkmark \\\hline
|
||||
Use versioning for data, model, configurations and training scripts\textsuperscript{1, 2} & \texttt{@use\_model}, versioning & \checkmark\checkmark \\\hline
|
||||
Run automated regression tests\textsuperscript{1} & \texttt{*\_ground\_truth} & \checkmark \\\hline
|
||||
Use continuous integration\textsuperscript{1} & Docker Image, WSGI application & \checkmark \\\hline
|
||||
Use static analysis to check code quality\textsuperscript{1} & Fully typed API with generics & \checkmark \\\hline
|
||||
Assure application security\textsuperscript{1} & Code is automatically audited & $\sim$ \\\hline
|
||||
Automate model deployment, enable shadow deployment\textsuperscript{1, 2} & Docker Image \& scripts & \checkmark \\\hline
|
||||
Enable automatic rollbacks for production models\textsuperscript{1, 2} & Docker Image \& scripts & $\sim$ \\\hline
|
||||
Continuously monitor the behaviour of deployed models\textsuperscript{1, 2} & Dashboard, metrics endpoints & \checkmark\checkmark \\\hline
|
||||
Log production predictions with the model's version and input data\textsuperscript{1} & \texttt{@GreatAI.create} & \checkmark\checkmark \\\hline
|
||||
|
||||
\end{tabular}}
|
||||
\begin{tablenotes}
|
||||
\item[1] SE4ML best practices from Table 2 of \cite{serban2020adoption}, and Table 1 of \cite{serban2021practices}.
|
||||
\item[2] Reported state-of-the-art and state-of-practice practices from Tables 2, 3, and 4 of \cite{john2020architecting}.
|
||||
\end{tablenotes}
|
||||
\end{threeparttable}
|
||||
\end{table}
|
||||
|
||||
\begin{table}
|
||||
\centering
|
||||
\begin{threeparttable}
|
||||
\caption{A subset of AI lifecycle best practices and the level of support GreatAI provides for them. The level of support is one of \textit{Fully automated} (\checkmark\checkmark) which means that no action is required from the user, \textit{Supported} ($\checkmark$) only automates the reasonably automatable aspects, while \textit{Partially supported} ($\sim$) provides some useful features but the client is expected to build on top of these.}
|
||||
|
||||
\label{table:best-practices-2}
|
||||
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
|
||||
\begin{tabular}{p{7cm}@{\hskip 0.5cm}l@{\hskip 0cm}c} \hline
|
||||
|
||||
\textbf{Best practice} & \textbf{Implementation} & \textbf{Support} \\\hline
|
||||
Execute validation techniques: error rates and cross-validation\textsuperscript{2} & \texttt{*\_ground\_truth} & \checkmark \\\hline
|
||||
% Track models, dependencies, experiments, versions\textsuperscript{2} & \texttt{great\_ai.use\_model}, Dashboard & \checkmark\checkmark \\\hline
|
||||
Store models in a single format for ease of use\textsuperscript{2} & \texttt{save\_model} & \checkmark\checkmark \\\hline
|
||||
Rewrite from data analysis to industrial development language\textsuperscript{2} & Jupyter Notebook deployment & \checkmark \\\hline
|
||||
Equip with web interface, package image, provide REST API\textsuperscript{2} & \texttt{@GreatAI.create} & \checkmark\checkmark \\\hline
|
||||
Provide simple API for serving batch and realtime requests\textsuperscript{2} & \texttt{@GreatAI.create} & \checkmark\checkmark \\\hline
|
||||
For reproducibility, use standard runtime and configuration files\textsuperscript{2} & \texttt{utilities.ConfigFile}, Dockerfile & \checkmark \\\hline
|
||||
Integration with existing data infrastructure\textsuperscript{2} & GridFS, S3 support & \checkmark\checkmark \\\hline
|
||||
Select ML solution fully integrated with databases\textsuperscript{2} & MongoDB, PostgreSQL support & \checkmark\checkmark \\\hline
|
||||
Querying, visualising and understanding metrics and event logging\textsuperscript{2} & Dashboard, Traces API & \checkmark\checkmark \\\hline
|
||||
% Monitor status and performance\textsuperscript{1, 2} & Dashboard, Status (metadata) API & \checkmark\checkmark \\\hline
|
||||
Measure accuracy of deployed model to ensure data drifts are noticed\textsuperscript{2} & Feedback API & \checkmark \\\hline
|
||||
Apply automation to trigger model retraining\textsuperscript{2} & Feedback API & $\sim$ \\\hline
|
||||
% Employ Agile, DevOps-style workflows, allow automatic rollback\textsuperscript{2} & Docker Image, WSGI application & \checkmark \\\hline
|
||||
% Deploy different versions of same application\textsuperscript{2} & Complex versioning support & $\sim$ \\\hline
|
||||
Allow experimentation with the inference code\textsuperscript{3} & Development mode \& auto-reload & \checkmark\checkmark \\\hline
|
||||
Keep the model's API and documentation together\textsuperscript{3} & Dashboard and Swagger & \checkmark\checkmark \\\hline
|
||||
Parallelise feature extraction\textsuperscript{3} & \texttt{parallel\_map} & \checkmark\checkmark \\\hline
|
||||
Cache predictions\textsuperscript{3} & \texttt{@GreatAI.create} & \checkmark\checkmark \\\hline
|
||||
Async support for top-down chaining models\textsuperscript{3} & All decorators support async & \checkmark\checkmark \\\hline
|
||||
Common schemas for common prediction tasks\textsuperscript{3} & \texttt{views} & \checkmark \\\hline
|
||||
|
||||
\end{tabular}}
|
||||
\begin{tablenotes}
|
||||
\item[2] Reported state-of-the-art and state-of-practice practices from Tables 2, 3, and 4 of \cite{john2020architecting}.
|
||||
\item[3] Additional software engineering best practices applicable to AI/ML deployments encountered while designing and using GreatAI.
|
||||
\end{tablenotes}
|
||||
\end{threeparttable}
|
||||
\end{table}
|
||||
|
||||
Quantifying the number of implemented best practices would be misleading since their scope and importance cover a wide --- sometimes overlapping --- range. Especially because there is some overlap between the different reviews and even within the reviews. However, it is still clear that a large number of best practices can be given a \textit{Fully automated} implementation by GreatAI's design while and even larger number of them can be augmented by the library. This proves the feasibility of designing simple API's using the techniques of Chapter \ref{chapter:design} for decreasing the complexity of correctly deploying AI services (\textbf{RQ2}).
|
||||
|
||||
\section{Interviews} \label{section:interviews}
|
||||
|
||||
Let us finally tackle the question of generalisability.
|
||||
|
||||
One of the takeaways of Chapter \ref{chapter:background} was that Seldon Core is useful for implementing or helping to implement most best practices. Nonetheless, it also has an initial threshold that has to be surmounted before implementing even a single best practice. According to the adoption rate surveys, this stops a large portion of practitioners from using it or other similar frameworks. GreatAI offers a different mix of features, the initial threshold is virtually non-existent: best practices can be immediately applied. But at the same time, the presented solution covers a smaller number of practices. The hypothesis is that the latter approach aligns better with the expectations of professionals.
|
||||
|
||||
\subsection{Threats to validity}
|
||||
|
||||
\section{Future work}
|
||||
|
||||
The primary purpose of GreatAI was to serve as a proxy through which its design decisions could be tested and evaluated in their practical context. For this reason, its design aimed to be a proof-of-principle for validating hypotheses and answering research questions. After successfully doing that, it has been turned into a practical software library suitable for production-use\footnote{\href{https://pypi.org/project/great-ai/}{pypi.org/project/great-ai}}. Although it has already proved its utility, it has also shown that extending its functionality would be worthwhile. Therefore, a number of potential improvements to GreatAI are presented below.
|
||||
|
||||
\subsection{More data science}
|
||||
|
||||
The cases presented in Chapter \ref{chapter:case} revolved around NLP. This, unsurprisingly, heavily influenced the design process. The two most notable effects can be found in the REST API's \texttt{/predict} endpoint and some \texttt{utilities} functions. The former is streamlined to accept JSON compatible data while the latter gives robust feature extraction support for only textual inputs. Supporting the easy, direct upload of larger non-JSON files and extending \texttt{utilities} to handle multimedia formats should be sufficient for widely extending the scope of applicability of GreatAI.
|
||||
|
||||
\subsection{More software engineering}
|
||||
|
||||
In order to greatly simplify its API, each GreatAI Trace is a single document with a well-defined, multi-level schema that clients can also extend by calling \texttt{log\_metric}. MongoDB provides a convenient (and popular) method for persisting such documents, however, if there is some existing database in the environment, storing Traces in that can be favourable. \href{https://www.postgresql.org/}{PostgreSQL} is a popular choice and it also features good JSON document support. Hence, introducing first-class integration for PostgreSQL could benefit some clients.
|
||||
|
||||
As described in Designing Data-intensive Applications \cite{kleppmann2017designing}, services can fall into three broad categories: online systems, batch processing, and stream processing (near-teal-time systems). As of yet, GreatAI only provides streamlined support for the first two. Thus, developer experience could be improved by providing simple, direct integration with popular message queues/protocols, such as \href{https://kafka.apache.org/}{Apache Kafka} \cite{kreps2011kafka}, \href{https://aws.amazon.com/sqs/}{AWS SQS} \cite{garfinkel2007evaluation}, or \href{https://www.amqp.org/}{AMQP} \cite{vinoski2006advanced}.
|
||||
|
||||
Some metrics of GreatAI, such as the cache statistics, versions, and derived data from traces can be already conveniently queried from its REST API. Nevertheless, adding support for the de facto standard metric gathering tool \href{https://prometheus.io/}{Prometheus} could save the library's users from one more integration steps.
|
||||
|
||||
The common theme among the above-mentioned opportunities is that they could be reasonably well implemented without any user input, making them inline with the library's philosophy. Of course, the open-source nature of GreatAI also allows anyone to already provide support for a wide range of integrations.
|
||||
|
|
@ -1,9 +1,9 @@
|
|||
\chapter{Conclusion} \label{chapter:conclusion}
|
||||
|
||||
% even if you already implemented these solutions by hand, you no longer have to -> you have more time -> you can spend that time implementing more advanced best practices
|
||||
todo
|
||||
|
||||
\textit{GreatAI} might have the potential to bridge the gap between data science and software engineering. Stemming from the bidirectional nature of bridges, we can look at the framework from two perspectives: for professionals closer to the field of data science, it provides an automatic scaffolding of software facilities that are required for deploying, monitoring, and iterating on their models. For software engineers, it highlights the necessary steps required for robust and improvable deployments, while at the same time saves them from the menial work of implementing these constructs. While most importantly, it serves as a proxy for the design decisions through which they can be tested and evaluated in their practical context.
|
||||
\textit{GreatAI} may have the potential to bridge the gap between data science and software engineering. Stemming from the bidirectional nature of bridges, we can look at the framework from two perspectives: for professionals closer to the field of data science, it provides an automatic scaffolding of software facilities that are required for deploying, monitoring, and iterating on their models. For software engineers, it highlights the necessary steps required for robust and improvable deployments --- while at the same time --- saves them from the menial work of implementing these constructs manually. While most importantly, it proves that increasing the adoption rate of AI/ML deployment best practices is viable by designing narrower and deeper API-s.
|
||||
|
||||
\section{Future work}
|
||||
Good deployments benefit all of us. Continued research into the means of good deployments remains crucial. However, next to that --- as the presented results show --- better deployments can also be achieved by facilitating the \textit{transition} step of the AI lifecycle. Having automated implementations, even if for just simpler best practices, leaves professionals more time to tackle other deployment challenges and less opportunities to miss crucial steps. Overall, resulting in more implemented practices, hence, robust and trustworthy production software.
|
||||
|
||||
\section{Concluding remarks}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
\documentclass{report}
|
||||
|
||||
\usepackage{placeins}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{pdfpages}
|
||||
\usepackage{hyperref}
|
||||
|
|
@ -11,9 +12,10 @@
|
|||
\usepackage[compact]{titlesec}
|
||||
\usepackage{framed}
|
||||
\usepackage{quoting}
|
||||
\usepackage{caption}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{minted}
|
||||
|
||||
\usepackage{tocloft}
|
||||
\usepackage[a4paper,
|
||||
bindingoffset=0,
|
||||
left=3cm,
|
||||
|
|
@ -72,13 +74,12 @@
|
|||
% Block quote
|
||||
\definecolor{bg}{RGB}{186, 233, 255}
|
||||
\colorlet{shadecolor}{bg}
|
||||
\usepackage{lipsum}
|
||||
\newenvironment{displayquote}
|
||||
{\begin{shaded*}
|
||||
{\begin{samepage}\begin{shaded*}
|
||||
\quoting[leftmargin=0pt, vskip=0pt]
|
||||
}
|
||||
{\endquoting
|
||||
\end{shaded*}
|
||||
}
|
||||
{\endquoting
|
||||
\end{shaded*}\end{samepage}
|
||||
}
|
||||
|
||||
% Section numbering
|
||||
|
|
@ -109,8 +110,9 @@
|
|||
|
||||
\setcounter{secnumdepth}{2}
|
||||
|
||||
% \renewcommand{\cftbeforetoctitleskip}{-0.25in} % position TOC title vertically
|
||||
|
||||
\renewcommand\@pnumwidth{2em}
|
||||
\renewcommand\@tocrmarg{3.5em}
|
||||
\setcounter{tocdepth}{1}
|
||||
\setlength{\parskip}{0.55em}
|
||||
\linespread{1.2}
|
||||
|
|
@ -140,14 +142,18 @@
|
|||
|
||||
\includepdf[pages=-]{frontpage/frontpage.pdf}
|
||||
\include{chapters/0_abstract}
|
||||
|
||||
\tableofcontents
|
||||
|
||||
\chapter*{Acknowledgements}
|
||||
I wish to extend my special thanks to my supervisors, Prof. dr. ir. Joost Visser and Dr. Suzan Verberne, for their invaluable assistance and guidance. I would also like to thank ScoutinScience B.V. for our fruitful collaboration and for allowing me to use their software for experimentation, validation, and demonstration purposes. Last but not least, I would like to individually thank Name, Name, ..., and Name for participating and providing insightful feedback in our interviews.
|
||||
|
||||
\input{chapters/1_introduction}
|
||||
\input{chapters/2_background}
|
||||
\input{chapters/3_methods}
|
||||
\input{chapters/4_design}
|
||||
\input{chapters/5_cases/main}
|
||||
\input{chapters/6_interviews}
|
||||
\input{chapters/6_results}
|
||||
\input{chapters/7_conclusion}
|
||||
|
||||
\input{chapters/appendix}
|
||||
|
|
|
|||
|
|
@ -803,3 +803,48 @@ month = {oct},
|
|||
pages = {11–13},
|
||||
numpages = {3}
|
||||
}
|
||||
|
||||
@incollection{hasselbring2002component,
|
||||
title={Component-based software engineering},
|
||||
author={Hasselbring, Wilhelm},
|
||||
booktitle={Handbook of Software Engineering and Knowledge Engineering: Volume II: Emerging Technologies},
|
||||
pages={289--305},
|
||||
year={2002},
|
||||
publisher={World Scientific}
|
||||
}
|
||||
|
||||
@InProceedings{ shammamah_hossain-proc-scipy-2019,
|
||||
author = { {S}hammamah {H}ossain },
|
||||
title = { {V}isualization of {B}ioinformatics {D}ata with {D}ash {B}io },
|
||||
booktitle = { {P}roceedings of the 18th {P}ython in {S}cience {C}onference },
|
||||
pages = { 126 - 133 },
|
||||
year = { 2019 },
|
||||
editor = { {C}hris {C}alloway and {D}avid {L}ippa and {D}illon {N}iederhut and {D}avid {S}hupe },
|
||||
doi = { 10.25080/Majora-7ddc1dd1-012 }
|
||||
}
|
||||
|
||||
@inproceedings{kreps2011kafka,
|
||||
title={Kafka: A distributed messaging system for log processing},
|
||||
author={Kreps, Jay and Narkhede, Neha and Rao, Jun and others},
|
||||
booktitle={Proceedings of the NetDB},
|
||||
volume={11},
|
||||
pages={1--7},
|
||||
year={2011}
|
||||
}
|
||||
|
||||
@article{garfinkel2007evaluation,
|
||||
title={An evaluation of Amazon's grid computing services: EC2, S3, and SQS},
|
||||
author={Garfinkel, Simson},
|
||||
year={2007}
|
||||
}
|
||||
|
||||
@article{vinoski2006advanced,
|
||||
title={Advanced message queuing protocol},
|
||||
author={Vinoski, Steve},
|
||||
journal={IEEE Internet Computing},
|
||||
volume={10},
|
||||
number={6},
|
||||
pages={87--89},
|
||||
year={2006},
|
||||
publisher={IEEE}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue