andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Update thesis

Browse source
This commit is contained in:
Andras Schmelczer 2022-06-21 20:02:26 +02:00
parent 0e1f8e6215
commit 73c6a1a0ec
23 changed files with 560 additions and 70 deletions
Download patch file Download diff file Expand all files Collapse all files

36
thesis/chapters/1_introduction.tex
View file

@ -6,11 +6,11 @@ However, the successful integration of AI components into production-ready appli
Concerningly, a peculiar tendency seems to be unfolding: even though industry professionals already have access to numerous frameworks for deploying AI correctly and responsibly, case-studies and developer surveys have found that a considerable fraction of deployments do not follow best practices \cite{serban2020adoption,haakman2021ai,amershi2019software,de2019understanding,sculley2015hidden}. Utilising state-of-the-art machine-learning (ML) models has become reasonably simple; applying them properly is as difficult and nuanced as ever.
This thesis sets out to investigate the reasons behind the apparent asymmetry between the adoption of accessible AI libraries and existing reusable solutions for robust AI deployments. It is hypothesised that the primary reason for the underwhelming adoption rate of best practises is the short supply or professionals equally proficient in the domains of both data science and software engineering. Nevertheless, even without their presence, practitioners could rely on frameworks for automated mature deployment processes. However, the barrier of entry for using such existing libraries is too high, especially when compared with the complexity of AI-libraries.
This thesis sets out to investigate the reasons behind the apparent asymmetry between the adoption of accessible AI libraries and existing reusable solutions for robust AI deployments. It is hypothesised that the primary reason for the underwhelming adoption rate of best practices is the short supply or professionals equally proficient in the domains of both data science and software engineering. Nevertheless, even without their presence, practitioners could rely on frameworks for automated mature deployment processes. However, the barrier of entry for using such existing libraries is too high, especially when compared with the complexity of AI-libraries.
Therefore, a software framework, named \textit{GreatAI}, is designed and its design is presented in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to easily facilitate the responsible and robust deployment of algorithms and models in an attempt to overcome the practical drawbacks of other, similar frameworks. Its name stands for its main aim --- namely --- to assist creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
Therefore, a software framework --- called \textit{GreatAI} --- is designed and its design is presented in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to facilitate the responsible and robust deployment of algorithms and models by designing an accessible API in an attempt to overcome the practical drawbacks of other, similar frameworks. Its name stands for its main aim: to assist easily creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
The utility of \textit{GreatAI} is validated using the principles of design science methodology \cite{wieringa2014design} through iteratively designing its API and implementation along with the text mining pipeline for a commercial product in collaboration with ScoutinScience B.V. The goal of the aforementioned product is to evaluate technical transfer opportunities in scientific publications. Subsequently, a survey is conducted among practitioners for validating the generalisability of the design.
The utility of \textit{GreatAI} is validated using the principles of design science methodology \cite{wieringa2014design} through iteratively designing its API and implementation along with the text mining pipeline for a commercial product in collaboration with ScoutinScience B.V. The goal of the aforementioned software suite is to evaluate technical transfer opportunities in scientific publications. Subsequently, interviews are conducted with practitioners for validating the generalisability of the design.
\section{Research questions}
@ -19,32 +19,38 @@ I hypothesise that facilitating the adoption of AI deployment best practices is
\begin{rqlist}
\item Does the complexity of AI deployment frameworks hinder industrial projects?
\item What is an effective way of decreasing the complexity of existing frameworks?
\item Does \textit{GreatAI}'s design improve the efficiency of a team working with AI while also introducing best practices?
\item Can the design of \textit{GreatAI} decrease the barrier of entry of applying best practices for other teams?
\item Does \textit{GreatAI}'s design improve the efficiency of working with AI while also introducing best practices?
\item Can the design of \textit{GreatAI} decrease the barrier of entry for applying best practices in other contexts?
\end{rqlist}
In this case, complexity is used to refer to the difficulty faced by professionals (data scientists and software engineers alike) when integrating libraries with their solutions. This could also be described as the barrier of entry or steepness of the learning curve. If the aforementioned hypothesis is correct, the adoption of best practices can be efficiently increased by decreasing this complexity.
AI deployment best practices entail the technical steps that ought to be taken in order to achieve robust, end-to-end, automated, and trustworthy deployments. The exact definitions of these are shown in Section \ref{section:requirements}. The best practices with which the \textit{GreatAI} design is concerned are a subset of those compiled by Serban et al. \cite{serban2020adoption}.
AI deployment best practices entail the technical steps that ought to be taken in order to achieve robust, end-to-end, automated, and trustworthy deployments. These are detailed in Section \ref{section:requirements}.
The existence question regarding the problem itself (\textbf{RQ1}) is answered by reviewing the literature of the more than 30 published case-studies. \textbf{RQ2} and \textbf{RQ3} are closely connected, the design and evaluation phases utilised to answer them follow an iterative process. They are examined in Chapter \ref{chapter:design} and Chapter \ref{chapter:case} respectively. The final evaluation step is to ascertain the capability of the framework design to generalise beyond a single subdomain and problem context. This question, \textbf{RQ4}, is investigated through interviews with industry professionals in Chapter \ref{chapter:interviews}.
\section{Core ideas}
\section{Requirements} \label{section:requirements}
Existing frameworks oftentimes suffer from the entanglement of numerous levels of abstractions. Complexity may be effectively reduced by preferring deep and narrow modules \cite{ousterhout2018philosophy}. Instead of exposing each implementation detail and encouraging users to interact with most of them, many of these can be abstracted away. Where configuration may be helpful for advanced users, default values can still be chosen automatically while providing an override option where necessary \cite{ousterhout2018philosophy,hermans2021programmer}.
The best practices (which will be referenced throughout the thesis) with which the \textit{GreatAI} design is concerned are a subset of those compiled by Serban et al. \cite{serban2020adoption}. The core requirements --- sets of covered best practices --- for a software solution that has the potential of improving our problem context are presented in the following along with some explanation and clarification of each of them.
For example, tracing the evaluations and the model versions used in a distributed fashion is very much expected of a trustworthy system. Hence, turning this feature on by default but allowing opting-out from it can result in less scaffolding required from the library's user. It also decreases their up-front cognitive load which by definition flattens the learning-curve \cite{hermans2021programmer}. Similar features can be imagined for providing an access API for the algorithms and for giving feedback, marking outliers, etc.
\paragraph{General} Albeit not explicitly in the list of best practices, compatibility is vital in encouraging adoption. Large projects oftentimes end up depending on numerous packages, each of which may impose some restrictions on the code: since these all have to be satisfied simultaneously, this can result in severe constraints on the application.
There are best practices which require more complex features, such as using shared infrastructure for storing the models and data \cite{serban2020adoption}. For simplifying this, we can reuse familiar API-s, such as the \texttt{open()} method from Python. A method is proposed which provides the same interface, however, the backing storage for it is a mixture of local disk space and any S3-compatible storage. Various features may be implemented using close to trivial API-s, including support for shadow deployments, automated regression tests, integrated documentation and model cards, etc.
The open-source scene of data-related libraries is vibrant. To take the example of data validation, there are at least 4 popular choices which offer varying but similar features: \href{https://github.com/SeldonIO/alibi-detect}{Alibi detect}, \href{https://github.com/PAIR-code/facets}{Facets}, \href{https://github.com/great-expectations/great_expectations}{Great Expectations}, and Data Linter \cite{hynes2017data}. The responsibility of choosing the most fitting solution falls on the user, thus, they should not be limited in this by \textit{GreatAI}.
Providing the users with only a high-level of abstraction is not unheard of in the domain of practical AI platforms. Many software-as-a-service products offer features for hiding the details of machine learning applications. However --- as we will see in Section \ref{section:existing} --- these tend to abstract away the details of both data science and AI-engineering, overall hindering the development process. The design proposed here aims to simplify only the deployment related concepts.
The programming language (PL) of the library should be its only non-general property. Fortunately, the de facto PL for data science is Python, hence, implementing the library in it should not significantly limit its applicability.
Nonetheless, simple API-s come with a high technical cost. The library has to implement these in a way that still allows high-performance in production \cite{kleppmann2017designing} and avoids being tied to specific libraries or technologies. Inspiration for the latter may be gained from the pipelines of Prado et al. \cite{prado2020bonseyes}: they show that more freedom can be achieved with plug-and-play steps and preconfigured defaults.
\paragraph{Robustness} in software development can be achieved by preparing the application to gracefully handle errors, even unexpected ones \cite{bishop1998robust}. Errors can and will happen in practice: storing and investigating what has led to them is required to prevent future ones. In the case of ML, errors might not be as obvious to detect as in more traditional applications (see the above mentioned data validators). Even if a single feature's value falls outside the expected distribution, unexpected results can happen. In cases where this might lead to real-world repercussions, extra care has to be taken to construct as many safe-guards as feasible. \textit{GreatAI} should support its clients in doing so.
With these kept in mind, \textit{GreatAI} has the potential to bridge the gap between data science and software engineering. Stemming from the bidirectional nature of bridges, we can look at the framework from two perspectives: for professionals closer to the field of data science, it provides an automatic scaffolding of software facilities that are required for deploying, monitoring, and iterating on their models. For software engineers, it highlights the necessary steps required for robust and improvable deployments, while at the same time saves them from the grunt work of implementing these constructs. While most importantly, it serves as a proxy for the design decisions through which they can be tested and evaluated in their practical context.
\paragraph{End-to-end} In this case, it refers to end-to-end feedback. That is, feedback should be gathered on the real-world performance of the system, and this should be taken into account when designing/training the next iteration of the model. Static datasets may fail to capture the changing nature of real-life and can become outdated if they are not revised continuously. A well packaged deployment should make it trivial to integrate new training data.
TODO
\paragraph{Automated} The available time of data scientists and software engineers is limited and expensive. For this reason, humans should only be involved when their involvement is necessary. Steps in the development process that can be automated without negative consequences must be automated in order to achieve efficient development processes and let the experts focus on the issues that require their attention the most.
\paragraph{Trustworthy} As detailed by the \textit{Ethics guidelines for trustworthy AI}\footnote{\href{https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}{digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}}, human oversight, transparency, and accountability are some of the key requirements for trustworthy AI applications. For increasing public acceptance and trust while minimising negative societal impact, trustworthiness is essential.
These requirements were chosen stemming from their general importance and potential to be mostly handled (implemented) by a software framework\footnote{The terms \textit{framework} and \textit{library} are used interchangeably in this work stemming from their vague and often holistic differentiation.}. That is why, these provide an ideal initial direction for tackling the issue. Of course, these do not cover all best practices, for instance, the ones relating to organisational processes fall outside the realm of software engineering.
\newpage
\section{Structure}
The rest of the thesis is organised as follows: Chapter \ref{chapter:background} approaches the problem and the state-of-the-art from three perspectives: the trends of AI library API design, the experiences gained from practical applications, and a comparison of existing deployment options. Next, the methodology utilised for the subsequent chapters is described in Chapter \ref{chapter:methods}. The design cycle is broken into two chapters, Chapter \ref{chapter:design} and \ref{chapter:case}. The former describes the main technological contributions of the novel design, while the latter details the specifics of the practical use-case and the framework's interaction with it. The results are further validated by conducting a survey in Chapter \ref{chapter:survey}. The thesis is concluded in Chapter \ref{chapter:conclusion}.
The rest of the thesis is organised as follows: Chapter \ref{chapter:background} approaches the problem and the state-of-the-art from three perspectives: the trends of AI library API design, the experiences gained from practical applications, and a comparison of existing deployment options. Next, the methodology utilised for the subsequent chapters is described in Chapter \ref{chapter:methods}. The design cycle is broken into two chapters, Chapter \ref{chapter:design} and \ref{chapter:case}. The former clarifies the scope and describes the design principles, while the latter details the specifics of the practical use-case and the framework's interaction with it, and technological contributions of the novel design. The results are further validated by conducting interviews with industry professionals in Chapter \ref{chapter:interviews}. The thesis is concluded in Chapter \ref{chapter:conclusion}.