50 lines
9.8 KiB
TeX
50 lines
9.8 KiB
TeX
\chapter{Introduction}
|
|
|
|
Artificial intelligence (AI) techniques have recently started enjoying widespread industry awareness and adoption; the use of AI is increasingly prevalent in all sectors \cite{wirtz2019artificial,bosch2021engineering}. The reasons behind this are manifold \cite{jordan2015machine}, to name a few: recent breakthroughs in deep-learning, increased public awareness, abundance of available data, access to powerful low-cost commodity hardware, education, but most interestingly, the rise of high-level libraries making ready-to-use state-of-the-art (SOTA) models easily available. The latter practically abolishes the barrier of entry for applying AI --- and with that --- can help use-cases in many areas.
|
|
|
|
However, the successful integration of AI components into production-ready applications demands strong engineering methods in order to achieve robust deployments \cite{serban2020adoption}. That is why it is as important as ever to also focus on the quality and robustness of deployed models and software. For instance, the lack of a proper overview of the data transformation steps may lead to suboptimal performance and to introducing unintended biases which may contribute to the ever-increasing negative externality of misused AI \cite{o2016weapons}.
|
|
|
|
Concerningly, a peculiar tendency seems to be unfolding: even though industry professionals already have access to numerous frameworks for deploying AI correctly and responsibly, case-studies and developer surveys have found that a considerable fraction of deployments do not follow best practices \cite{serban2020adoption,haakman2021ai,amershi2019software,de2019understanding,sculley2015hidden}. Utilising state-of-the-art machine-learning (ML) models has become reasonably simple; applying them properly is as difficult and nuanced as ever.
|
|
|
|
This thesis sets out to investigate the reasons behind the apparent asymmetry between the adoption of accessible AI libraries and existing reusable solutions for robust AI deployments. It is hypothesised that the primary reason for the underwhelming adoption rate of best practises is the short supply or professionals equally proficient in the domains of both data science and software engineering. Nevertheless, even without their presence, practitioners could rely on frameworks for automated mature deployment processes. However, the barrier of entry for using such existing libraries is too high, especially when compared with the complexity of AI-libraries.
|
|
|
|
Therefore, a software framework, named \textit{GreatAI}, is designed and its design is presented in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to easily facilitate the responsible and robust deployment of algorithms and models in an attempt to overcome the practical drawbacks of other, similar frameworks. Its name stands for its main aim --- namely --- to assist creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
|
|
|
|
The utility of \textit{GreatAI} is validated using the principles of design science methodology \cite{wieringa2014design} through iteratively designing its API and implementation along with the text mining pipeline for a commercial product in collaboration with ScoutinScience B.V. The goal of the aforementioned product is to evaluate technical transfer opportunities in scientific publications. Subsequently, a survey is conducted among practitioners for validating the generalisability of the design.
|
|
|
|
\section{Research questions}
|
|
|
|
I hypothesise that facilitating the adoption of AI deployment best practices is viable by finding less complex framework designs which are easier to adopt in order to decrease the negative externality of misused AI. This paper is set out to investigate this hypothesis by answering the following research questions.
|
|
|
|
\begin{rqlist}
|
|
\item Does the complexity of AI deployment frameworks hinder industrial projects?
|
|
\item What is an effective way of decreasing the complexity of existing frameworks?
|
|
\item Does \textit{GreatAI}'s design improve the efficiency of a team working with AI while also introducing best practices?
|
|
\item Can the design of \textit{GreatAI} decrease the barrier of entry of applying best practices for other teams?
|
|
\end{rqlist}
|
|
|
|
In this case, complexity is used to refer to the difficulty faced by professionals (data scientists and software engineers alike) when integrating libraries with their solutions. This could also be described as the barrier of entry or steepness of the learning curve. If the aforementioned hypothesis is correct, the adoption of best practices can be efficiently increased by decreasing this complexity.
|
|
|
|
AI deployment best practices entail the technical steps that ought to be taken in order to achieve robust, end-to-end, automated, and trustworthy deployments. The exact definitions of these are shown in Section \ref{section:requirements}. The best practices with which the \textit{GreatAI} design is concerned are a subset of those compiled by Serban et al. \cite{serban2020adoption}.
|
|
|
|
The existence question regarding the problem itself (\textbf{RQ1}) is answered by reviewing the literature of the more than 30 published case-studies. \textbf{RQ2} and \textbf{RQ3} are closely connected, the design and evaluation phases utilised to answer them follow an iterative process. They are examined in Chapter \ref{chapter:design} and Chapter \ref{chapter:case} respectively. The final evaluation step is to ascertain the capability of the framework design to generalise beyond a single subdomain and problem context. This question, \textbf{RQ4}, is investigated through interviews with industry professionals in Chapter \ref{chapter:interviews}.
|
|
|
|
\section{Core ideas}
|
|
|
|
Existing frameworks oftentimes suffer from the entanglement of numerous levels of abstractions. Complexity may be effectively reduced by preferring deep and narrow modules \cite{ousterhout2018philosophy}. Instead of exposing each implementation detail and encouraging users to interact with most of them, many of these can be abstracted away. Where configuration may be helpful for advanced users, default values can still be chosen automatically while providing an override option where necessary \cite{ousterhout2018philosophy,hermans2021programmer}.
|
|
|
|
For example, tracing the evaluations and the model versions used in a distributed fashion is very much expected of a trustworthy system. Hence, turning this feature on by default but allowing opting-out from it can result in less scaffolding required from the library's user. It also decreases their up-front cognitive load which by definition flattens the learning-curve \cite{hermans2021programmer}. Similar features can be imagined for providing an access API for the algorithms and for giving feedback, marking outliers, etc.
|
|
|
|
There are best practices which require more complex features, such as using shared infrastructure for storing the models and data \cite{serban2020adoption}. For simplifying this, we can reuse familiar API-s, such as the \texttt{open()} method from Python. A method is proposed which provides the same interface, however, the backing storage for it is a mixture of local disk space and any S3-compatible storage. Various features may be implemented using close to trivial API-s, including support for shadow deployments, automated regression tests, integrated documentation and model cards, etc.
|
|
|
|
Providing the users with only a high-level of abstraction is not unheard of in the domain of practical AI platforms. Many software-as-a-service products offer features for hiding the details of machine learning applications. However --- as we will see in Section \ref{section:existing} --- these tend to abstract away the details of both data science and AI-engineering, overall hindering the development process. The design proposed here aims to simplify only the deployment related concepts.
|
|
|
|
Nonetheless, simple API-s come with a high technical cost. The library has to implement these in a way that still allows high-performance in production \cite{kleppmann2017designing} and avoids being tied to specific libraries or technologies. Inspiration for the latter may be gained from the pipelines of Prado et al. \cite{prado2020bonseyes}: they show that more freedom can be achieved with plug-and-play steps and preconfigured defaults.
|
|
|
|
With these kept in mind, \textit{GreatAI} has the potential to bridge the gap between data science and software engineering. Stemming from the bidirectional nature of bridges, we can look at the framework from two perspectives: for professionals closer to the field of data science, it provides an automatic scaffolding of software facilities that are required for deploying, monitoring, and iterating on their models. For software engineers, it highlights the necessary steps required for robust and improvable deployments, while at the same time saves them from the grunt work of implementing these constructs. While most importantly, it serves as a proxy for the design decisions through which they can be tested and evaluated in their practical context.
|
|
|
|
TODO
|
|
|
|
\section{Structure}
|
|
|
|
The rest of the thesis is organised as follows: Chapter \ref{chapter:background} approaches the problem and the state-of-the-art from three perspectives: the trends of AI library API design, the experiences gained from practical applications, and a comparison of existing deployment options. Next, the methodology utilised for the subsequent chapters is described in Chapter \ref{chapter:methods}. The design cycle is broken into two chapters, Chapter \ref{chapter:design} and \ref{chapter:case}. The former describes the main technological contributions of the novel design, while the latter details the specifics of the practical use-case and the framework's interaction with it. The results are further validated by conducting a survey in Chapter \ref{chapter:survey}. The thesis is concluded in Chapter \ref{chapter:conclusion}.
|