Add thesis source

scope.drawio

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2022-06-17T15:50:41.770Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" version="20.0.1" etag="3pJNM2M7l2PHQ0unmgej" type="device"><diagram id="ccudx8biEBM-x_P_p45R" name="Page-1">1Vpbb+I4FP41SO0Do9xI4ZEFZnalVlOJXe300SQmWE3ijGMo2V+/x4mdOyVAoG0fID62T2yf7zsX04E5C/Y/GIo2T9TF/sDQ3P3AnA8Mw5hYBnwJSZJJdN00M4nHiCtlhWBJ/sNSqEnplrg4rgzklPqcRFWhQ8MQO7wiQ4zRt+qwNfWrb42QhxuCpYP8pvRf4vJNJh0bD4X8T0y8jXqzbk+yngCpwXIn8Qa59K0kMhcDc8Yo5dlTsJ9hX5yeOpds3vcDvfnCGA55lwlyxTFP1N6wC1uVzZCG8PXHhgc+tHR4XNOQS3voFrSb75NLiOmWOVKlNDdHzMNylJWJxMtK0+Qaf2AaYM4SGMCwjzjZVc8dSfN5+bhih/AgN9m+YbmWHfK3UukzwxGjDo5j7ApQIo4ahwJmisSjk/gkdDEzYetvG8LxMkLpLt8A6dWTWtEtjHQfV7kAOa8eE9KfWw5qsJTH8jhH7x3nDjOO9+8eley1FLQkuUzVfiuQqmtStimjVAkvOV77RDzBPlnyCxqaaryIxreRas735c55Um49Y0ZghZhJ4VngNJvgND4KnGYDnAPD9rkADzx4PN1TJhB7rRy0/XtLVccww9QUBuhWtC86lZZ5CnINOb+3JCac0FDphSVmqquvA3FjCSumJHdS3yL0ANdgldC7P8ghsAfyfexTj6EAJkclK1b6SuY9RrY12WMVJ/R+eGRoNR6ZLTwat/Bo3AONxrdwy1YT+aOPQr51K+Qv9pFPGcogD5ayUSCwlH02kZ0PCFex+ApkMrPaEt8FmPdCm6VDcOjgL0+Zkd2BMvaVKKObNwo9da65KN5gVyndE/6rUAOtl1JPoVE0klKjHso6EXjUJPCkbwKnU6eMoaQ0IKIk5HFJ87MQFEAY13IQa6zVTJlpLAybL62TrUcNb/E3QxB5RP6WMjSu87aN6xqkfZzyBCgGqsM1ZoKFadng4q+cAj7UediWAhrXSgF16/MSUbsZEVWp2jMTG1Sr2zpP/5WKbJ1yVrkSrCka6VVFo7pTznbXUHSpbzDsfn2Dbh9PJeYYkoAkJQTwQUAVhS5i7vH4T5mHQhKn+QMSr4WVbagbdwv4J6cwRlsKU1/T3Rzvfkbx/cFc5AOzis6JaXfvpuuWfTTNyL1Q32lGTv9zU/NWL6WeS47vsJc6t2Q+17vpkxb3Zt/GvY1HVW/xUI9Qnd3buKrI1ru5tzM8UMsdUwMzIguIupMhv8pEK6VBaz9vOWtS97KtHGlJAXohyeS4C/4nFpkWEkXY9C/4cFOXHODUZk3PuURoCW99Tr+0W3rbu6fHxdPAmImFvr4Smj1i7ny7byaZX9EB53A97oEVmFryyVYw2X2kk9p7YDr3GgDqhTBeUxakNTxk/5weD/5QL7hbR4T+IcPIFScSY7YjomjoVO8/PWZx+pPgwUj7fX9GfcrSpZj6CunYkONK8nX6J0oVzugrLvVomr2Yfr8UWTdB0qW3apdX+nnoLqL1S7nvotCdh5kjsfs6dwSNiDt8GLWnaaeG7qFeU9SxMjmneGhGrinjZI0cnsbgYEXCzGPwDVaXDUUdIYJYQmUQW+GYDyMGU8FBxAdJf+37hE9CcNlrVk05abmYsK51MZH/5nr6xcSJdO3y49Q5/qMj57Um5/XeSX/g1KFZ/HqdMbD4JwBz8T8=</diagram></mxfile>

thesis/chapters/0_abstract.tex

@@ -0,0 +1,16 @@
+\begin{abstract}
+
+\absdiv{Background}
+Despite its long-standing history, artificial intelligence (AI) has only recently started enjoying widespread industry awareness and adoption; partly thanks to the prevalence of accessible frameworks exposing state-of-the-art models through simple API-s. However, in order to achieve robust deployments, the successful integration of AI components demands strong engineering methods. Concerningly, a tendency seems to be unfolding: even though industry professionals already have access to numerous frameworks for deploying AI correctly and responsibly, case-studies and developer surveys have found that a considerable fraction of deployments do not follow best practices.
+\absdiv{Objective}
+This thesis sets out to investigate the reasons behind the asymmetry between the adoption of accessible AI libraries and preexisting reusable solutions to robust deployments. A software framework called \textit{GreatAI} is designed which aims to facilitate \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated \underline{T}rustworthy AI deployments while attempting to overcome the practical drawbacks of similar, existing frameworks.
+\absdiv{Method}
+The utility of \textit{GreatAI} is validated using the principles of design science methodology through iteratively designing its API and implementation along with the text mining pipeline of a commercial product. Subsequently, interviews are conducted among practitioners for validating the generalisability of the design.
+\absdiv{Results}
+To do.
+\absdiv{Conclusions}
+To do.
+
+\keywords{SE4ML \and AI engineering \and Trustworthy AI \and Deployment \and Text mining}
+
+\end{abstract}

thesis/chapters/1_introduction.tex

@@ -0,0 +1,50 @@
+\chapter{Introduction}
+
+Artificial intelligence (AI) techniques have recently started enjoying widespread industry awareness and adoption; the use of AI is increasingly prevalent in all sectors \cite{wirtz2019artificial,bosch2021engineering}. The reasons behind this are manifold \cite{jordan2015machine}, to name a few: recent breakthroughs in deep-learning, increased public awareness, abundance of available data, access to powerful low-cost commodity hardware, education, but most interestingly, the rise of high-level libraries making ready-to-use state-of-the-art (SOTA) models easily available. The latter practically abolishes the barrier of entry for applying AI --- and with that --- can help use-cases in many areas. 
+
+However, the successful integration of AI components into production-ready applications demands strong engineering methods in order to achieve robust deployments \cite{serban2020adoption}. That is why it is as important as ever to also focus on the quality and robustness of deployed models and software. For instance, the lack of a proper overview of the data transformation steps may lead to suboptimal performance and to introducing unintended biases which may contribute to the ever-increasing negative externality of misused AI \cite{o2016weapons}.
+
+Concerningly, a peculiar tendency seems to be unfolding: even though industry professionals already have access to numerous frameworks for deploying AI correctly and responsibly, case-studies and developer surveys have found that a considerable fraction of deployments do not follow best practices \cite{serban2020adoption,haakman2021ai,amershi2019software,de2019understanding,sculley2015hidden}. Utilising state-of-the-art machine-learning (ML) models has become reasonably simple; applying them properly is as difficult and nuanced as ever. 
+
+This thesis sets out to investigate the reasons behind the apparent asymmetry between the adoption of accessible AI libraries and existing reusable solutions for robust AI deployments. It is hypothesised that the primary reason for the underwhelming adoption rate of best practises is the short supply or professionals equally proficient in the domains of both data science and software engineering. Nevertheless, even without their presence, practitioners could rely on frameworks for automated mature deployment processes. However, the barrier of entry for using such existing libraries is too high, especially when compared with the complexity of AI-libraries.
+
+Therefore, a software framework, named \textit{GreatAI}, is designed and its design is presented in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to easily facilitate the responsible and robust deployment of algorithms and models in an attempt to overcome the practical drawbacks of other, similar frameworks. Its name stands for its main aim --- namely --- to assist creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
+
+The utility of \textit{GreatAI} is validated using the principles of design science methodology \cite{wieringa2014design} through iteratively designing its API and implementation along with the text mining pipeline for a commercial product in collaboration with ScoutinScience B.V. The goal of the aforementioned product is to evaluate technical transfer opportunities in scientific publications. Subsequently, a survey is conducted among practitioners for validating the generalisability of the design.
+
+\section{Research questions}
+
+I hypothesise that facilitating the adoption of AI deployment best practices is viable by finding less complex framework designs which are easier to adopt in order to decrease the negative externality of misused AI. This paper is set out to investigate this hypothesis by answering the following research questions. 
+
+\begin{rqlist}
+  \item Does the complexity of AI deployment frameworks hinder industrial projects?
+  \item What is an effective way of decreasing the complexity of existing frameworks?
+  \item Does \textit{GreatAI}'s design improve the efficiency of a team working with AI while also introducing best practices?
+  \item Can the design of \textit{GreatAI} decrease the barrier of entry of applying best practices for other teams?
+\end{rqlist}
+
+In this case, complexity is used to refer to the difficulty faced by professionals (data scientists and software engineers alike) when integrating libraries with their solutions. This could also be described as the barrier of entry or steepness of the learning curve. If the aforementioned hypothesis is correct, the adoption of best practices can be efficiently increased by decreasing this complexity.
+
+AI deployment best practices entail the technical steps that ought to be taken in order to achieve robust, end-to-end, automated, and trustworthy deployments. The exact definitions of these are shown in Section \ref{section:requirements}. The best practices with which the \textit{GreatAI} design is concerned are a subset of those compiled by Serban et al. \cite{serban2020adoption}.
+
+The existence question regarding the problem itself (\textbf{RQ1}) is answered by reviewing the literature of the more than 30 published case-studies. \textbf{RQ2} and \textbf{RQ3} are closely connected, the design and evaluation phases utilised to answer them follow an iterative process. They are examined in Chapter \ref{chapter:design} and Chapter \ref{chapter:case} respectively. The final evaluation step is to ascertain the capability of the framework design to generalise beyond a single subdomain and problem context. This question, \textbf{RQ4}, is investigated through interviews with industry professionals in Chapter \ref{chapter:interviews}.
+
+\section{Core ideas}
+
+Existing frameworks oftentimes suffer from the entanglement of numerous levels of abstractions. Complexity may be effectively reduced by preferring deep and narrow modules \cite{ousterhout2018philosophy}. Instead of exposing each implementation detail and encouraging users to interact with most of them, many of these can be abstracted away. Where configuration may be helpful for advanced users, default values can still be chosen automatically while providing an override option where necessary \cite{ousterhout2018philosophy,hermans2021programmer}. 
+
+For example, tracing the evaluations and the model versions used in a distributed fashion is very much expected of a trustworthy system. Hence, turning this feature on by default but allowing opting-out from it can result in less scaffolding required from the library's user. It also decreases their up-front cognitive load which by definition flattens the learning-curve \cite{hermans2021programmer}. Similar features can be imagined for providing an access API for the algorithms and for giving feedback, marking outliers, etc.
+
+There are best practices which require more complex features, such as using shared infrastructure for storing the models and data \cite{serban2020adoption}. For simplifying this, we can reuse familiar API-s, such as the \texttt{open()} method from Python. A method is proposed which provides the same interface, however, the backing storage for it is a mixture of local disk space and any S3-compatible storage. Various features may be implemented using close to trivial API-s, including support for shadow deployments, automated regression tests, integrated documentation and model cards, etc.
+
+Providing the users with only a high-level of abstraction is not unheard of in the domain of practical AI platforms. Many software-as-a-service products offer features for hiding the details of machine learning applications. However --- as we will see in Section \ref{section:existing} --- these tend to abstract away the details of both data science and AI-engineering, overall hindering the development process. The design proposed here aims to simplify only the deployment related concepts.
+
+Nonetheless, simple API-s come with a high technical cost. The library has to implement these in a way that still allows high-performance in production \cite{kleppmann2017designing} and avoids being tied to specific libraries or technologies. Inspiration for the latter may be gained from the pipelines of Prado et al. \cite{prado2020bonseyes}: they show that more freedom can be achieved with plug-and-play steps and preconfigured defaults. 
+
+With these kept in mind, \textit{GreatAI} has the potential to bridge the gap between data science and software engineering. Stemming from the bidirectional nature of bridges, we can look at the framework from two perspectives: for professionals closer to the field of data science, it provides an automatic scaffolding of software facilities that are required for deploying, monitoring, and iterating on their models. For software engineers, it highlights the necessary steps required for robust and improvable deployments, while at the same time saves them from the grunt work of implementing these constructs. While most importantly, it serves as a proxy for the design decisions through which they can be tested and evaluated in their practical context.
+
+TODO
+
+\section{Structure}
+
+The rest of the thesis is organised as follows: Chapter \ref{chapter:background} approaches the problem and the state-of-the-art from three perspectives: the trends of AI library API design, the experiences gained from practical applications, and a comparison of existing deployment options. Next, the methodology utilised for the subsequent chapters is described in Chapter \ref{chapter:methods}. The design cycle is broken into two chapters, Chapter  \ref{chapter:design} and \ref{chapter:case}. The former describes the main technological contributions of the novel design, while the latter details the specifics of the practical use-case and the framework's interaction with it. The results are further validated by conducting a survey in Chapter \ref{chapter:survey}. The thesis is concluded in Chapter \ref{chapter:conclusion}.

thesis/chapters/2_background.tex

@@ -0,0 +1,80 @@
+\chapter{Background} \label{chapter:background}
+
+Despite the long-standing history of artificial intelligence (AI), industry awareness and adoption has only recently started to meaningfully catch up \cite{wirtz2019artificial}. At the same time, more regulations and guidelines are being published, for instance, the Ethics guidelines for trustworthy AI by the European Commission's High-Level Expert Group on AI\footnote{\href{https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}{digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}}. This contains seven key requirements, including human agency and oversight, technical robustness, safety, transparency, and accountability. When it comes to accountability, there are meaningful advances being made \cite{raji2020closing}, however, in the case of the other requirements, the situation is more nuanced. Thankfully, the domain of software engineering for machine learning (SE4ML)\footnote{Both in practice and in the literature, this is sometimes also referred to as \textit{AI engineering} and has a large intersection with --- or arguably is the same as --- \textit{MLOps}.} has been working towards finding ways to assist data scientists in ensuring these (and more) expectations are met by their software.
+
+In the following, the context of the problem is presented from three perspectives. Starting with its possible cause: the democratisation of state-of-the-art AI algorithms and models. Subsequently, the challenges encountered when applying AI in practice is outlined by case studies and survey data. Lastly, the existing approaches and solutions are introduced.
+
+\section{Accessible AI}
+
+Most companies prefer not to develop new models but instead reuse prior ones \cite{bosch2021engineering} and they are able to do so increasingly easier. In recent years, there has been a proliferation of highly accessible AI libraries. For example, let us consider the domain of natural language processing (NLP). There are various options for finding AI solutions that work out of the box: FLAIR \cite{akbik2019flair} and Huggingface's transformers \cite{wolf2019huggingface} let developers access the state-of-the-art models and methods in only a couple of lines of code (in many cases 2 or 3). With the advent of fine-tuneable models such as BERT \cite{devlin2018bert} and its many improved variations, Huggingface enables developers to leverage vast amounts of knowledge learned by any particular model and fine-tune it for their specific use case. The API for this is also extremely accessible.
+
+It is not just these two packages, the list of readily available tools for language processing is vast: SpaCy \cite{srinivasa2018natural}, Gensim \cite{vrehuuvrek2011gensim}, and scikit-learn \cite{pedregosa2011scikit} are other great examples. The situation is similar in all subdomains of artificial intelligence: some domain expertise is, admittedly, beneficial but not a hard-requirement. This, combined with the exponentially increasing computing power affordably available to consumers and business alike \cite{sun2019summarizing}, results in AI that is accessible by many.
+
+\section{State of the industry} \label{section:industry}
+
+In contrast with this trend, the software landscape around packaging, deploying, and maintaining machine learning (ML) --- and in general --- data-heavy applications paints a different picture. Fortunately, the related issues and their ramifications have been already thoroughly investigated.
+
+When looking at ML code in practice through the lens of technical debt, Sculley et al. \cite{sculley2015hidden} emphasise the repercussions of writing \textit{glue code} between the algorithms and different systems or libraries and define it as an anti-pattern. The consequence of this is the advice against using generic libraries because their rigid API-s may inhibit improvements, cause lock-in, and result in large amounts of glue code. This is a recurring theme in discussions with industry professionals.
+
+Haakman et al. \cite{haakman2021ai} interviewed 17 people at ING which is a well-known fintech company undergoing a digital transformation to embrace AI. They found that the existing tools for ML do not meet the particularities of the field. For instance, a Feature Engineer working in the Data \& Analytics department explained that regular spreadsheets are preferred over existing solutions like MLFlow for keeping track of experiment results. The reason behind this is simplicity. Additionally, multiple other interviewees described the need to self-develop (or highly-customize) dashboards for monitoring deployed models which results in many non-reusable solutions across the company for the same problem. The authors conclude that there is a research gap between the ever-improving SOTA techniques and the challenges of developing real-world ML systems. In short, additional tool support is needed for facilitating the ML lifecycle.
+
+In a case study at Microsoft, Amershi et al. \cite{amershi2019software} interviewed 14 people and surveyed another 551 AI and ML professionals from the company. One of the main concerns surfaced was relating to automation which is a vital cross-cutting concern, especially for testing. At the same time, a human-in-the-loop is still favoured. The survey data pointed out the difficulty posed by integrating AI, especially in the case of less experienced respondents. This was elaborated on by describing the preferences of software engineers as striving for elegant, abstract, modular, and simple systems; in contrast, data tends to be of large volume, context-specific and heterogeneous. Reconciling these inherent differences requires significant effort, nevertheless, Microsoft manages to overcome this with a highly sophisticated internal infrastructure.
+
+Using AI is not unique to large companies, in a study conducted with the collaboration of three startups \cite{de2019understanding}, the aim was to fill in the gap of understanding how professionals develop ML systems in small companies. Overall, the results showed they have similar priorities to that of large companies, including an emphasis on the online monitoring of deployed models. However, less structure is present in the development lifecycle, as one interviewee had explained: some steps are left out from time to time because they are forgotten about.
+%The paper does not give detail about the use or in-house development of ML tools.
+Similarly, Thiée \cite{thiee2021systematic} describes the slow but ever-growing rate of ML adoption by small and medium-sized enterprises (SMEs). With the caveat that many more of these companies would wish to adopt data-driven approaches but are facing new challenges stemming from the domain's complexity.
+
+Serban et al. \cite{serban2020adoption,serban2021practices} describe the results of their global surveys aiming to ascertain the SOTA in how teams develop, deploy, and maintain ML systems. In \cite{serban2020adoption}, they compiled a set of 29 actionable best practices. These were analysed and validated with a survey of 313 participants to discover the adoption rate and relative importance of each best practice. For example, they determined the most important best practice to be \textit{logging production prediction traces}, however, the adoption was measured to be below 40\%. In more than three quarters of the cases, newcomers to AI reported that they \textit{partially} or \textit{not at all} follow best practices. This tendency decreases with more years of experience, reaching a minimum of just below 40\%. In a similar fashion, Serban et al. in \cite{serban2021practices}, identify another 14 best practices that concern trustworthy AI mainly through data governance. They strive to complement high-level checklists with actionable best practices. Analysing 42 survey responses reveals a familiar pattern. Most best practices have less than 50\% adoption.
+
+Finally, Bosch et al. \cite{bosch2021engineering} organise and structure the problem space of AI engineering research based on their 16 primary case-studies. The authors note the increasing and broad adoption of ML in the industry, while also emphasising that \textit{transition from prototype to production-quality deployment} proves to be challenging for many companies. Large amounts of software engineering expertise is required to create additional facilities for the application such as data pipelines, monitoring, and logging. They define \textit{deployment \& compliance} to be one of the four main categories of problems and describe it as highly underestimated and the source of ample struggle.
+
+\section{Existing solutions} \label{section:existing}
+
+From the previous section, it is noticeable that given enough resources and at the scale of 4195 AI professionals, Microsoft managed to create a comprehensive in-house solution. A similar impression is given by Uber \cite{li2017scaling}; they built a highly sophisticated infrastructure using techniques from distributed and high-performance computing. Though, the authors note that even this solution has its shortcomings in the form rigidity (number of supported libraries and model types) but it still allows the easy extension of the system.
+
+Given the nature of problems faced and amount of available resources, it is not surprising the both of these high-tech, Fortune 500 companies needed to, and did overcome the problems presented by deploying AI. We can learn from their approaches, nonetheless, using them may be infeasible for individuals and SMEs, thus, the issues remain for the majority of practitioners. Luckily, the open-source scene of AI/ML/DS tools, libraries, frameworks, and platforms is thriving. Additionally, there is a considerable number of closed-source --- usually platforms-as-a-service (PaaS) --- solutions next to them. Let us look at some prominent examples.
+
+IBM's AutoAI \cite{wang2020autoai} promises to provide automation for the entire machine learning lifecycle, including deployment. It is a closed-sourced, paid service which --- from their documentation --- seems to focus mostly on non-technical users by providing them with a UI for authoring models. The restrictions caused by the encapsulation of the entire process can be severe. The challenges of integration were emphasised above \cite{sculley2015hidden}. Additionally, an engineer working on Microsoft's comparable solution, the Azure ML Studio, highlighted that once users gain enough understanding of ML, such visual tools can get in their way, and they may need to seek out other solutions \cite{amershi2019software}. Unfortunately, the main value proposition of Azure ML Studio is also to provide a UI for laypeople, and it has also been set to be retired by 2024. Its successor is Azure Machine Learning which shares many similarities with AWS's SageMaker suite \cite{joshi2020amazon}.
+
+SageMaker offers the most comprehensive suite of tools and service; most importantly it has a set of features called \textit{AWS SageMaker MLOps}. This provides easy and/or default implementations for industry best practices described by Serban et al. \cite{serban2020adoption,serban2021practices}. Among others, it promotes the use of CI/CD, model monitoring, tracing, model versioning, storing both data and models on shared infrastructure, numerous collaboration tools, etc. Nonetheless, SageMaker does not enjoy universal adoption as indicated by survey data. The cause of this may be the lack of self-hosting option and its relatively high prices: many companies prefer on-premise hosting for privacy and financial reasons \cite{bosch2021engineering}. Additionally, vendor lock-in, and possibly --- in the case where it is not already used for the project --- the initial effort required for setting up AWS integration could be possible deterrents.
+
+When it comes to open-source libraries, we can find the MLOps libraries of both TensorFlow and PyTorch: TensorFlow Extended (TFX) \cite{baylor2017tfx} and TorchX\footnote{\href{https://pytorch.org/torchx/latest/}{pytorch.org/torchx/latest}}. TFX comes with a more mature set of features with the caveat that initial time-investment is needed for their setup. The features of TorchX only concern the distributed deployment to a wide range of providers, including Kubernetes, AWS Batch, or Ray. There is no augmentation for the SE4ML best practices. Given the tight coupling between these libraries and their corresponding ML frameworks, they cannot generalise to models\footnote{The Open Neural Network Exchange (\href{https://onnx.ai/}{onnx.ai}) format could be an option for overcoming these incompatibilities, however, a more universal support is needed for seamless integration.} or algorithms of other frameworks and technologies. 
+
+Open-source platforms also exist such as MLflow and Seldon Core. They both rely on Kubernetes (k8s) to provide their features. MLflow puts more emphasis on the training phase (in deployment, it lacks a feedback loop which is essential to reach many of the best-practices), while Seldon Core focuses on the deployment stage. The latter comes integrated with a powerful explanation engine, Alibi Explain \cite{klaise2021alibi}. It also boasts the most comprehensive suite of features including outlier detection, online model selection (with multi-armed bandit theory), and distributed tracing. In short, it seems to be the ideal candidate for the title of \textit{framework for robust end-to-end AI deployments}. Its only downside is the amount of complexity propagated to its clients: it is built on top of Kubernetes, and relies on Helm, Ambasador/Istio, Prometheus, and Jaeger for its features. Hence, the first step in using it setting up a k8s cluster with all the required components, then when it comes to model deployment, a Kubernetes configuration file has to be created to make use of Seldon's Custom Resource Definition. These are smaller obstacles if the project is already built on top of k8s; however, even then, software engineers with strong cloud and DevOps background are actively required for using Seldon Core.
+
+\begin{table}
+\centering
+\begin{threeparttable}
+\caption{High-level comparison of popular AI deployment platforms and libraries.}
+\label{table:platform-comparison}
+\setlength{\tabcolsep}{0.25em} % for the horizontal padding
+{\renewcommand{\arraystretch}{1.2} % for the vertical padding
+\begin{tabular}{|l|c|c|c|c|c|c|c|}
+\hline
+                             &  AutoAI    & Azure ML   & SageMaker  & TFX        & TorchX     & MLflow     & Seldon Core \\ \hline
+Open-source                  &            &            &            & \checkmark & \checkmark & \checkmark & \checkmark  \\ \hline
+Free                         &            &            &            & \checkmark & \checkmark & \checkmark & \checkmark  \\ \hline
+Vendor-agnostic              &            &            &            & \checkmark & \checkmark & \checkmark & \checkmark  \\ \hline
+AI-agnostic                  &            & \checkmark & \checkmark &            &            & \checkmark & \checkmark  \\ \hline
+E2E feedback                 &            & \checkmark & \checkmark &            &            &            & \checkmark  \\ \hline
+Distributed monitoring       &            & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\textsuperscript{*} & \checkmark  \\ \hline
+Online model selection       & \checkmark & \checkmark & \checkmark &            &            &            & \checkmark  \\ \hline
+Versioning                   & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark  \\ \hline
+Quick setup                  & \checkmark & \checkmark &            &            &            &            &             \\ \hline
+No dependencies (k8s, cloud) &            &            &            &            & \checkmark &            &             \\ \hline
+\end{tabular}}
+\begin{tablenotes}
+\item[*] Only partial support.
+\end{tablenotes}
+\end{threeparttable}
+\end{table}
+
+Table \ref{table:platform-comparison} shows a high-level overview about the general properties, and some of the features relating to the \textit{Deployment} stage of the CRISP-DM model \cite{wirth2000crisp}. It makes it apparent that there is a coexistence of persisting problems and their promised solutions.
+
+Additionally, increasing attention is given to ML deployments in embedded systems both from a theoretical \cite{john2020ai} and practical \cite{prado2020bonseyes} point of view. Prado et al. \cite{prado2020bonseyes} survey the available deployment frameworks and end-to-end solutions including those for embedded devices. They note their inefficiencies that come from the lack of features and too much rigidity. They introduce their framework for embedded AI deployments which can be used out-of-the box but also lets the users easily replace and extend its pipeline with steps to fit their changing needs and advancements of the field. While Meenu et al. \cite{john2020ai} present and compare different architectural choices for large-scale deployments in edge-computing. They also note that: \textit{"...there is a need to consider and adapt well established SE practices which have been ignored or had a very narrow focus in ML literature"}.
+
+\section{Summary}
+
+The surveys and case-studies have shown the industry's continuous struggle to evolve their prototypes into robust and responsible production-ready deployments. Simultaneously, platforms aiming to help overcome this challenge already exist but lack widespread adoption. The frequently recurring explanations for not adopting pre-existing solutions surfaced in Section \ref{section:industry} revolve around their complexity and rigidity. These complaints are validated when looking at the available frameworks in Section \ref{section:existing}. While using AI has become more accessible than ever, deploying remains challenging owing to the lack of any \textit{easy-to-use framework for robust end-to-end AI deployments}.
+
+The coexistence of multiple major obstacles along with their promised solutions and the lack of their wide-spread adoption leads us to believe that current frameworks are inadequate for many contexts. There is an unmet need for accessible AI deployment methods. The revolution brought by FLAIR, HuggingFace, and similar libraries for the domain of ML remains unmatched in the domain of AI engineering.

thesis/chapters/3_methods.tex

@@ -0,0 +1,25 @@
+\chapter{Methods} \label{chapter:methods}
+
+The chosen methodology for this study is Design Science which emphasises the need to design and investigate artifacts in their context \cite{wieringa2014design}. It consists of a design and an empirical cycle. The purpose of the former is to improve a problem context with a new or redesigned artifact. While in the latter, the problem is investigated and its treatment is validated concurrently. The design cycle shares many similarities with Action Research \cite{davison2004principles}, where the researchers attempt to solve a real-world problem while simultaneously studying the experience of solving said problem. 
+
+As for the empirical cycle, the pragmatist approach is taken since the value of this research lies in its utility. Moreover, pragmatism adopts an engineering approach to research \cite{shull2007guide} which is inline with the philosophy of design science. Additionally, as no research method is without flaws, it is imperative to try to compensate their weaknesses by applying multiple methods. Hence, the study also relies on interviews with potential professionals for validating the design decisions of \textit{GreatAI}.
+
+\section{Problem context}
+
+The problem context is the difficulty in responsibly transitioning (while following best practices) from prototype industrial AI applications to production-ready deployments. With the possible treatment being libraries with high-level API-s and a set of default settings. It is important to note that \textit{GreatAI} is merely a proof-of-concept\footnote{As of now.}, and its aim is to serve as the proxy for the design decisions behind it. Through this, the design can be indirectly evaluated. Hopefully, a by-product will be a library that can be successfully applied to this problem context.
+
+The practical cases used for the evaluation are further elaborated in Chapter \ref{chapter:case}. In short, they focus on individual components of a growing commercial platform with the aim to find tech-transfer opportunities in academic publications. The main input of the system as a whole are individual PDF-files while the output is a list of metrics describing various aspects of the paper, such as interesting sentences, scientific domains, and the scientific contribution. The output also includes a predicted score used for ranking. This ranking is subsequently used by the business developers of Technology Transfer Offices (TTO-s) of multiple Dutch and German universities who later give feedback on the results.
+
+Overall, this problem context carries the properties of typical industry use-cases: it utilises a wide-range of text mining methods, contains complex interactions between the services, benefits from the integration of end-to-end feedback, and has to provide the clients with a platform that they can rely on in their organisation's core processes. Since the final ranking affects real people, explainability and robustness are also central questions.
+
+\section{Design \& empirical cycles}
+
+The aim of the project can be summarised using the terminology of design science in the following way: 
+\textit{Facilitate the easy adoption of AI deployment best practices
+by finding a less complex framework design 
+which is easier to adopt
+to decrease the negative externality of misused AI.}
+
+However, before generalising, the design of the framework is iteratively refined using the feedback acquired from applying it in a practical context which in this case is the development of smaller and the rewriting of larger AI-based applications. The treatment is finding a simpler design which still leads to high-quality deployments. \textbf{RQ2} and \textbf{RQ3} captures this process; for investigating the feedback acquired from iteratively working --- which is the definition of action research --- on the case will be valuable. 
+
+To answer how well the design of \textit{GreatAI} can generalise (\textbf{RQ4}), interviews will be conducted from a population of software engineers and data scientists with varying levels of professional background. Since me and my colleagues are likely to have a bias for (or against) the proposed designs, the first step of checking its applicability in other practical contexts is to ask the opinion of non-affiliated fellow practitioners.

thesis/chapters/4_design.tex

@@ -0,0 +1,47 @@
+\chapter{Designing the framework} \label{chapter:design}
+
+\footnote{Of course, given the saturation of MLOps libraries, extra care has to be taken to meaningfully extend the current state-of-the-art. \href{https://xkcd.com/927/}{xkcd.com/927}}
+
+\section{Requirements} \label{section:requirements}
+
+\paragraph{General}
+\paragraph{Robust}
+\paragraph{End-to-end}
+\paragraph{Automated}
+\paragraph{Trustworthy}
+
+\subsection{Out of scope}
+% % just for data there are so many options, let the users choose
+% % https://github.com/SeldonIO/alibi-detect: for "both" tensorflwo and torch drift detection
+
+% % https://docs.seldon.io/projects/alibi-detect/en/stable/od/methods/llr.html
+% % https://github.com/PAIR-code/facets
+% % https://github.com/great-expectations/great_expectations
+% % The data linter: Lightweight, automated sanity checking for ml data sets. 
+
+% % Git Large File System (LFS: bad
+% % DVC - one more tool
+
+% The obstacles coming from the intertwined nature of different models is widely recognised \cite{sculley2015hidden,haakman2021ai,amershi2019software}. This can lead to non-monotonic error propagation, meaning that improvements in one part of the system might decrease the overall system quality \cite{amershi2019software}. The importance of schema versioning in an environment of rapidly changing models and transformations is highlighted and solved for a specific use-case in \cite{van2017versioning}.
+
+% The main requirements I have for it is to be library-agnostic, a PyTorch-based deep-learning module or a call to a web API should be handled the same way. Everything is included in a single python package. It's not a PaaS. It's a framework encapsulating the "AI" part of a service and turning it into a more classical piece of software that can be handled with well-known, proven technologies.
+
+% The algorithm is exposed through a web API, predictions are followed from beginning to end, the traces are persisted (a db driver can be given, it can be local filesystem, s3, sql, nosql, hfs, etc.). A/B testing, shadow deployments, automatic versioning are supported by default. Input data statistics can be generated and exposed through a REST API (maybe a minimal frontend can also be part of the package). Using the feedback, the performance of the components can be observed real-time the same way.
+
+% Basically, you define a pipeline in code with one or more of your algorithms. Visualisation and persistence steps can be also used. Each model must have a version, but defining A/B or shadow deployments works the same way. The models are uploaded to somewhere (S3 for example) from your machine, and when the service is deployed (though a pre-existing CI/CD system even). It can handle the training data as well, using it, regression tests can be run.
+
+% I don't see too much reason to help clients with the training part, there are already a plethora of solutions for that, including some of the aforementioned ones. But it could have some integration with Ray Tune just to cover everything.
+
+% In short, I believe that compliance with many of the best practices isn't too difficult once the developers and managers are aware of them and have the appropriate tooling to conduct the necessary steps. I would like to provide them with this tooling.
+
+
+
+\section{Design principles}
+
+
+For principles of API design:
+A Philosophy of Software Design \cite{ousterhout2018philosophy}
+The Programmers Brain \cite{hermans2021programmer}
+“12. Designing and improving larger systems”
+“ We call this version cognitive dimensions of code bases (CDCB) and use it to examine a codebase to understand how it”
+\cite{kleppmann2017designing}

thesis/chapters/5_case.tex

@@ -0,0 +1,2 @@
+\chapter{The ScoutinScience platform} \label{chapter:case}
+\cite{jurafsky2019speech}

thesis/chapters/6_survey.tex

@@ -0,0 +1,3 @@
+\chapter{Interviews} \label{chapter:interviews}
+
+\section{Threats to validity}

thesis/chapters/7_conclusion.tex

@@ -0,0 +1,8 @@
+\chapter{Conclusion}  \label{chapter:conclusion}
+
+
+\section{Future work}
+
+\section{Concluding remarks}
+
+% I feel I could relate to many of them, either because I had faced them or I'm still facing the issues described in them. They inspired me to help in facilitating the general adoption of best practices.

thesis/frontpage/frontpage.tex

@@ -0,0 +1,85 @@
+\documentclass[12pt]{article}
+\usepackage{graphicx}
+\usepackage[ddmmyyyy]{datetime}
+
+
+\setlength{\textheight}{24.7cm}
+\setlength{\textwidth}{16cm}
+\setlength{\unitlength}{1mm}
+\setlength{\topskip}{1truecm}
+\topmargin 280mm \advance \topmargin -\textheight
+\divide \topmargin by 2 \advance \topmargin -1in
+\headheight 0pt \headsep 0pt
+\leftmargin 210mm \advance \leftmargin -\textwidth
+\divide \leftmargin by 2 \advance \leftmargin -1in
+\oddsidemargin \leftmargin \evensidemargin \leftmargin
+\parindent=0pt
+\frenchspacing
+
+\newcommand{\bree}[1]{\makebox[4.1cm][l]{#1:}}
+
+\begin{document}
+
+%No page number
+\thispagestyle{empty}
+\sf
+
+\begin{tabular}[t]{p{3.5cm}@{\hspace{4mm}\vrule width 1.5pt\hspace{4mm}}l}
+%Logo Leiden University
+\makebox(20,0)[t]{\includegraphics{../figures/UL_PMS-kleur.eps}}
+&
+\begin{minipage}[t]{12.25cm}
+\begin{Huge}
+\vspace*{0.4cm}
+\textbf{}
+\\[2ex]
+\textbf{Master Computer Science}
+\end{Huge}
+
+\vspace*{4cm}
+
+\begin{Large}
+\hfill GreatAI: An easy-to-use framework for robust end-to-end AI deployments
+
+\vspace*{3mm}
+
+\hfill
+
+\vspace*{4.5cm}
+
+\bree{Name}%
+András Schmelczer
+\\
+\bree{Student ID}%
+s3052249
+\\[1ex]
+\bree{Date}%
+\today
+\\[1ex]
+\bree{Specialisation}%
+Advanced Computing and Systems
+\\[1ex]
+\bree{1st supervisor}%
+Prof. dr. ir. Joost Visser
+\\ 
+\bree{2nd supervisor}%
+Dr. Suzan Verberne
+\end{Large}
+
+\begin{large}
+\vspace*{2.5cm}
+Master's Thesis in Computer Science
+
+\vspace*{5mm}
+Leiden Institute of Advanced Computer Science (LIACS)\\
+Leiden University\\
+Niels Bohrweg 1\\
+2333 CA Leiden\\
+The Netherlands
+\end{large}
+
+
+\end{minipage}
+\end{tabular}
+
+\end{document}

thesis/main.tex

@@ -0,0 +1,76 @@
+\documentclass[runningheads]{llncs}
+
+\usepackage{graphicx}
+\usepackage{pdfpages}
+\usepackage{hyperref}
+\usepackage{amssymb}
+\usepackage{fancyhdr}
+\usepackage{enumitem}
+\usepackage{threeparttable}
+\usepackage{multicol}
+
+% Header & footer
+\pagestyle{fancy}
+\fancyhf{}
+\renewcommand{\headrulewidth}{0pt}
+\fancyfoot[C]{\thepage}
+
+\makeatletter
+\renewenvironment{thebibliography}[1]
+     {\begin{multicols}{2}[\section*{\refname}]%
+      \@mkboth{\MakeUppercase\refname}{\MakeUppercase\refname}%
+      \small
+      \list{\@biblabel{\@arabic\c@enumiv}}%
+          {\settowidth\labelwidth{\@biblabel{#1}}%
+            \leftmargin\labelwidth
+            \advance\leftmargin\labelsep
+            \@openbib@code
+            \usecounter{enumiv}%
+            \let\p@enumiv\@empty
+            \renewcommand\theenumiv{\@arabic\c@enumiv}}%
+      \sloppy
+      \clubpenalty4000
+      \@clubpenalty \clubpenalty
+      \widowpenalty4000%
+      \sfcode`\.\@m}
+     {\def\@noitemerr
+      {\@latex@warning{Empty `thebibliography' environment}}%
+      \endlist\end{multicols}}
+\makeatother
+
+% Make links blue
+\renewcommand\UrlFont{\color{blue}\rmfamily}
+
+% Structured abstract headings
+\newcommand{\absdiv}[1]{
+  \par
+  \noindent\textbf{#1:}
+}
+
+% Research questions list
+\newlist{rqlist}{enumerate}{1}
+\setlist[rqlist, 1]{
+  label=\textbf{RQ\arabic{rqlisti}.},
+  leftmargin=2cm,
+  rightmargin=1.25cm
+}
+
+\begin{document}
+
+\includepdf[pages=-]{frontpage/frontpage.pdf}
+\include{chapters/0_abstract}
+\tableofcontents
+
+\input{chapters/1_introduction}
+\input{chapters/2_background}
+\input{chapters/3_methods}
+\input{chapters/4_design}
+\input{chapters/5_case}
+\input{chapters/6_survey}
+\input{chapters/7_conclusion}
+
+\clearpage
+\bibliographystyle{splncs04}
+\bibliography{ref}
+
+\end{document}

thesis/ref.bib

@@ -0,0 +1,308 @@
+
+@article{haakman2021ai,
+  title={AI lifecycle models need to be revised},
+  author={Haakman, Mark and Cruz, Lu{\'\i}s and Huijgens, Hennie and van Deursen, Arie},
+  journal={Empirical Software Engineering},
+  volume={26},
+  number={5},
+  pages={1--29},
+  year={2021},
+  publisher={Springer}
+}
+
+@inproceedings{amershi2019software,
+  title={Software engineering for machine learning: A case study},
+  author={Amershi, Saleema and Begel, Andrew and Bird, Christian and DeLine, Robert and Gall, Harald and Kamar, Ece and Nagappan, Nachiappan and Nushi, Besmira and Zimmermann, Thomas},
+  booktitle={2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)},
+  pages={291--300},
+  year={2019},
+  organization={IEEE}
+}
+
+@book{wieringa2014design,
+  title={Design science methodology for information systems and software engineering},
+  author={Wieringa, Roel J},
+  year={2014},
+  publisher={Springer}
+}
+
+@inproceedings{serban2020adoption,
+  title={Adoption and effects of software engineering best practices in machine learning},
+  author={Serban, Alex and van der Blom, Koen and Hoos, Holger and Visser, Joost},
+  booktitle={Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)},
+  pages={1--12},
+  year={2020}
+}
+
+@inproceedings{de2019understanding,
+  title={Understanding development process of machine learning systems: Challenges and solutions},
+  author={de Souza Nascimento, Elizamary and Ahmed, Iftekhar and Oliveira, Edson and Palheta, M{\'a}rcio Piedade and Steinmacher, Igor and Conte, Tayana},
+  booktitle={2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)},
+  pages={1--6},
+  year={2019},
+  organization={IEEE}
+}
+
+@inproceedings{serban2021practices,
+  title={Practices for engineering trustworthy machine learning applications},
+  author={Serban, Alex and van der Blom, Koen and Hoos, Holger and Visser, Joost},
+  booktitle={2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN)},
+  pages={97--100},
+  year={2021},
+  organization={IEEE}
+}
+
+@article{sculley2015hidden,
+  title={Hidden technical debt in machine learning systems},
+  author={Sculley, David and Holt, Gary and Golovin, Daniel and Davydov, Eugene and Phillips, Todd and Ebner, Dietmar and Chaudhary, Vinay and Young, Michael and Crespo, Jean-Francois and Dennison, Dan},
+  journal={Advances in neural information processing systems},
+  volume={28},
+  year={2015}
+}
+
+@book{ousterhout2018philosophy,
+  title={A philosophy of software design},
+  author={Ousterhout, John K},
+  volume={98},
+  year={2018},
+  publisher={Yaknyam Press Palo Alto}
+}
+
+@book{hermans2021programmer,
+  title={The Programmer's Brain: What every programmer needs to know about cognition},
+  author={Hermans, Felienne},
+  year={2021},
+  publisher={Simon and Schuster}
+}
+
+@inproceedings{van2017versioning,
+  title={Versioning for end-to-end machine learning pipelines},
+  author={Van Der Weide, Tom and Papadopoulos, Dimitris and Smirnov, Oleg and Zielinski, Michal and Van Kasteren, Tim},
+  booktitle={Proceedings of the 1st Workshop on Data Management for End-to-End Machine Learning},
+  pages={1--9},
+  year={2017}
+}
+
+@inproceedings{li2017scaling,
+  title={Scaling machine learning as a service},
+  author={Li, Li Erran and Chen, Eric and Hermann, Jeremy and Zhang, Pusheng and Wang, Luming},
+  booktitle={International Conference on Predictive Applications and APIs},
+  pages={14--29},
+  year={2017},
+  organization={PMLR}
+}
+
+@inproceedings{wirth2000crisp,
+  title={CRISP-DM: Towards a standard process model for data mining},
+  author={Wirth, R{\"u}diger and Hipp, Jochen},
+  booktitle={Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining},
+  volume={1},
+  pages={29--40},
+  year={2000},
+  organization={Manchester}
+}
+
+@article{wirtz2019artificial,
+  title={Artificial intelligence and the public sector—applications and challenges},
+  author={Wirtz, Bernd W and Weyerer, Jan C and Geyer, Carolin},
+  journal={International Journal of Public Administration},
+  volume={42},
+  number={7},
+  pages={596--615},
+  year={2019},
+  publisher={Taylor \& Francis}
+}
+
+@inproceedings{akbik2019flair,
+  title={FLAIR: An easy-to-use framework for state-of-the-art NLP},
+  author={Akbik, Alan and Bergmann, Tanja and Blythe, Duncan and Rasul, Kashif and Schweter, Stefan and Vollgraf, Roland},
+  booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)},
+  pages={54--59},
+  year={2019}
+}
+
+@article{wolf2019huggingface,
+  title={Huggingface's transformers: State-of-the-art natural language processing},
+  author={Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Pierric and Rault, Tim and Louf, R{\'e}mi and Funtowicz, Morgan and others},
+  journal={arXiv preprint arXiv:1910.03771},
+  year={2019}
+}
+
+@article{devlin2018bert,
+  title={Bert: Pre-training of deep bidirectional transformers for language understanding},
+  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
+  journal={arXiv preprint arXiv:1810.04805},
+  year={2018}
+}
+
+@book{srinivasa2018natural,
+  title={Natural Language Processing and Computational Linguistics: A practical guide to text analysis with Python, Gensim, spaCy, and Keras},
+  author={Srinivasa-Desikan, Bhargav},
+  year={2018},
+  publisher={Packt Publishing Ltd}
+}
+
+@article{vrehuuvrek2011gensim,
+  title={Gensim—statistical semantics in python},
+  author={Rehrek, Radim and Sojka, Petr and others},
+  journal={Retrieved from genism. org},
+  year={2011}
+}
+
+@article{pedregosa2011scikit,
+  title={Scikit-learn: Machine learning in Python},
+  author={Pedregosa, Fabian and Varoquaux, Ga{\"e}l and Gramfort, Alexandre and Michel, Vincent and Thirion, Bertrand and Grisel, Olivier and Blondel, Mathieu and Prettenhofer, Peter and Weiss, Ron and Dubourg, Vincent and others},
+  journal={the Journal of machine Learning research},
+  volume={12},
+  pages={2825--2830},
+  year={2011},
+  publisher={JMLR. org}
+}
+
+@article{sun2019summarizing,
+  title={Summarizing CPU and GPU design trends with product data},
+  author={Sun, Yifan and Agostini, Nicolas Bohm and Dong, Shi and Kaeli, David},
+  journal={arXiv preprint arXiv:1911.11313},
+  year={2019}
+}
+
+@article{jordan2015machine,
+  title={Machine learning: Trends, perspectives, and prospects},
+  author={Jordan, Michael I and Mitchell, Tom M},
+  journal={Science},
+  volume={349},
+  number={6245},
+  pages={255--260},
+  year={2015},
+  publisher={American Association for the Advancement of Science}
+}
+
+@book{o2016weapons,
+  title={Weapons of math destruction: How big data increases inequality and threatens democracy},
+  author={O'neil, Cathy},
+  year={2016},
+  publisher={Broadway Books}
+}
+
+@inproceedings{wang2020autoai,
+  title={Autoai: Automating the end-to-end ai lifecycle with humans-in-the-loop},
+  author={Wang, Dakuo and Ram, Parikshit and Weidele, Daniel Karl I and Liu, Sijia and Muller, Michael and Weisz, Justin D and Valente, Abel and Chaudhary, Arunima and Torres, Dustin and Samulowitz, Horst and others},
+  booktitle={Proceedings of the 25th International Conference on Intelligent User Interfaces Companion},
+  pages={77--78},
+  year={2020}
+}
+
+@incollection{joshi2020amazon,
+  title={Amazon’s machine learning toolkit: Sagemaker},
+  author={Joshi, Ameet V},
+  booktitle={Machine Learning and Artificial Intelligence},
+  pages={233--243},
+  year={2020},
+  publisher={Springer}
+}
+
+@inproceedings{baylor2017tfx,
+  title={Tfx: A tensorflow-based production-scale machine learning platform},
+  author={Baylor, Denis and Breck, Eric and Cheng, Heng-Tze and Fiedel, Noah and Foo, Chuan Yu and Haque, Zakaria and Haykal, Salem and Ispir, Mustafa and Jain, Vihan and Koc, Levent and others},
+  booktitle={Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
+  pages={1387--1395},
+  year={2017}
+}
+
+@article{thiee2021systematic,
+  title={A systematic literature review of machine learning canvases},
+  author={Thi{\'e}e, Lukas-Walter},
+  journal={INFORMATIK 2021},
+  year={2021},
+  publisher={Gesellschaft f{\"u}r Informatik, Bonn}
+}
+
+@inproceedings{raji2020closing,
+  title={Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing},
+  author={Raji, Inioluwa Deborah and Smart, Andrew and White, Rebecca N and Mitchell, Margaret and Gebru, Timnit and Hutchinson, Ben and Smith-Loud, Jamila and Theron, Daniel and Barnes, Parker},
+  booktitle={Proceedings of the 2020 conference on fairness, accountability, and transparency},
+  pages={33--44},
+  year={2020}
+}
+
+@article{prado2020bonseyes,
+  title={Bonseyes AI pipeline—Bringing AI to you: End-to-end integration of data, algorithms, and deployment tools},
+  author={Prado, Miguel De and Su, Jing and Saeed, Rabia and Keller, Lorenzo and Vallez, Noelia and Anderson, Andrew and Gregg, David and Benini, Luca and Llewellynn, Tim and Ouerhani, Nabil and others},
+  journal={ACM Transactions on Internet of Things},
+  volume={1},
+  number={4},
+  pages={1--25},
+  year={2020},
+  publisher={ACM New York, NY, USA}
+}
+
+@inproceedings{john2020ai,
+  title={AI Deployment Architecture: Multi-Case Study for Key Factor Identification},
+  author={John, Meenu Mary and Olsson, Helena Holmstr{\"o}m and Bosch, Jan},
+  booktitle={2020 27th Asia-Pacific Software Engineering Conference (APSEC)},
+  pages={395--404},
+  year={2020},
+  organization={IEEE}
+}
+
+@incollection{bosch2021engineering,
+  title={Engineering ai systems: A research agenda},
+  author={Bosch, Jan and Olsson, Helena Holmstr{\"o}m and Crnkovic, Ivica},
+  booktitle={Artificial Intelligence Paradigms for Smart Cyber-Physical Systems},
+  pages={1--19},
+  year={2021},
+  publisher={IGI global}
+}
+
+@article{klaise2021alibi,
+  title={Alibi Explain: algorithms for explaining machine learning models},
+  author={Klaise, Janis and Van Looveren, Arnaud and Vacanti, Giovanni and Coca, Alexandru},
+  journal={Journal of Machine Learning Research},
+  volume={22},
+  number={181},
+  pages={1--7},
+  year={2021}
+}
+
+@misc{jurafsky2019speech,
+  title={Speech and Language Processing (3rd (draft) ed.)},
+  author={Jurafsky, Dan and Martin, James H},
+  year={2019},
+  publisher={Stanford Univ}
+}
+
+@book{shull2007guide,
+  title={Guide to advanced empirical software engineering},
+  author={Shull, Forrest and Singer, Janice and Sj{\o}berg, Dag IK},
+  year={2007},
+  publisher={Springer}
+}
+
+@book{kleppmann2017designing,
+  title={Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems},
+  author={Kleppmann, Martin},
+  year={2017},
+  publisher={" O'Reilly Media, Inc."}
+}
+
+@article{davison2004principles,
+  title={Principles of canonical action research},
+  author={Davison, Robert and Martinsons, Maris G and Kock, Ned},
+  journal={Information systems journal},
+  volume={14},
+  number={1},
+  pages={65--86},
+  year={2004},
+  publisher={Wiley Online Library}
+}
+
+@article{krosnick1999survey,
+  title={Survey research},
+  author={Krosnick, Jon A},
+  journal={Annual review of psychology},
+  volume={50},
+  number={1},
+  pages={537--567},
+  year={1999},
+  publisher={Annual Reviews 4139 El Camino Way, PO Box 10139, Palo Alto, CA 94303-0139, USA}
+}