86 lines
20 KiB
TeX
86 lines
20 KiB
TeX
\chapter{Background} \label{chapter:background}
|
|
|
|
Despite the long-standing history of artificial intelligence, industry awareness and adoption have only recently started to catch up meaningfully \cite{wirtz2019artificial}. At the same time, more regulations and guidelines are being published, for instance, the Ethics guidelines for trustworthy AI by the European Commission's High-Level Expert Group on AI\footnote{\href{https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}{digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai}}. This contains seven key requirements, including human agency and oversight, technical robustness, safety, transparency, and accountability. When it comes to accountability, clear advances are being made \cite{raji2020closing}; however, in the case of the other requirements, the situation is more nuanced. Thankfully, the field of software engineering for machine learning (SE4ML)\footnote{Both in practice and literature, this is sometimes also referred to as \textit{AI Engineering} and has a large intersection with --- or arguably is the same as --- \textit{MLOps}.} has been working towards finding ways to assist data scientists and software engineers in ensuring these (and more) expectations are met by their software.
|
|
|
|
In the following, the context of the problem is presented from three perspectives. Starting with its possible cause: the democratisation of state-of-the-art AI algorithms and models. Subsequently, the challenges encountered when applying AI in practice are outlined by case studies and survey data. Lastly, the existing approaches and solutions are introduced.
|
|
|
|
\section{Accessible AI} \label{section:accessible-ai}
|
|
|
|
Most companies prefer not to develop new models but instead reuse prior ones \cite{bosch2021engineering}, and they are able to do so increasingly easier. In recent years, there has been a proliferation of highly accessible AI libraries. For example, let us consider the domain of natural language processing (NLP). There are various options for finding AI solutions that work out of the box: FLAIR \cite{akbik2019flair} and Huggingface's transformers \cite{wolf2019huggingface} let developers access the state-of-the-art models and methods in only a couple of lines of code (in many cases 2 or 3). Using transfer-learning, Huggingface enables developers to leverage vast amounts of knowledge learned by pretrained models (such as BERT \cite{devlin2018bert} and its many improved variations) and fine-tune them for their specific use case. The API exposing this is also extremely accessible.
|
|
|
|
It is not just these two packages, the list of readily available tools is vast: SpaCy \cite{srinivasa2018natural}, Gensim \cite{vrehuuvrek2011gensim}, and scikit-learn \cite{pedregosa2011scikit}, XGBoost \cite{Chen_2016} are other great examples. The situation is similar in all subdomains of artificial intelligence: some domain expertise is --- admittedly --- beneficial but not a hard-requirement. This, combined with the exponentially increasing computing power affordably available to consumers and businesses alike \cite{sun2019summarizing}, results in AI that is accessible by many.
|
|
|
|
\section{State of the industry} \label{section:industry}
|
|
|
|
In contrast to this trend, the software landscape around packaging, deploying, and maintaining machine learning (ML) --- and in general --- data-heavy applications paints a different picture. Fortunately, the related issues and their ramifications have already been thoroughly investigated.
|
|
|
|
When looking at AI/ML\footnote{The terms AI and ML are often not differentiated and are used as synonyms in practice, for instance, see this study by the FDA \cite{food2019proposed}. ML is a well-defined subdomain of AI. However, most modern AI applications are also ML applications, hence, conflating the two terms may be slightly imprecise but usually not wrong.} code in practice through the lens of technical debt, Sculley et al. \cite{sculley2015hidden} emphasise the repercussions of writing \textit{glue code} between the algorithms and different systems or libraries and define it as an anti-pattern. The consequence of this is the advice against using generic libraries because their rigid APIs may inhibit improvements, cause lock-in, and result in large amounts of glue code. This is a recurring theme in discussions with industry professionals.
|
|
|
|
Haakman et al. \cite{haakman2021ai} interviewed 17 people at ING, which is a well-known fintech company undergoing a digital transformation to embrace AI. They found that the existing tools for ML do not meet the particularities of the field. For instance, a Feature Engineer working in the Data \& Analytics department explained that regular spreadsheets are preferred over existing solutions like MLFlow for keeping track of experiment results. The reason behind this is simplicity. Additionally, multiple other interviewees described the need to self-develop (or highly-customise) dashboards for monitoring deployed models, resulting in many non-reusable solutions across the company for the same problem. The authors conclude that there is a research gap between the ever-improving SOTA techniques and the challenges of developing real-world ML systems. In short, additional tool support is needed for facilitating the ML lifecycle.
|
|
|
|
In a case study at Microsoft, Amershi et al. \cite{amershi2019software} interviewed 14 people and surveyed another 551 AI and ML professionals from the company. One of the main concerns surfaced was relating to automation which is a vital cross-cutting concern, especially for testing. At the same time, a human-in-the-loop is still favoured. The survey data pointed out the difficulty posed by integrating AI, especially in the case of less experienced respondents. This was elaborated on by describing the preferences of software engineers as striving for elegant, abstract, modular, and simple systems; in contrast, data tends to be of large volume, context-specific and heterogeneous. Reconciling these inherent differences requires significant effort. Nevertheless, Microsoft manages to overcome this with a highly sophisticated internal infrastructure.
|
|
|
|
Using AI is not unique to large corporations; in a study conducted with the collaboration of three startups \cite{de2019understanding}, the aim was to fill in the gap of understanding how professionals develop ML systems in small companies. Overall, the results showed they have similar priorities to that of large companies, including an emphasis on the online monitoring of deployed models. However, less structure is present in the development lifecycle, as one interviewee explained: some steps are left out from time to time because they are forgotten about.
|
|
|
|
Similarly, Thiée \cite{thiee2021systematic} described the slow but ever-growing rate of ML adoption by small and medium-sized enterprises (SMEs). With the caveat that many more of these companies would wish to adopt data-driven approaches but are facing new challenges stemming from the domain's complexity.
|
|
|
|
Serban et al. \cite{serban2020adoption,serban2021practices} described the results of their global surveys aiming to ascertain the SOTA in how teams develop, deploy, and maintain ML systems. In \cite{serban2020adoption}, they compiled a set of 29 actionable best practices. These were analysed and validated with a survey of 313 participants to discover the adoption rate and relative importance of each best practice. For example, they determined the most important best practice to be \textit{logging production prediction traces}; however, the adoption was measured to be below 40\%. In more than three-quarters of the cases, newcomers to AI reported that they \textit{partially} or \textit{not at all} follow best practices. This tendency decreases with more years of experience, reaching a maximum adoption rate of just above 60\%. Similarly, Serban et al. in \cite{serban2021practices}, identified another 14 best practices that concern trustworthy AI, mainly through data governance. They strove to complement high-level checklists with actionable best practices. Analysing 42 survey responses revealed a familiar pattern: most best practices have less than 50\% adoption.
|
|
|
|
John et al. \cite{john2020architecting} compared and contrasted recent scientific and grey literature of AI deployments from which they extracted concrete challenges and practices. They also observed that most companies are placing many more models into production compared with previous years. Additionally, they pointed out that many deployment techniques are absent in contemporary literature, which is speculated to be caused by the immaturity of deployment processes employed in academia. Because for instance, most models in scientific literature experience only initial deployment and are not constantly replaced or refreshed as their performance degrades over time.
|
|
|
|
Finally, in a follow-up study to \cite{john2020architecting}, Bosch et al. \cite{bosch2021engineering} organised and structured the problem space of AI engineering research based on their 16 primary case studies. The authors noted the increasing and broad adoption of ML in the industry while also emphasising that the \textit{transition from prototype to production-quality deployment} proves to be challenging for many companies. Solid software engineering expertise is required to create additional facilities for the application, such as data pipelines, monitoring, and logging. They defined \textit{deployment \& compliance} to be one of the four main categories of problems and described it as highly underestimated and the source of ample struggle.
|
|
|
|
\section{Existing solutions} \label{section:existing}
|
|
|
|
From the previous section, it is noticeable that given enough resources and at the scale of 4195 AI professionals, Microsoft managed to create a comprehensive in-house solution. A similar impression is given by Uber \cite{li2017scaling}; they built a highly sophisticated infrastructure using techniques from distributed and high-performance computing. Though the authors note that even though this solution has shortcomings in the form of rigidity (number of supported libraries and model types), it still allows the easy extension of the system.
|
|
|
|
Given the nature of the problems faced and the amount of available resources, it is not surprising that both of these high-tech Fortune 500 companies needed to, and did overcome the problems presented by deploying AI. We can learn from their approaches; nonetheless, using them may be infeasible for individuals and SMEs. Thus, the issues remain for the majority of practitioners. Luckily, the open-source scene of AI/ML/DS tools, libraries, frameworks\footnote{The terms \textit{framework} and \textit{library} will be used interchangeably in this work stemming from their vague and often holistic differentiation.}, and platforms is thriving. Additionally, there is a considerable number of closed-source --- usually platforms-as-a-service (PaaS) --- solutions next to them. Let us look at some prominent examples.
|
|
|
|
IBM's AutoAI \cite{wang2020autoai} promises to provide automation for the entire machine learning lifecycle, including deployment. It is a closed-sourced, paid service which --- from their documentation --- seems to focus primarily on non-technical users by providing them with a UI for authoring models. The restrictions caused by the encapsulation of the entire process can be severe. The challenges of integration were emphasised above \cite{sculley2015hidden}. Additionally, an engineer working on Microsoft's comparable solution, the Azure ML Studio, highlighted that once users gain enough understanding of ML, such visual tools can get in their way, and they may need to seek out other solutions \cite{amershi2019software}. Unfortunately, the main value proposition of Azure ML Studio is also to provide a UI for laypeople, and it has also been set to be retired by 2024. Its successor is Azure Machine Learning which shares many similarities with AWS's SageMaker suite \cite{joshi2020amazon}.
|
|
|
|
SageMaker offers the most comprehensive suite of tools and services; most importantly, it has a set of features called \textit{AWS SageMaker MLOps}. This provides easy and/or default implementations for multiple industry best practices described in \cite{serban2020adoption,serban2021practices,john2020ai}. Among others, it promotes using CI/CD, model monitoring, tracing, model versioning, storing both data and models on shared infrastructure, numerous collaboration tools, etc. Nonetheless, SageMaker does not enjoy universal adoption, as indicated by the survey data. The cause of this may be the lack of a self-hosting option and its relatively high prices: many companies prefer on-premise hosting for privacy and financial reasons \cite{bosch2021engineering}. Additionally, vendor lock-in and possibly --- in the case where it is not already used for the project --- the initial effort required for setting up AWS integration could be likely deterrents.
|
|
|
|
When it comes to open-source libraries, we can find the MLOps libraries of both TensorFlow and PyTorch: TensorFlow Extended (TFX) \cite{baylor2017tfx} and TorchX\footnote{\href{https://pytorch.org/torchx/latest/}{pytorch.org/torchx/latest}}. TFX comes with a more mature set of features with the caveat that initial time investment is needed for their setup. The features of TorchX only concern the distributed deployment to a wide range of providers, including Kubernetes (K8s), AWS Batch, or Ray \cite{moritz2018ray}. There is no augmentation for the SE4ML best practices. Given the tight coupling between these libraries and their corresponding ML frameworks, they cannot generalise to models\footnote{The Open Neural Network Exchange (\href{https://onnx.ai/}{onnx.ai}) format could be an option for overcoming these incompatibilities. However, more universal support is needed for seamless integration.} or algorithms of other frameworks and technologies.
|
|
|
|
Open-source platforms also exist, such as MLflow and Seldon Core. They both rely on Kubernetes to provide their features. MLflow emphasises the training phase (in deployment, it lacks a feedback loop which is essential for reaching many of the best practices), while Seldon Core focuses on the deployment stage. The latter comes integrated with a powerful explanation engine, Alibi Explain \cite{klaise2021alibi}. It also boasts the most comprehensive suite of features, including outlier detection, online model selection (with multi-armed bandit theory), and distributed tracing. In short, it seems to be the ideal candidate for the title of \textit{framework for robust end-to-end AI deployments}. Its only downside is the amount of complexity propagated to its clients: it is built on top of Kubernetes and relies on Helm, Ambassador/Istio, Prometheus, and Jaeger for its features. Hence, the first step in using it is setting up a K8s cluster with all the required components; then, when it comes to model deployment, a Kubernetes configuration file must be created to use Seldon's Custom Resource Definition. These are more minor obstacles if the project is already built on top of K8s; however, even then, software engineers with solid cloud and DevOps backgrounds are actively required to use Seldon Core.
|
|
|
|
Additionally, increasing attention is given to ML deployments in embedded systems both from a theoretical \cite{john2020ai} and practical \cite{prado2020bonseyes} point of view. Prado et al. \cite{prado2020bonseyes} survey the available deployment frameworks and end-to-end solutions, including those for embedded devices. They note the inefficiencies of these that come from the lack of features and too much rigidity. They introduce their framework for embedded AI deployments, which can be used out-of-the-box but also lets users easily replace and extend its pipeline with steps to fit their changing needs and advancements in the field. At the same time, Meenu et al. \cite{john2020ai} present and compare different architectural choices for large-scale deployments in edge computing. They also note that: \textit{"...there is a need to consider and adapt well established SE practices which have been ignored or had a very narrow focus in ML literature"}.
|
|
|
|
In summary, the problems expressed in Section \ref{section:industry} can be understood when looking at the available solutions. Table \ref{table:platform-comparison} shows a high-level comparison of frameworks along the dimensions in which practitioners reportedly face difficulties in the \textit{Deployment} stage of the CRISP-DM model \cite{wirth2000crisp}.
|
|
|
|
\begin{table}
|
|
\centering
|
|
\begin{threeparttable}
|
|
\caption{High-level comparison of popular AI deployment platforms and libraries.}
|
|
\label{table:platform-comparison}
|
|
\setlength{\tabcolsep}{0.25em} % for the horizontal padding
|
|
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
|
|
\begin{tabular}{|l|c|c|c|c|c|c|c|}
|
|
\hline
|
|
& AutoAI & Azure ML & SageMaker & TFX & TorchX & MLflow & Seldon Core \\ \hline
|
|
Open-source\textsuperscript{1} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
|
Self-hosted\textsuperscript{1} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
|
Vendor-agnostic\textsuperscript{2} & & & & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
|
AI-agnostic\textsuperscript{2} & & \checkmark & \checkmark & & & \checkmark & \checkmark \\ \hline
|
|
E2E feedback\textsuperscript{3} & & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
|
Distributed monitoring\textsuperscript{3} & & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\textsuperscript{*} & \checkmark \\ \hline
|
|
Online model selection\textsuperscript{3} & \checkmark\textsuperscript{*} & \checkmark & \checkmark & & & & \checkmark \\ \hline
|
|
Versioning\textsuperscript{3} & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\ \hline
|
|
Quick setup\textsuperscript{4} & \checkmark & \checkmark & & & & & \\ \hline
|
|
No DevOps dependencies\textsuperscript{4}& & & & & \checkmark & & \\ \hline
|
|
\end{tabular}}
|
|
\begin{tablenotes}
|
|
\item[1] For privacy and accountability reasons. \cite{bosch2021engineering}
|
|
\item[2] Minimising required glue code. \cite{sculley2015hidden}
|
|
\item[3] Implementing best practices. \cite{serban2020adoption,serban2021practices,john2020architecting}
|
|
\item[4] Easy integration into existing processes. \cite{haakman2021ai,thiee2021systematic}
|
|
\item[*] Only partial support.
|
|
\end{tablenotes}
|
|
\end{threeparttable}
|
|
\end{table}
|
|
|
|
\section{Summary}
|
|
|
|
The surveys and case studies have shown the industry's continuous struggle to evolve prototypes into robust and responsible production-ready deployments. Simultaneously, platforms aiming to help overcome this challenge already exist but lack widespread adoption. The frequently recurring explanations for not adopting existing solutions surfaced in Section \ref{section:industry} revolve around their complexity and rigidity. These complaints are validated when looking at the available frameworks in Section \ref{section:existing}. While using AI has become more accessible than ever, deploying remains challenging owing to the lack of any \textit{easy-to-adopt framework for robust end-to-end AI deployments}.
|
|
|
|
The coexistence of multiple major obstacles, along with their promised solutions and the lack of their widespread adoption, leads us to believe that current frameworks are inadequate for many contexts. Thus, the answer to \textbf{RQ1} is that the complexity of deploying AI can severely hinder industrial applications even in the presence of existing frameworks. There is an unmet need for accessible AI deployment methods. The revolution brought by FLAIR, HuggingFace, and similar libraries for the domain of ML remains unmatched in the field of AI Engineering.
|