Clarify some phrasing in the thesis
This commit is contained in:
parent
4653ef3459
commit
04c3486b83
8 changed files with 37 additions and 42 deletions
|
|
@ -1,18 +1,18 @@
|
|||
\begin{abstract}
|
||||
|
||||
\absdiv{Background}
|
||||
Despite its long-standing history, artificial intelligence (AI) has only recently started enjoying widespread industry awareness and adoption, partly thanks to the prevalence of libraries that accessibly expose state-of-the-art models. However, the transition from prototypes to production-ready AI applications is still a source of struggle across the industry. Even though professionals already have access to frameworks for deploying AI correctly, case studies and developer surveys have found that many deployments do not follow best practices.
|
||||
Despite its long-standing history, artificial intelligence (AI) has only recently started enjoying widespread industry awareness and adoption, partly thanks to the prevalence of libraries that accessibly expose state-of-the-art models. However, the transition from prototypes to production-ready AI applications is still a source of struggle across the industry. Even though professionals already have access to frameworks for deploying AI, case studies and developer surveys have found that many deployments do not follow best practices.
|
||||
|
||||
\absdiv{Objective}
|
||||
This thesis investigates the causes of and a possible resolution to the asymmetry between the adoption of libraries for applying and deploying AI. The potential solution is validated through designing a software framework, called \textit{GreatAI}, which aims to facilitate \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated \underline{T}rustworthy deployments while attempting to overcome the practical drawbacks of its predecessors.
|
||||
This thesis investigates the causes of and designs a possible solution to the asymmetry between the adoption of libraries for \textit{applying} and those for \textit{deploying} AI. The potential solution is validated through designing a software framework called \textit{GreatAI}, which aims to facilitate \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated \underline{T}rustworthy deployments while attempting to overcome the practical drawbacks of earlier similar tools e.g., \textit{Seldon Core}, \textit{AWS SageMaker}, and \textit{TensorFlow Extended}.
|
||||
|
||||
\absdiv{Methods}
|
||||
\textit{GreatAI} serves as a proxy for exploring the proposed design decisions; moreover, its initial focus is limited to the domain of natural language processing (NLP). Its design is validated by applying the principles of design science methodology through iteratively shaping it in two case studies of a commercial NLP pipeline. Subsequently, interviews are conducted with ten practitioners to assess its applicability and generalisability.
|
||||
|
||||
\absdiv{Results}
|
||||
\textit{GreatAI} successfully helps implement 33 best practices through an accessible interface. These target the transition between the prototype and production phases of the AI development lifecycle. The feedback from professional data scientists and software engineers showed that ease of use and functionality are equally important in deciding to adopt deployment technologies, and the proposed framework was rated overwhelmingly positively in both dimensions.
|
||||
\textit{GreatAI} helps implement 33 best practices through an accessible interface. These target the transition between the prototype and production phases of the AI development lifecycle. Feedback from professional data scientists and software engineers showed that ease of use and functionality are equally important in deciding to adopt deployment technologies, and the proposed framework was rated quite positively in both dimensions.
|
||||
|
||||
\absdiv{Conclusions}
|
||||
Increasing the overall maturity of industrial AI deployments by devising APIs with ease of adoption in mind is proved to be feasible. Additionally, the created software was deemed effective by experts and a candidate for raising awareness about the utility of following best practices.
|
||||
Increasing the overall maturity of industrial AI deployments by devising APIs with ease of adoption in mind is proved to be feasible. While \textit{GreatAI} focuses on NLP, the results show that the development and deployment of trustworthy AI pipelines, in general, can be assisted by frameworks prioritising easy adoption while still streamlining the implementation of various best practices.
|
||||
|
||||
\end{abstract}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
\chapter{Introduction}
|
||||
|
||||
Artificial intelligence (AI) techniques have recently started enjoying widespread industry awareness and adoption; the use of AI is increasingly prevalent in all sectors \cite{wirtz2019artificial,bosch2021engineering}. The reasons behind this are manifold \cite{jordan2015machine}, to name a few: recent breakthroughs in deep learning (DL), increased public awareness, an abundance of available data, access to powerful low-cost commodity hardware, education, but most interestingly, the rise of high-level libraries making ready-to-use state-of-the-art (SOTA) models easily available. The latter practically abolishes the barrier of entry for applying AI --- and with that --- can help use cases in various areas.
|
||||
Artificial intelligence (AI) techniques have recently started enjoying widespread industry awareness and adoption; the use of AI is increasingly prevalent in all sectors \cite{wirtz2019artificial,bosch2021engineering}. The reasons behind this are manifold \cite{jordan2015machine}, to name a few: recent breakthroughs in deep learning (DL), increased public awareness, an abundance of available data, access to powerful low-cost commodity hardware, education, but most interestingly, the rise of high-level libraries making ready-to-use state-of-the-art (SOTA) models easily available. The latter radically lowers the barrier of entry for applying AI --- and with that --- can help use cases in various areas.
|
||||
|
||||
However, to achieve robust deployments, the successful integration of AI components into production-ready applications demands strong engineering methods \cite{serban2020adoption}. That is why it is as essential as ever to also focus on the quality and robustness of deployed models and software. For instance, the lack of a proper overview of data transformation steps may lead to suboptimal performance and to introducing unintended biases, which might contribute to the ever-increasing negative externality of misused AI \cite{o2016weapons}.
|
||||
|
||||
|
|
@ -8,7 +8,7 @@ Concerningly, a peculiar tendency seems to be unfolding: even though industry pr
|
|||
|
||||
This thesis sets out to investigate the reasons behind the apparent asymmetry between industry adoption of accessible AI-libraries and existing reusable solutions for robust AI deployments. It is hypothesised that the primary reason for the underwhelming adoption rate of best practices is the short supply of professionals equally proficient in the domains of both data science and software engineering. Nevertheless, even without their presence, practitioners could rely on frameworks to achieve some level of automation and maturity in their deployment processes. However, the barrier of entry for using such existing libraries is too high, especially when compared with the simplicity of AI-libraries.
|
||||
|
||||
Therefore, a software framework --- called \href{https://github.com/schmelczer/great-ai}{\textit{GreatAI}} --- is designed, and its design is presented in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to facilitate the responsible and robust deployment of algorithms and models by designing a more accessible API in an attempt to overcome the practical drawbacks of other similar frameworks. Its name stands for its main aim: to assist easily creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
|
||||
Therefore, we design a software framework called \href{https://github.com/schmelczer/great-ai}{\textit{GreatAI}} and present it in this thesis. The principal motivation behind the construction of \textit{GreatAI} is to facilitate the responsible and robust deployment of algorithms and models by designing a more accessible API in an attempt to overcome the practical drawbacks of other similar frameworks. Its name stands for its main aim: to assist easily creating \underline{G}eneral \underline{R}obust \underline{E}nd-to-end \underline{A}utomated, and \underline{T}rustworthy AI deployments.
|
||||
|
||||
The utility of \textit{GreatAI} is validated using the principles of design science methodology \cite{wieringa2014design} through iteratively designing its API and implementation in a case study concerning the natural language processing (NLP) pipeline for a commercial product in collaboration with \href{https://scoutinscience.com/}{ScoutinScience B.V.} The goal of the aforementioned software suite is to evaluate technology transfer opportunities in scientific publications. Subsequently, interviews are conducted with practitioners to validate the generalisability of the design.
|
||||
|
||||
|
|
@ -16,7 +16,7 @@ The choice of case study subject is no coincidence; while working on the Scoutin
|
|||
|
||||
\section{Research questions}
|
||||
|
||||
I hypothesise that facilitating the adoption of AI deployment best practices is viable by finding less complex framework designs that are easier to adopt in order to decrease the negative externality of misused AI. This paper investigates the hypothesis by answering the following research questions.
|
||||
We hypothesise that facilitating the adoption of AI deployment best practices is viable by finding less complex framework designs that are easier to adopt in order to decrease the negative externality of misused AI. This paper investigates the hypothesis by answering the following research questions.
|
||||
|
||||
\begin{rqlist}
|
||||
\item To what extent does the complexity of deploying AI hinder industrial applications?
|
||||
|
|
|
|||
|
|
@ -6,9 +6,9 @@ In the following, the context of the problem is presented from three perspective
|
|||
|
||||
\section{Accessible AI} \label{section:accessible-ai}
|
||||
|
||||
Most companies prefer not to develop new models but instead reuse prior ones \cite{bosch2021engineering}, and they are able to do so increasingly easily. In recent years, there has been a proliferation of highly accessible AI-libraries. For example, let us consider the domain of natural language processing. There are various options for finding AI solutions that work out of the box: FLAIR \cite{akbik2019flair} and Hugging Face's transformers \cite{wolf2019huggingface} let developers access the state-of-the-art models and methods in only a couple of lines of code (in many cases 2 or 3). Using transfer-learning, Hugging Face enables developers to leverage vast amounts of knowledge learned by pretrained models (such as BERT \cite{devlin2018bert} and its many improved variations) and fine-tune them for their specific use case. The API exposing this is also extremely accessible.
|
||||
Most companies prefer not to develop new models but instead reuse prior ones \cite{bosch2021engineering}, and they are able to do so increasingly easily. In recent years, there has been a proliferation of highly accessible AI-libraries, many of which provide reusable models. For example, let us consider the domain of natural language processing. There are various options for finding AI solutions that work out of the box: FLAIR \cite{akbik2019flair} and Hugging Face's transformers \cite{wolf2019huggingface} let developers access the state-of-the-art models and methods in only a couple of lines of code (in many cases 2 or 3). Using transfer-learning, Hugging Face enables developers to leverage vast amounts of knowledge learned by pretrained models (such as BERT \cite{devlin2018bert} and its many improved variations) and fine-tune them for their specific use case. The API exposing this is also extremely accessible.
|
||||
|
||||
It is not just these two packages, the list of readily available tools is vast: SpaCy \cite{srinivasa2018natural}, Gensim \cite{vrehuuvrek2011gensim}, and scikit-learn \cite{pedregosa2011scikit}, XGBoost \cite{Chen_2016} are other great examples. The situation is similar in all subdomains of artificial intelligence: some domain expertise is --- admittedly --- beneficial but not a hard-requirement. This, combined with the exponentially increasing computing power affordably available to consumers and businesses alike \cite{sun2019summarizing}, results in AI that is accessible by many.
|
||||
It is not just these two libraries, the list of readily available solutions is vast: SpaCy \cite{srinivasa2018natural}, Gensim \cite{vrehuuvrek2011gensim}, and scikit-learn \cite{pedregosa2011scikit}, XGBoost \cite{Chen_2016} are other great examples. The situation is similar in all subdomains of artificial intelligence: some domain expertise is --- admittedly --- beneficial but not a hard-requirement. This, combined with the exponentially increasing computing power affordably available to consumers and businesses alike \cite{sun2019summarizing}, results in AI that is accessible by many.
|
||||
|
||||
\section{State of the industry} \label{section:industry}
|
||||
|
||||
|
|
|
|||
|
|
@ -24,9 +24,7 @@ The best practices (which are referenced throughout the thesis) with which the d
|
|||
|
||||
\paragraph{General} Albeit not explicitly in the list of best practices, compatibility is vital in encouraging adoption. Large projects frequently end up depending on numerous packages, each of which may impose some restrictions on the code: since these all have to be satisfied simultaneously, this can result in severe constraints.
|
||||
|
||||
The open-source scene of data-related libraries is vibrant. To take the example of data validation, there are at least four popular choices which offer varying but similar features: \href{https://github.com/SeldonIO/alibi-detect}{Alibi detect}, \href{https://github.com/PAIR-code/facets}{Facets}, \href{https://github.com/great-expectations/great_expectations}{Great Expectations}, and Data Linter \cite{hynes2017data}. The responsibility of choosing the most fitting solution falls on the user. Thus, they should not be limited in this by \textit{GreatAI}.
|
||||
|
||||
The programming language (PL) of the library should be its only non-general property. Fortunately, the de facto PL for data science is Python, so implementing the library in it should not significantly limit its applicability.
|
||||
The open-source scene of data-related libraries is vibrant. To take the example of data validation, there are at least four popular choices which offer varying but similar features: \href{https://github.com/SeldonIO/alibi-detect}{Alibi detect}, \href{https://github.com/PAIR-code/facets}{Facets}, \href{https://github.com/great-expectations/great_expectations}{Great Expectations}, and Data Linter \cite{hynes2017data}. The responsibility of choosing the most fitting solution falls on the user. Thus, they should not be limited in this by \textit{GreatAI}. On the contrary, the programming language (PL) of the library may be its only non-general property. Fortunately, the de facto PL for data science is Python, so implementing the library in it should not significantly limit its applicability.
|
||||
|
||||
\paragraph{Robustness} in software development can be achieved by preparing the application to handle errors gracefully, even unexpected ones \cite{bishop1998robust}. Errors can and will happen in practice: storing and investigating what has led to them is required to prevent future ones. In the case of ML, errors might not be as obvious to detect as in more traditional applications (see the above-mentioned data validators). Even if a single feature's value falls outside the expected distribution, unexpected results can happen. In cases where this might lead to real-world repercussions, extra care has to be taken to construct as many safeguards as practicable. \textit{GreatAI} should support its clients in this.
|
||||
|
||||
|
|
@ -40,7 +38,7 @@ The requirements were chosen stemming from their general importance and potentia
|
|||
|
||||
\section{Design principles} \label{section:principles}
|
||||
|
||||
Before diving into the concrete issues solved, let us detail the principles that should be used for implementing them in the scope of this framework. As implied in Section \ref{section:scope}, the Unix philosophy \cite{ritchie1978unix,salus1994quarter} of software design is followed. Most notably, the design goal that encourages to \textit{write programs that do one thing and do it well.}\footnote{\textit{Write programs to work together} is also applicable since allowing interoperability is part of the core requirements.}. Apart from providing a clear and simple picture of the intended use cases for the library, this is also in line with the main notion of \textit{A Philosophy of Software Design} \cite{ousterhout2018philosophy}: APIs should be narrow and deep. A narrow width refers to having a small exposed surface area, i.e. having a small number of functions and classes in the public API. In contrast, depth implies that each accomplishes an involved, complex goal.
|
||||
Before diving into the concrete issues being solved, let us detail the principles that should be used for implementing them in the scope of this framework. As implied in Section \ref{section:scope}, the Unix philosophy \cite{ritchie1978unix,salus1994quarter} of software design is followed. Most notably, the design goal that encourages to \textit{write programs that do one thing and do it well.}\footnote{\textit{Write programs to work together} is also applicable since allowing interoperability is part of the core requirements.}. Apart from providing a clear and simple picture of the intended use cases for the library, this is also in line with the main notion of \textit{A Philosophy of Software Design} \cite{ousterhout2018philosophy}: APIs should be narrow and deep. A narrow width refers to having a small exposed surface area, i.e. having a small number of functions and classes in the public API. In contrast, depth implies that each accomplishes an involved, complex goal.
|
||||
|
||||
In a way, the width of an API is the price users have to pay (the effort required for learning it) to use it, while the depth is analogous to the return they get from it. Having to learn little and being provided with a lot of functionality maximises return on investment (ROI), hence, developer experience (DX). The theoretical frameworks presented in \textit{The Programmer's Brain} \cite{hermans2021programmer} provides us with explanations and vocabulary from psychology for arguing about the cognitive aspects of API design. In the following, two of them will be used for detailing the design principles: cognitive dimensions of code bases (CDCB) which is an extension of the cognitive dimensions of notation (CDN) framework \cite{blackwell2001cognitive}, and linguistic anti-patterns \cite{arnaoudova2016linguistic}. The former comes with a set of dimensions describing different (often competing) cognitive aspects of code that influence one's ability to perform specific tasks.
|
||||
|
||||
|
|
@ -72,7 +70,7 @@ Once again, we might notice two competing interests: the level of detail and the
|
|||
|
||||
A large portion of software documentations is automatically generated from source code, and this has the advantage of always keeping it in sync with code changes. However, it might also signal that the API is too large because it is inconvenient for the developers to document it by hand. Striking the right balance between handcrafted and automatically extracted documentation may be a vital component of good documentation.
|
||||
|
||||
When it comes to example code, showing at least a minimal starter code and the way of customising it has to be showcased front and centre. It is a well-known observation that developers only read the documentation when they are stuck, and there might be some merit to this. Making them not get stuck --- by providing a starter code from which they can explore the API using IntelliSense-like solutions --- should be preferred. Take the example of another widely popular Python web framework, Flask\footnote{\href{https://flask.palletsprojects.com/en/2.1.x/}{flask.palletsprojects.com/en/2.1.x}}, at this time, has 324 homogeneously styled links on its landing page. Out of these, only two lead to the quick-start code. Of course, it is not hidden, but I argue that the DX could be improved by displaying where to start more prominently.
|
||||
When it comes to example code, showing at least a minimal starter code and the way of customising it has to be showcased front and centre. It is a well-known observation that developers only read the documentation when they are stuck, and there might be some merit to this. Making them not get stuck --- by providing a starter code from which they can explore the API using IntelliSense-like solutions --- should be preferred. Take the example of another widely popular Python web framework, Flask\footnote{\href{https://flask.palletsprojects.com/en/2.1.x/}{flask.palletsprojects.com/en/2.1.x}}, at this time, has 324 homogeneously styled links on its landing page. Out of these, only two lead to the quick-start code. Of course, it is not hidden, but we argue that the DX could be improved by displaying where to start more prominently.
|
||||
|
||||
\subsection{Developer experience}
|
||||
|
||||
|
|
|
|||
|
|
@ -117,7 +117,7 @@ The obstacles coming from the intertwined nature of different models are widely
|
|||
|
||||
We must address two data storage needs: training data and trained models. Proper version control is one of the most basic expectations for commercial codebases. Based on developer surveys, it is likely that our code is already tracked under Git and synchronised with GitHub\footnote{\href{https://octoverse.github.com/\#lets-look-back-at-the-code-and-communities-built-on-git-hub-this-year}{octoverse.github.com}}. Therefore, using Git Large File Storage (LFS) might seem intriguing. However, it is a paid (and surprisingly expensive) service of GitHub, especially when we factor in the expected sizes of the models and training data with the fact that the only way to remove files counting towards our quota is to delete the entire repository\footnote{\href{https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage\#git-lfs-objects-in-your-repository}{docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage}}.
|
||||
|
||||
An open-source tool, the Data Version Control (DVC)\footnote{\href{https://dvc.org/}{dvc.org}} provides a nearly perfect alternative. It comes with a command-line interface (CLI) inspired by Git's and can be integrated with several backend storage servers. Its only downside is, of course, that it is one more tool that increases the complexity of the project and the initial setup time. If this is an acceptable price to pay, then I personally recommend opting for DVC. Nevertheless, if this may prohibit a team\footnote{As was the case with MLFlow tracking in an ING team described in Section \ref{section:industry}.} from properly handling data according to the best practices, I present a simpler solution.
|
||||
An open-source tool, the Data Version Control (DVC)\footnote{\href{https://dvc.org/}{dvc.org}} provides a nearly perfect alternative. It comes with a command-line interface (CLI) inspired by Git's and can be integrated with several backend storage servers. Its only downside is, of course, that it is one more tool that increases the complexity of the project and the initial setup time. If this is an acceptable price to pay, then we highly recommend opting for DVC. Nevertheless, if this may prohibit a team\footnote{As was the case with MLFlow tracking in an ING team described in Section \ref{section:industry}.} from properly handling data according to the best practices, we present a simpler solution.
|
||||
|
||||
The complexity of an API can be decreased by relying on its users' preexisting knowledge, and known patterns \cite{hermans2021programmer,ousterhout2018philosophy}. Therefore, we can reuse familiar APIs, such as the \texttt{open()} method from Python. Therefore, a method is proposed which provides the same interface; however, the backing storage can be a mixture of local disk space, S3-compatible storage, MongoDB, or any other storage backend. It provides a superset of \texttt{open()}'s interface\footnote{\href{https://docs.python.org/3/library/functions.html\#open}{docs.python.org/3/library/functions.html\#open}}: the same parameters can be used with it resulting in similar observed behaviour. The expected features: versioning, progress bars, caching, garbage collecting the cache, and automatically deleting old remote versions are all present and come with recommended --- but easy to see and change --- configuration.
|
||||
|
||||
|
|
@ -127,7 +127,7 @@ Easing development is not merely about automating everything but also about maki
|
|||
|
||||
It is easy to notice multiple recurring tasks when it comes to processing text. Cleaning it from various extraction artifacts and normalising characters is one of the most common. But splitting sentences, language tagging, and robustly lemmatising are also often recurring tasks. Because having reusable and tested feature extraction code covers two best practices, it seems straightforward that a utility module could be created for this, which could be extensively tested through unit testing.
|
||||
|
||||
This is exactly the motivation behind \texttt{great\_ai.utilities}. Extra care has to be taken not to overfit these utilities on the cases considered in this chapter; however, I believe these are versatile enough to be helpful in many text-related contexts. A conclusive answer to this assumption will be found during the interviews.
|
||||
This is exactly the motivation behind \texttt{great\_ai.utilities}. Extra care has to be taken not to overfit these utilities on the cases considered in this chapter; however, we believe these are versatile enough to be helpful in many text-related contexts. A conclusive answer to this assumption will be found during the interviews.
|
||||
|
||||
Implementing the unit tests uncovered multiple edge cases and even runtime errors; hence, the merit of \textit{Test all Feature Extraction Code} best practice is unequivocal. There is one more best practice that could be partially covered here, especially because its solution also helps both during batch inference but also at training/feature extraction time: \textit{Enable Parallel Training Experiments}.
|
||||
|
||||
|
|
@ -136,7 +136,7 @@ A function called \texttt{parallel\_map()} is implemented which closely mimics t
|
|||
\subsection{Deployment approach}
|
||||
|
||||
Some of the expectations one might have for data-intensive (such as AI) software are similar to that for software in general. These are also captured by the best practices: \textit{Use Continuous Integration}, \textit{Automate Model Deployment}, and \textit{Enable Automatic Roll Backs for Production Model} to name a few. It is important to notice that these have already been solved by software engineering, more specifically, by the DevOps paradigm \cite{leite2019survey}.
|
||||
In line with the findings of John et al. \cite{john2020architecting} on the SOTA of AI deployments, I suggest we wrap the applications in a format more compatible with existing DevOps toolkits. Instead of reinventing the wheel, we should rely on more established DevOps best practices for implementing the SE4ML deployment best practices. Besides, organisations are expected to have their deployment processes for classical applications; thus, allowing them to reuse those for AI applications seems to be the most convenient approach.
|
||||
In line with the findings of John et al. \cite{john2020architecting} on the SOTA of AI deployments, we suggest wrapping the applications in a format more compatible with existing DevOps toolkits. Instead of reinventing the wheel, we should rely on more established DevOps best practices for implementing the SE4ML deployment best practices. Besides, organisations are expected to have their deployment processes for classical applications; thus, allowing them to reuse those for AI applications seems to be the most convenient approach.
|
||||
|
||||
Based on personal experiences, three types of software artifacts are identified (in the context of Python) for which a wide range of established practices exist. WSGI server\footnote{\href{https://peps.python.org/pep-3333/}{peps.python.org/pep-3333}} compatible applications, executable scripts, and Docker Images\footnote{\href{https://docs.docker.com/registry/spec/manifest-v2-2/}{docs.docker.com/registry/spec/manifest-v2-2}}. To achieve this, \textit{GreatAI} provides a compatibility layer between simple Python inference functions and all the above common artifacts. Taking functions as input for the first step also satisfies the requirement to be \textbf{General}. Nevertheless, to also allow customisation, additional configuration, metadata, and behavioural specification can be given as well.
|
||||
|
||||
|
|
@ -158,7 +158,7 @@ def greeter(name: str) -> str:
|
|||
\label{listing:hello-world}
|
||||
\end{listing}
|
||||
|
||||
The main advantage of the wrapping approach is that it does not require any input from the clients (by default). I opted for a decorator \cite{gamma1995design}, which lets users wrap their function by adding a single additional line of code as shown in Listing \ref{listing:hello-world}. After which, the created WSGI application can be accessed through the \texttt{greeter.app} property where \texttt{greeter} is the identifier of the user-defined function. A CLI script (\texttt{great-ai}), along with a \texttt{Dockerfile} are also provided to cover the other two deployment artifacts.
|
||||
The main advantage of the wrapping approach is that it does not require any input from the clients (by default). We opted for a decorator \cite{gamma1995design}, which lets users wrap their function by adding a single additional line of code as shown in Listing \ref{listing:hello-world}. After which, the created WSGI application can be accessed through the \texttt{greeter.app} property where \texttt{greeter} is the identifier of the user-defined function. A CLI script (\texttt{great-ai}), along with a \texttt{Dockerfile} are also provided to cover the other two deployment artifacts.
|
||||
|
||||
|
||||
\begin{listing}
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
The ScoutinScience Dashboard contains a full-page evaluation view for academic publications. On this, the known metadata, historical trends about the paper's topics, social media mentions, a PDF viewer showing the document, and other augmentation tools are displayed. One of these is the \textit{Highlights} section, which aims to summarise the paper from a technology-transfer perspective.
|
||||
|
||||
The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended with the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but inadequate for providing an accurate overview. Thus, this is the baseline I attempt to improve on in this section.
|
||||
The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended with the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but inadequate for providing an accurate overview. Thus, this is the baseline we attempt to improve on in this section.
|
||||
|
||||
\begin{displayquote}
|
||||
Compared with Section \ref{section:simple-case}, this time around, the toolset of \textit{GreatAI} is available at our disposal. Hopefully, this will streamline the development and --- especially --- the deployment. Given its arguably higher complexity, the experiment falls closer to industrial use cases and hence, can give more accurate feedback on how to further improve the API.
|
||||
|
|
@ -61,22 +61,6 @@ The best validation results were achieved after eight epochs which is slightly m
|
|||
\label{fig:scibert-confusion}
|
||||
\end{figure}
|
||||
|
||||
\begin{table}
|
||||
\centering
|
||||
\begin{threeparttable}
|
||||
\caption{Accuracty metrics of the fine-tuned SciBERT model on the \textit{summary candidate sentences} dataset.}
|
||||
\label{table:scibert-pr}
|
||||
\setlength{\tabcolsep}{0.75em} % for the horizontal padding
|
||||
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
|
||||
\begin{tabular}{|l|r|r|r|}
|
||||
\hline
|
||||
{} & \textbf{Precision} & \textbf{Recall} & \textbf{Support} \\\hline
|
||||
\textsc{non-relevant} & 0.93 & 0.83 & 191 \\\hline
|
||||
\textsc{relevant} & 0.73 & 0.88 & 109 \\\hline
|
||||
\end{tabular}}
|
||||
\end{threeparttable}
|
||||
\end{table}
|
||||
|
||||
Let us check how well the selected sentences correspond with the tech-transfer potential. Users and in-house experts can rate publications (from a tech-transfer perspective) by assigning them to one of four categories: \texttt{A}, \texttt{B}, \texttt{C}, and \texttt{D} with \texttt{A} being the most and \texttt{D} the least promising. This feedback is stored and used for analytic and training purposes. Since both the feedback grade and the relevant (summary candidate) sentences are supposed to reflect the same aspect of papers, we can reasonably expect some correlation between the grades and relevant sentence counts.
|
||||
|
||||
Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as predicted by the fine-tuned model in 4 categories (grades) of papers. This dataset does not overlap with the training data; hence, the results come solely from the model's ability to generalise. It is interesting to see that the Spearman's rank correlation coefficient \cite{spearman1961proof} between the normalised ``highlights'' counts and the ratings of papers is \textbf{0.4784} and is statistically significant ($P = 5.4 \times 10^{-74}$). This proves the presence of a monotonic association. For context, the correlation between the grades and the number of sentences chosen by the baseline approach is 0.06597 ($P = 0.03$). We can conclude that the classifier's output is indicative of publications' tech-transfer potential.
|
||||
|
|
@ -89,6 +73,22 @@ Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as pr
|
|||
\label{fig:histograms}
|
||||
\end{figure}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{threeparttable}
|
||||
\caption{Accuracty metrics of the fine-tuned SciBERT model on the \textit{summary candidate sentences} dataset.}
|
||||
\label{table:scibert-pr}
|
||||
\setlength{\tabcolsep}{0.75em} % for the horizontal padding
|
||||
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
|
||||
\begin{tabular}{|l|r|r|r|}
|
||||
\hline
|
||||
{} & \textbf{Precision} & \textbf{Recall} & \textbf{Support} \\\hline
|
||||
\textsc{non-relevant} & 0.93 & 0.83 & 191 \\\hline
|
||||
\textsc{relevant} & 0.73 & 0.88 & 109 \\\hline
|
||||
\end{tabular}}
|
||||
\end{threeparttable}
|
||||
\end{table}
|
||||
|
||||
\subsection{Deployment}
|
||||
|
||||
To implement the summarisation, at most, the top 7 selected sentences are chosen as ranked by their log probabilities. They are subsequently reordered according to their position in the text. As a quasi-explanation, the tokens' attention scores are visualised and overlaid on the highlighted sentences. The \textit{i}-th token's visualised attention comes from summing up the attention weights of each of the last layer's heads between the \texttt{[CLS]} and the \textit{i}-th token. To improve the end-user experience, a high-pass filter and a stop-word list are applied to the scores to avoid highlighting the syntax-related tokens (punctuation, determiners). The service --- after being integrated into the dashboard --- can be seen in Figure \ref{fig:dashboard-highlights}.
|
||||
|
|
@ -145,7 +145,7 @@ Even though the REST API of \textit{GreatAI} services exposes all necessary feat
|
|||
|
||||
\begin{figure}
|
||||
\centering
|
||||
\includegraphics[width=1\linewidth]{figures/greatai-header.png}
|
||||
\includegraphics[width=1\textwidth]{figures/greatai-header.png}
|
||||
\captionsetup{width=.9\linewidth}
|
||||
\caption{The header of the automatically generated dashboard of the service from Section \ref{section:simple-case}. A generated documentation is shown on the left, while the histogram of response times is rendered on the right. The current configuration is prominently displayed on the bottom.}
|
||||
\label{fig:greatai-header}
|
||||
|
|
|
|||
|
|
@ -168,7 +168,7 @@ During the larger discussions, \textit{GreatAI} was deemed appropriate for aware
|
|||
|
||||
My overall takeaway from this is that most features were well-received, and the high mean value of \textit{perceived utility} is credible. The criticism of being NLP-centric is also justified: the initial scope of the proof-of-principle framework was limited to this domain. Nonetheless, learning the experts' opinion that they wish to have a similarly specific solution to their problem contexts is reassuring because it proves that the API is not only generalisable but is expected to be generalised. At the same time, it is crucial to admit that no one-size-fits-all solution can exist for such a diverse domain. Therefore, allowing customizability and easy extension of the system must remain central design questions.
|
||||
|
||||
Regarding the API's level of abstraction, I have to agree with the experts that the problem of deployment cannot be ``magically'' solved by a trivial API. However, solving deployment problems can be streamlined, at least in simpler cases. At the same time, the complex ones can be left to the professionals with relevant knowledge. This parallels the AI-libraries that have inspired \textit{GreatAI}. For instance, Hugging Face \texttt{transformers} streamlines fine-tuning and applying SOTA models, but it does not provide any facilities to help you create the next SOTA architecture because that is a vastly more complex task that most users are not expected to tackle.
|
||||
Regarding the API's level of abstraction, we have to agree with the experts that the problem of deployment cannot be ``magically'' solved by a trivial API. However, solving deployment problems can be streamlined, at least in simpler cases. At the same time, the complex ones can be left to the professionals with relevant knowledge. This parallels the AI-libraries that have inspired \textit{GreatAI}. For instance, Hugging Face \texttt{transformers} streamlines fine-tuning and applying SOTA models, but it does not provide any facilities to help you create the next SOTA architecture because that is a vastly more complex task that most users are not expected to tackle.
|
||||
|
||||
In order to reach its goal of improving best practice adoption, \textit{GreatAI} can help raise awareness by presenting a verifiable value proposition, i.e. a couple of lines of code can already result in more maintainable, robust, high-quality deployments. This might prompt users or technical decision-makers to invest more in software engineering in AI/ML projects. Additionally, it can help the effectiveness of AI/software engineers by handling the grunt work of implementing some best practices, leaving them with more resources to focus on the complex and creative aspects of \textit{GREAT} deployments.
|
||||
|
||||
|
|
@ -194,10 +194,8 @@ Supporting the easy, direct upload of larger non-JSON files --- e.g. by saving t
|
|||
|
||||
\subsection{More best practices}
|
||||
|
||||
In order to greatly simplify its API, each \textit{GreatAI} Trace is a single document with a well-defined, multi-level schema that clients can also extend by calling \texttt{log\_metric}. MongoDB provides a convenient (and popular) method for persisting such documents; however, if there is some existing database in the environment, storing Traces in that can be favourable. \href{https://www.postgresql.org/}{PostgreSQL} is a popular choice, and it also features good JSON document support. Hence, introducing first-class integration for PostgreSQL could benefit some clients.
|
||||
In order to greatly simplify its API, each \textit{GreatAI} Trace is a single document with a well-defined schema that clients can also extend by calling \texttt{log\_metric}. MongoDB provides a convenient (and popular) method for persisting such documents; however, if there is some existing database in the environment, storing Traces in that can be favourable. \href{https://www.postgresql.org/}{PostgreSQL} is a popular choice, and it also features good JSON document support. Hence, introducing first-class integration for PostgreSQL could benefit some clients.
|
||||
|
||||
As described in Designing Data-intensive Applications \cite{kleppmann2017designing}, services can fall into three broad categories: online systems, batch processing, and stream processing (near-teal-time systems). As of yet, \textit{GreatAI} only provides streamlined support for the first two. Thus, developer experience could be improved by providing simple, direct integration with popular message queues/protocols, such as \href{https://kafka.apache.org/}{Apache Kafka} \cite{kreps2011kafka}, \href{https://aws.amazon.com/sqs/}{AWS SQS} \cite{garfinkel2007evaluation}, or \href{https://www.amqp.org/}{AMQP} \cite{vinoski2006advanced}.
|
||||
|
||||
Some metrics of \textit{GreatAI}, such as the cache statistics, versions, and derived data from traces, can already be conveniently queried from its REST API. Nevertheless, adding support for the de facto standard metric gathering tool \href{https://prometheus.io/}{Prometheus} could save the library's users from one more integration step.
|
||||
Data-intensive services can fall into three broad categories: online systems, batch processing, and stream processing (near-teal-time systems) \cite{kleppmann2017designing}. As of yet, \textit{GreatAI} only provides streamlined support for the first two. Thus, developer experience could be improved by providing simple, direct integration with popular message queues/protocols, such as \href{https://kafka.apache.org/}{Apache Kafka} \cite{kreps2011kafka}, \href{https://aws.amazon.com/sqs/}{AWS SQS} \cite{garfinkel2007evaluation}, or \href{https://www.amqp.org/}{AMQP} \cite{vinoski2006advanced}. Moreover, some metrics of \textit{GreatAI}, such as the cache statistics, versions, and derived data from traces, can already be conveniently queried from its REST API. Nevertheless, adding support for the de facto standard metric gathering tool \href{https://prometheus.io/}{Prometheus} could save the library's users from one more integration step.
|
||||
|
||||
The common theme among the opportunities mentioned above is that they could be implemented reasonably well without any user input, which aligns with the library's philosophy. Of course, the open-source nature of \textit{GreatAI} already allows anyone to provide support for a wide range of integrations. Additionally, the scope could be reasonably extended, i.e. more practices could be incorporated by including more criteria next to the \textit{GREAT} ones.
|
||||
|
|
|
|||
|
|
@ -144,7 +144,6 @@
|
|||
\fi}
|
||||
\makeatother
|
||||
|
||||
|
||||
\begin{document}
|
||||
|
||||
\includepdf[pages=-]{frontpage/frontpage.pdf}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue