\chapter{Methods} \label{chapter:methods}

The chosen methodology for this study is Design Science which emphasises the need to design and investigate artifacts in their context \cite{wieringa2014design}. It consists of a design and an empirical cycle. The purpose of the former is to improve a problem context with a new or redesigned artifact, while in the latter, the problem is investigated and its potential treatment is validated concurrently. This procedure seems fitting for our problem in consequence of its practical nature.

The design cycle shares similarities with Action Research \cite{davison2004principles} in which researchers attempt to solve a real-world problem while simultaneously studying the experience of solving said problem. As for the empirical cycle, the pragmatist approach is taken since the value of this research lies in its utility. Moreover, pragmatism adopts an engineering approach to research \cite{shull2007guide} which happens to be in line with the philosophy of design science. Additionally, as no research method is without flaws, it is imperative to try to compensate their weaknesses by applying multiple methods. Hence, the study also relies on interviews with professionals for validating the design decisions and determining the generalisability of \textit{GreatAI}.

\section{Design \& empirical cycles}

The aim of \textit{GreatAI} can be summarised using the terminology of design science in the following way: 
\textit{Facilitate the easy adoption of AI deployment best practices
by finding a less complex framework design 
which is easier to adopt
in order to decrease the negative externality of misused AI.}

The problem context is the difficulty in responsibly transitioning (while following best practices) from prototype industrial AI applications to production-ready deployments. With the possible treatment being libraries with high-level APIs and a set of default settings. It is important to note that \textit{GreatAI} is merely a proof-of-concept, and its aim is to serve as a proxy for the design decisions behind it. Through this, the design can be indirectly evaluated. Hopefully, a by-product will be a library that can be effectively applied to this problem context.

The practical cases used for the evaluation are further elaborated in Chapter \ref{chapter:case}. In short, they focus on individual components of a growing commercial platform with the aim of finding tech-transfer opportunities in academic publications. The main input of the system as a whole are PDFs while the output is a list of metrics describing various aspects of each paper, such as interesting sentences, scientific domains, and the scientific contribution. The output also includes a predicted score used for ranking. This ranking is subsequently processed by the business developers of Technology Transfer Offices (TTOs) of multiple Dutch and German universities who later give feedback on the results.

Overall, this problem context carries the properties of typical industry use-cases: it utilises a wide-range of natural language processing methods, contains complex interactions between the services, benefits from the integration of end-to-end feedback, and has to provide the clients with a platform that they can rely on in their organisation's core processes. Since the final ranking affects real people, explainability and robustness are also central questions.

Before generalising, the design of the framework is iteratively refined using the feedback acquired from applying it in practical contexts which in this case is the research and development of a smaller and a more complex AI component using the work-in-progress framework. The treatment is finding a simple, less cognitively straining to use, design which still leads to high-quality deployments as defined in Section \ref{section:requirements}.

\section{Applicability \& generalisability} \label{section:interview-setup}

In order to conclusively answer \textbf{RQ3} and \textbf{RQ4}, interviews are conducted from a population of software engineers and data scientists with varying levels of professional background. Since me and my colleagues are likely to have a bias for (or against) the proposed design, the first step of checking its applicability in other practical contexts is to ask the opinion of non-affiliated practitioners.

First, before their interview, interviewees are requested to complete a questionnaire (shown in Appendix \ref{appendix:practices}) about their last completed AI project; the questions refer to the best practices implemented by \textit{GreatAI} as described in Tables \ref{table:best-practices-1} and \ref{table:best-practices-2}. They are also advised to take a quick look at the tutorial page of the documentation. The interviews are divided into two halves. In the first part, after a brief introduction, participants are asked to solve a real-world task by finishing a partially completed example application using \textit{GreatAI}, they are also encouraged to think out loud so that their feedback can be noted. Successfully completing the task creates a system implementing a known number of best practices. This way, the added value --- in terms of larger number of implemented best practices --- can be quantitatively analysed by comparing the qualities of the finished implementation with the previously given answers. 

Notes are taken throughout the interviews and subsequently extended using reflective journaling \cite{halcomb2006verbatim} combined with thematic coding. After which, the insights from the interviewed professionals are distilled using the techniques of thematic analysis \cite{fereday2006demonstrating} following the methodologies of \cite{cruz2019catalog} and \cite{haakman2021ai}. These insights can then be combined with the numerical results to explain and elaborate on them. 

The second half consists of a short survey allowing to create the Technology Acceptance Model (TAM) \cite{davis1989perceived} of the problem context. The ultimate goal of the presented library is to help increase the adoption rate of best practices. In order to reach that goal, first, the library itself has to gain adoption. TAM and its numerous variations provide means of measuring users' willingness of adopting new technologies. TAM has been widely applied in literature \cite{marangunic2015technology} and due to its general psychological origins, it proves to be effective in other areas of technology, not just software \cite{riemenschneider2002explaining}. 

The parsimonious version of TAM will be employed which was measured to have similar predictive power to that of the original TAM while having fewer variables \cite{wu2011user}. Parsimonious TAM observes three interconnected human aspects that influence the actual behaviour (adoption): perceived usefulness, perceived ease of use, and intention to use. Participants are asked 10 questions corresponding to these aspects about their experience using \textit{GreatAI}. The questionnaire is shown in Appendix \ref{appendix:questions}. The internal consistency of the answers is calculated using Chronbach's Alpha \cite{bland1997statistics} after which the responses are reflected upon.
