andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 5a143b4 with MkDocs version: 1.3.1

Browse source
This commit is contained in:
2022-07-29 17:17:24 +00:00
parent e0d891df5b
commit a1bcbf4692
26 changed files with 589 additions and 1236 deletions
Download patch file Download diff file Expand all files Collapse all files

12
thesis/chapters/3_methods.tex
View file

@ -20,8 +20,14 @@ in order to decrease the negative externality of misused AI.}
However, before generalising, the design of the framework is iteratively refined using the feedback acquired from applying it in practical contexts which in this case is the research and development of a smaller and a more complex AI component using the work-in-progress framework. The treatment is finding a simple design (less cognitively straining to use) which still leads to high-quality deployments as defined in Section \ref{section:requirements}. Questions \textbf{RQ2} and \textbf{RQ3} capture this process; for investigating the feedback acquired from iteratively working on the case --- which is the definition of action research --- is of immense valuable.
\section{Generalisability}
\section{Applicability \& generalisability}
To answer how well the design of \textit{GreatAI} can generalise (\textbf{RQ4}), interviews are conducted from a population of software engineers and data scientists with varying levels of professional background. Since me and my colleagues are likely to have a bias for (or against) the proposed designs, the first step of checking its applicability in other practical contexts is to ask the opinion of non-affiliated fellow practitioners.
In order to conclusively answer \textbf{RQ3} and \textbf{RQ4}, interviews are conducted from a population of software engineers and data scientists with varying levels of professional background. Since me and my colleagues are likely to have a bias for (or against) the proposed design, the first step of checking its applicability in other practical contexts is to ask the opinion of non-affiliated practitioners.
todo
First, before their interview, interviewees are requested to complete a questionnaire about their last completed AI project; the questions come from a subset of the SE4ML survey. They are also advised to take a quick look at the tutorial page\footnote{\href{https://great-ai.scoutinscience.com/tutorial/}{great-ai.scoutinscience.com/tutorial}} of the documentation. The interviews are divided into two halves. In the first part, after a brief introduction, participants are asked to solve a real-world task by finishing a partially completed example application using GreatAI, they are also encouraged to think out loud so that their feedback can be noted. Successfully completing the task creates a system implementing a known number of best-practices. This way, the added value --- in terms of larger number of implemented best-practices --- can be quantitatively analysed by comparing the qualities of the finished implementation with the previously given answers.
Notes are taken throughout the interviews and subsequently extended using reflective journaling \cite{halcomb2006verbatim} combined with thematic coding. After which, the insights from the interviewed professionals are distilled using the techniques of thematic analysis \cite{fereday2006demonstrating} following the methodologies of \cite{cruz2019catalog} and \cite{haakman2021ai}. These insights can then be combined with the numerical results to explain and elaborate on them.
The second half consists of a short survey allowing to create the Technology Acceptance Model (TAM) \cite{davis1989perceived} of the problem context. The ultimate goal of the presented library is to help increase the adoption rate of best practices. In order to reach that goal, first, the library itself has to gain adoption. TAM and its numerous variations provide means of measuring users' willingness of adopting new technologies. TAM has been widely applied in literature \cite{marangunic2015technology} and due to its general psychological origins, it proves to be effective in other areas of technology, not just software \cite{riemenschneider2002explaining}.
The parsimonious version of TAM will be utilised which was measured to have similar predictive power to that of the original TAM while having fewer variables \cite{wu2011user}. Parsimonious TAM observes three interconnected human aspects that influence the actual behaviour (adoption): perceived usefulness, perceived ease of use, and intention to use. Participants are asked 10 questions corresponding to these aspects about their experience using GreatAI. The questionnaire is shown in Appendix \ref{appendix:questions}. The internal consistency of the answers is calculated using Chronbach's Alpha \cite{bland1997statistics} after which the responses are reflected upon.