andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Improve thesis' fluency

Browse source
This commit is contained in:
Andras Schmelczer 2022-08-06 20:12:47 +02:00
parent a549f3e131
commit f382390291
No known key found for this signature in database
GPG key ID: 0EA1BC97D0AB076E
11 changed files with 93 additions and 89 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/thesis/chapters/3_methods.tex
View file

@ -14,9 +14,9 @@ in order to decrease the negative externality of misused AI.}
The problem context is the difficulty in responsibly transitioning (while following best practices) from prototype industrial AI applications to production-ready deployments. With the possible treatment being libraries with high-level APIs and a set of default settings. It is important to note that \textit{GreatAI} is merely a proof-of-concept, and its aim is to serve as a proxy for the design decisions behind it. Through this, the design can be indirectly evaluated. Hopefully, a by-product will be a library that can be effectively applied to this problem context.
The practical cases used for the evaluation are further elaborated in Chapter \ref{chapter:case}. In short, they focus on individual components of a growing commercial platform\footnote{\href{https://dashboard.scoutinscience.com/}{dashboard.scoutinscience.com}} with the aim of finding tech-transfer opportunities in academic publications. The main input of the system as a whole are PDFs while the output is a list of metrics describing various aspects of each paper, such as interesting sentences, scientific domains, and the scientific contribution. The output also includes a predicted score used for ranking. This ranking is subsequently processed by the business developers of Technology Transfer Offices (TTOs) of multiple Dutch and German universities who later give feedback on the results.
The practical cases used for the evaluation are further elaborated in Chapter \ref{chapter:case}. In short, they focus on individual components of a growing commercial platform with the aim of finding tech-transfer opportunities in academic publications. The main input of the system as a whole are PDFs while the output is a list of metrics describing various aspects of each paper, such as interesting sentences, scientific domains, and the scientific contribution. The output also includes a predicted score used for ranking. This ranking is subsequently processed by the business developers of Technology Transfer Offices (TTOs) of multiple Dutch and German universities who later give feedback on the results.
Overall, this problem context carries the properties of typical industry use cases: it utilises a wide-range of natural language processing methods, contains complex interactions between the services, benefits from the integration of end-to-end feedback, and has to provide the clients with a platform that they can rely on in their organisation's core processes. Since the final ranking affects real people, explainability and robustness are also central questions.
Overall, this problem context carries the properties of typical industry use-cases: it utilises a wide-range of natural language processing methods, contains complex interactions between the services, benefits from the integration of end-to-end feedback, and has to provide the clients with a platform that they can rely on in their organisation's core processes. Since the final ranking affects real people, explainability and robustness are also central questions.
Before generalising, the design of the framework is iteratively refined using the feedback acquired from applying it in practical contexts which in this case is the research and development of a smaller and a more complex AI component using the work-in-progress framework. The treatment is finding a simple, less cognitively straining to use, design which still leads to high-quality deployments as defined in Section \ref{section:requirements}.
@ -24,7 +24,7 @@ Before generalising, the design of the framework is iteratively refined using th
In order to conclusively answer \textbf{RQ3} and \textbf{RQ4}, interviews are conducted from a population of software engineers and data scientists with varying levels of professional background. Since me and my colleagues are likely to have a bias for (or against) the proposed design, the first step of checking its applicability in other practical contexts is to ask the opinion of non-affiliated practitioners.
First, before their interview, interviewees are requested to complete a questionnaire (shown in Appendix \ref{appendix:practices}) about their last completed AI project; the questions refer to the best practices implemented by \textit{GreatAI} as described in Tables \ref{table:best-practices-1} and \ref{table:best-practices-2}. They are also advised to take a quick look at the tutorial page\footnote{\href{https://great-ai.scoutinscience.com/tutorial/}{great-ai.scoutinscience.com/tutorial}} of the documentation. The interviews are divided into two halves. In the first part, after a brief introduction, participants are asked to solve a real-world task by finishing a partially completed example application using \textit{GreatAI}, they are also encouraged to think out loud so that their feedback can be noted. Successfully completing the task creates a system implementing a known number of best practices. This way, the added value --- in terms of larger number of implemented best practices --- can be quantitatively analysed by comparing the qualities of the finished implementation with the previously given answers.
First, before their interview, interviewees are requested to complete a questionnaire (shown in Appendix \ref{appendix:practices}) about their last completed AI project; the questions refer to the best practices implemented by \textit{GreatAI} as described in Tables \ref{table:best-practices-1} and \ref{table:best-practices-2}. They are also advised to take a quick look at the tutorial page of the documentation. The interviews are divided into two halves. In the first part, after a brief introduction, participants are asked to solve a real-world task by finishing a partially completed example application using \textit{GreatAI}, they are also encouraged to think out loud so that their feedback can be noted. Successfully completing the task creates a system implementing a known number of best practices. This way, the added value --- in terms of larger number of implemented best practices --- can be quantitatively analysed by comparing the qualities of the finished implementation with the previously given answers.
Notes are taken throughout the interviews and subsequently extended using reflective journaling \cite{halcomb2006verbatim} combined with thematic coding. After which, the insights from the interviewed professionals are distilled using the techniques of thematic analysis \cite{fereday2006demonstrating} following the methodologies of \cite{cruz2019catalog} and \cite{haakman2021ai}. These insights can then be combined with the numerical results to explain and elaborate on them.