andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 04c3486 with MkDocs version: 1.3.1

Browse source
This commit is contained in:
2022-09-11 17:39:51 +00:00
parent 292f62958b
commit 4f272fae1f
44 changed files with 382 additions and 360 deletions
Download patch file Download diff file Expand all files Collapse all files

36
thesis/chapters/5_cases/scibert.tex
View file

@ -3,7 +3,7 @@
The ScoutinScience Dashboard contains a full-page evaluation view for academic publications. On this, the known metadata, historical trends about the paper's topics, social media mentions, a PDF viewer showing the document, and other augmentation tools are displayed. One of these is the \textit{Highlights} section, which aims to summarise the paper from a technology-transfer perspective.
The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended with the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but inadequate for providing an accurate overview. Thus, this is the baseline I attempt to improve on in this section.
The current approach uses a simple heuristic based on a set of phrases selected by business developers and extended with the help of a word2vec model \cite{mikolov2013efficient}. The user feedback deemed this implementation slightly helpful but inadequate for providing an accurate overview. Thus, this is the baseline we attempt to improve on in this section.
\begin{displayquote}
Compared with Section \ref{section:simple-case}, this time around, the toolset of \textit{GreatAI} is available at our disposal. Hopefully, this will streamline the development and --- especially --- the deployment. Given its arguably higher complexity, the experiment falls closer to industrial use cases and hence, can give more accurate feedback on how to further improve the API.
@ -61,22 +61,6 @@ The best validation results were achieved after eight epochs which is slightly m
\label{fig:scibert-confusion}
\end{figure}
\begin{table}
\centering
\begin{threeparttable}
\caption{Accuracty metrics of the fine-tuned SciBERT model on the \textit{summary candidate sentences} dataset.}
\label{table:scibert-pr}
\setlength{\tabcolsep}{0.75em} % for the horizontal padding
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
\begin{tabular}{|l|r|r|r|}
\hline
{} & \textbf{Precision} & \textbf{Recall} & \textbf{Support} \\\hline
\textsc{non-relevant} & 0.93 & 0.83 & 191 \\\hline
\textsc{relevant} & 0.73 & 0.88 & 109 \\\hline
\end{tabular}}
\end{threeparttable}
\end{table}
Let us check how well the selected sentences correspond with the tech-transfer potential. Users and in-house experts can rate publications (from a tech-transfer perspective) by assigning them to one of four categories: \texttt{A}, \texttt{B}, \texttt{C}, and \texttt{D} with \texttt{A} being the most and \texttt{D} the least promising. This feedback is stored and used for analytic and training purposes. Since both the feedback grade and the relevant (summary candidate) sentences are supposed to reflect the same aspect of papers, we can reasonably expect some correlation between the grades and relevant sentence counts.
Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as predicted by the fine-tuned model in 4 categories (grades) of papers. This dataset does not overlap with the training data; hence, the results come solely from the model's ability to generalise. It is interesting to see that the Spearman's rank correlation coefficient \cite{spearman1961proof} between the normalised ``highlights'' counts and the ratings of papers is \textbf{0.4784} and is statistically significant ($P = 5.4 \times 10^{-74}$). This proves the presence of a monotonic association. For context, the correlation between the grades and the number of sentences chosen by the baseline approach is 0.06597 ($P = 0.03$). We can conclude that the classifier's output is indicative of publications' tech-transfer potential.
@ -89,6 +73,22 @@ Figure \ref{fig:histograms} shows the ratio of summary candidate sentences as pr
\label{fig:histograms}
\end{figure}
\begin{table}[H]
\centering
\begin{threeparttable}
\caption{Accuracty metrics of the fine-tuned SciBERT model on the \textit{summary candidate sentences} dataset.}
\label{table:scibert-pr}
\setlength{\tabcolsep}{0.75em} % for the horizontal padding
{\renewcommand{\arraystretch}{1.2} % for the vertical padding
\begin{tabular}{|l|r|r|r|}
\hline
{} & \textbf{Precision} & \textbf{Recall} & \textbf{Support} \\\hline
\textsc{non-relevant} & 0.93 & 0.83 & 191 \\\hline
\textsc{relevant} & 0.73 & 0.88 & 109 \\\hline
\end{tabular}}
\end{threeparttable}
\end{table}
\subsection{Deployment}
To implement the summarisation, at most, the top 7 selected sentences are chosen as ranked by their log probabilities. They are subsequently reordered according to their position in the text. As a quasi-explanation, the tokens' attention scores are visualised and overlaid on the highlighted sentences. The \textit{i}-th token's visualised attention comes from summing up the attention weights of each of the last layer's heads between the \texttt{[CLS]} and the \textit{i}-th token. To improve the end-user experience, a high-pass filter and a stop-word list are applied to the scores to avoid highlighting the syntax-related tokens (punctuation, determiners). The service --- after being integrated into the dashboard --- can be seen in Figure \ref{fig:dashboard-highlights}.
@ -145,7 +145,7 @@ Even though the REST API of \textit{GreatAI} services exposes all necessary feat
\begin{figure}
\centering
\includegraphics[width=1\linewidth]{figures/greatai-header.png}
\includegraphics[width=1\textwidth]{figures/greatai-header.png}
\captionsetup{width=.9\linewidth}
\caption{The header of the automatically generated dashboard of the service from Section \ref{section:simple-case}. A generated documentation is shown on the left, while the histogram of response times is rendered on the right. The current configuration is prominently displayed on the bottom.}
\label{fig:greatai-header}