andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions
great-ai/examples/scibert/data/data.ipynb

310 lines
24 KiB
Text
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Explore data and feasibility of approach\n",
"\n",
"![screenshot fo the annotator tool](/media/annotator.png)\n",
"\n",
"We had asked our clients and in-house experts to annotate sentences using a rigorous guideline. The aim is to decide on which sentences they would like to see in a summary for a paper.\n",
"\n",
"The results are in JSON format, each annotator has a separate file. Let's load them."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"data/evaluation-experiment-2-stage #1-sa6a0y.json\n",
"data/evaluation-experiment-2-stage #1-2m6dmb.json\n"
]
}
],
"source": [
"from pathlib import Path\n",
"import json\n",
"\n",
"annotations = []\n",
"for p in Path(\"data\").glob(\"*.json\"):\n",
" with open(p, encoding=\"utf-8\") as f:\n",
" print(p)\n",
" annotations.append(json.load(f))\n",
"\n",
"evaluations = {\n",
" sentence: [\n",
" annotation[sentence] for annotation in annotations if sentence in annotation\n",
" ]\n",
" for sentence in {\n",
" sentence for annotation in annotations for sentence in annotation.keys()\n",
" }\n",
"}\n",
"\n",
"X = [s for s in evaluations.keys()]\n",
"y = [int(sum(e) > 0) for e in evaluations.values()]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Save the compiled and processed data for later use using LargeFileS3."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[38;5;39mCopying file for summary-train-dataset-small-0\u001b[0m\n",
"\u001b[38;5;39mCompressing summary-train-dataset-small-0\u001b[0m\n",
"\u001b[38;5;39mUploading summary-train-dataset-small-0 to S3 as summary-train-dataset-small/0\u001b[0m\n",
"\u001b[38;5;39mUploading summary-train-dataset-small-0.tar.gz 0.04/0.04 MB (100.0%)\u001b[0m\n"
]
}
],
"source": [
"from great_ai.large_file import LargeFileS3\n",
"import json\n",
"\n",
"LargeFileS3.configure_credentials_from_file(\"config.ini\")\n",
"\n",
"with LargeFileS3(\"summary-train-dataset-small\", \"w\", encoding=\"utf-8\") as f:\n",
" json.dump((X, y), f)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Filter out sentences which don't have enough annotations."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"y1 = [e[0] for e in evaluations.values() if len(e) == 2]\n",
"y2 = [e[1] for e in evaluations.values() if len(e) == 2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Calculate [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa).\n",
"\n",
"It's a bit low but the task itself is pretty subjective so it's not all that surprising."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.3546448712421808"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.metrics\n",
"\n",
"sklearn.metrics.cohen_kappa_score(y1, y2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Can we train anything on this data?\n",
"\n",
"Let's try with a trivial SVM."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"X = [s for s in evaluations.keys()]\n",
"y = [int(sum(e) > 0) for e in evaluations.values()]"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0.79, 0.75, 0.77, 0.69, 0.77])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from sklearn.svm import LinearSVC\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.model_selection import cross_val_score\n",
"\n",
"\n",
"model = Pipeline(\n",
" steps=[\n",
" (\"vectorizer\", TfidfVectorizer(sublinear_tf=True, min_df=3, max_df=0.3)),\n",
" (\"classifier\", LinearSVC()),\n",
" ]\n",
") # baseline model\n",
"\n",
"cross_val_score(model, X, y, cv=5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The cross-validation shows promising accuracies. But accuracy isn't everything, therefore, we should investigate the accuracy metrics."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;vectorizer&#x27;,\n",
" TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n",
" (&#x27;classifier&#x27;, LinearSVC())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vectorizer&#x27;,\n",
" TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n",
" (&#x27;classifier&#x27;, LinearSVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearSVC</label><div class=\"sk-toggleable__content\"><pre>LinearSVC()</pre></div></div></div></div></div></div></div>"
],
"text/plain": [
"Pipeline(steps=[('vectorizer',\n",
" TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n",
" ('classifier', LinearSVC())])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
"model.fit(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" False 0.78 0.78 0.78 51\n",
" True 0.78 0.78 0.78 49\n",
"\n",
" accuracy 0.78 100\n",
" macro avg 0.78 0.78 0.78 100\n",
"weighted avg 0.78 0.78 0.78 100\n",
"\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEaCAYAAACGrEV/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfQElEQVR4nO3deZxcVZn/8c+3O52NkI0OIYSQhEUWGZIwkVUxgGyKg7ihoIIDooLiiI7KojgI/FxYdJRlAiiogGwqgkqCSIblxxYghEDYSQIkIQtZydZd/cwf93a6snVXJ1V1q7q+79frvqi7P0VTD+fcc885igjMzCxRl3UAZmaVxEnRzCyPk6KZWR4nRTOzPE6KZmZ5nBTNzPI4KZpZlyGpXtLTku5O10dKekzSK5JukdS9o2s4KZpZV/INYHre+k+AyyNiF2ARcEpHF3BSNLMuQdIOwEeAa9N1AYcCt6eH3AB8rKPrOCmaWVfxc+A7QEu6vg2wOCKa0/U3gaEdXaRbSULLWOPA+hgxrCHrMKwTXpraO+sQrJOWsWhBRAzakmscechWsfCdXEHHPjl19XPAqrxN4yNiPICkY4B5EfGkpHFbElOXTIojhjXw+IRhWYdhnXDk9qOzDsE66R9x+8wtvcaCd3I8NmGHgo5tGPLqqogYu4ndBwH/JunDQE+gL/ALoL+kbmlpcQfgrY7u4+qzmWUoyEVLQUu7V4k4OyJ2iIgRwGeAf0bEicD9wCfTw04C7uwoIidFM8tMAC1EQctm+i5wlqRXSJ4xXtfRCV2y+mxm1SEImqKwZ4oFXzNiEjAp/fwasG9nzndSNLNMbUEpsCScFM0sMwHknBTNzNq4pGhmlgogV2FTojgpmllmgqDJJUUzs1RArrJyopOimWUneU+xsjgpmlmGRA5lHcQ6nBTNLDMBtLj6bGbWxiVFM7NUAE1RWUMwOCmaWWaSHi0uKZqZARCIXIUN1uWkaGaZagmXFM3MAFefzczWEYimqKw0VFnRmFnNcUnRzCwVIXJ+JcfMrE2LS4pmZomkocUlRTOzlKvPZmZrJd386rMOYx1OimaWGfdoMTNbT4urz2ZmCTe0mJnlCUTOfZ/NzNq0uKRoZpaIkFufzcxaBfg9RTOzfMVoaJHUE3gA6EGS126PiPMlXQ98EFiSHnpyRExp71pOimaWmUDFGmR2NXBoRCyX1AA8JOnv6b7/jIjbC72Qk6KZZaoYJcWICGB5utqQLps1eWplVebNrKYkg8zWF7R0RFK9pCnAPODeiHgs3XWRpKmSLpfUo6PrOCmaWWaCpEdLIQvQKGly3nLaOteKyEXEaGAHYF9JewFnA7sD7wMGAt/tKCZXn80sU50YeXtBRIzt6KCIWCzpfuCoiLgk3bxa0m+Ab3d0vkuKZpaZCHWmpLhJkgZJ6p9+7gUcDrwgaUi6TcDHgGkdxeSSopllqkjvKQ4BbpBUT1LYuzUi7pb0T0mDAAFTgK90dCEnRTPLTFCc6QgiYiowZiPbD+3stZwUzSwzgWhqcTc/M7O1PHSYmVmqiD1aisZJ0cwy5aHDzMxSEXiQWTOzfK4+m5mlWvs+V5LKqszXoFwOTj/8PXz/CyMBmDurO2d+ZFdOPnAPLvrycJrWbPz/on/45bacfOAenPL+3Zk8aeu125+4f2tOef/unHzgHtzyy23L8h1qxVmXzeKWqc/xP/98ce22DxyzmPH3v8Df33yGXfdesclzx45byrUPvsBvHp7Op7/29trtg4et5hd3v8xvHp7OOVfPoFtDS0m/Q6VJ+j6roKVcSpYUJeUkTclbRrRz7PJN7evq/nztIIbtunrt+rUXDeHjX5rP9f9/On3657jn5oEbnDPzpR5MunMA4+9/gYtueo1fnb0DuVySYK84ZwcuvPE1rpn0AvffOYCZL3U4KIgVaOItAzn3xJHrbJvxQk8uOHUEzz661SbPq6sLzrj4Lc47cSRfGrcbhxy7mB13XQXAqefO4Y/XNPLFg/Zg+eJuHPXZd0r6HSpPcbr5FVMp77QyIkbnLTNKeK+qNH92A4/f15ejT1gIJA+dn3loaz5wzGIADv/UOzxyT78NzntkQj/GHbuI7j2C7XZcw/YjVvPi07158enebD9iNUOGr6GhezDu2EU8MmHD823zTHusD8sWrfvE6Y1XevLmqz3bPW+3MSuYPaM7c2f1oLmpjkl39ueAI5cAwaj3L+fBu/sDcO9tAzjgqCXtXqsrakEFLeVStvQrqY+k+yQ9JelZScdu5Jghkh5IS5bTJH0g3X6EpEfSc2+T1KdccZfS1ecP5dTzZqP0r7D0nXq26pejPv3dNQ5pYsHchg3OWzCngUHbN61dbxzSxMK5DSycu+H2BXM2PN/Ka5vtmpg/u/va9QVzGmgc0kTfgTneXVJPS05t27drzirMTLS2PheylEspk2KvvKrzn4BVwHERsQ9wCHBpOnJFvhOACemYaKOAKZIagfOAD6XnTgbOWv9mkk5rHWdt/sJcCb9WcTx6b1/6Nzaz694rsw7FLDOBaG6pL2gpl1K2Pq9MkxsA6bwJF0s6GGgBhgKDgbl55zwB/Do99s8RMUXSB4E9gYfTHNodeGT9m0XEeGA8wNhRPTdrGPJyev6JrXh0Yl+euG9P1qwWK5bVc9UPhvLuknpyzVDfrbXk0LTBuY1Dmpg/u60EuGBOA9ukx62/vXHIhudbeSUl+DVr11tL8K01g7r6oCWntGZQey+ElLNqXIhytj6fCAwC/jVNlm8D6zyMiYgHgIOBt4DrJX2BZMife/OeTe4ZEaeUMe6S+Pdz5nDjk8/z28ef5+yrZjLq/cv43hWzGHVQ/jOmgemzp3Xtf8RSJt05gDWrxdxZ3Xnr9R7sNmYFu41ewVuv92DurO40rRGT7hzA/kcsLfM3s/W9OKU3Q0euYfCw1XRraGHcsYt5dGI/QDzzcJ+8Z8i19wy4Elufy/m/pX7AvIhoknQIMHz9AyQNB96MiGvSuRT2AS4CrpC0S0S8ImkrYGhEvFTG2MvmlHNnc/FXh3P9T4ewy14rOTJtjXxkQl9eeqY3J31nLiN2W8XBH13MaeN2p74++NrFb1Kf1i7OuOhNzjlhJ1py4ojPvMOI3VZl+G26lu9dOZO9D1hOv4HN/H7y8/zu0sEsW9SN0y98i37bNPOj373Oq8/15NwTdmbg4Ca+eckbfP/zyd/iinOHcvFNr1FXDxP/MJCZLyXlgesuGsI5V83k5O/M5ZVpvZiwkbcNurpytiwXQskkWCW4sLQ8IvrkrTcCdwF9SJ4L7g8cHREzWo+VdBLwn0ATycxcX4iI1yUdCvyEZE5XgPMi4i+buvfYUT3j8QnDSvK9rDSO3H501iFYJ/0jbn+ykOkB2jNw923jsF9/oqBjbz/o6i2+XyFKVlLMT4jp+gLggPaOjYgbgBs2sv+fJBPPmFkXUqxBZoup9p7qmlnFCKC5pbKqz06KZpYpDwhhZpbyILNmZuvxM0Uzs1bh6rOZ2VqtL29XEidFM8tM0vfZrc9mZmuFS4pmZm3c0GJmlgo3tJiZravSqs+V9YTTzGqMyLXUFbS0exWpp6THJT0j6TlJ/5VuHynpMUmvSLpFUvd2L4STopllqIjjKa4GDo2IUcBo4ChJ+5OMrnV5ROwCLAI6HIvVSdHMshPJc8VClnYvk2idFbQhXQI4FLg93X4D8LGOQnJSNLNMFWs2P0n1kqYA84B7gVeBxRHROhvYmyTToLTLDS1mlpmgUw0tjZIm562PT+dmSq4VkQNGS+oP/AnYfXNiclI0swx1apScBYWMvB0RiyXdTzKodX9J3dLS4g4k8z+1y9VnM8tUS4sKWtojaVBaQkRSL+BwYDpwP/DJ9LCTgDs7isclRTPLTNKIUpT3FIcAN0iqJyns3RoRd0t6HviDpAuBp4HrOrqQk6KZZaoYPVoiYiowZiPbXwP27cy1nBTNLFMlmlB0szkpmlmmKq2bn5OimWXGc7SYmeUrXkNL0Tgpmlm2/EzRzKxN1ZQUJf2SdnJ4RJxZkojMrKZUU+vz5Hb2mZltsU72fS6LTSbFiLghf11S74hYUfqQzKxmBEQHXfjKrcO+z5IOSLvKvJCuj5J0ZckjM7PaEAUuZVLIgBA/B44EFgJExDPAwSWMycxqhogobCmXglqfI+INaZ2gcqUJx8xqThU1tLR6Q9KBQEhqAL5BMiSPmdmWqcCXtwupPn8FOINkGO/ZJJPCnFHCmMysllTYM8UOS4oRsQA4sQyxmFktqraSoqSdJN0lab6keZLulLRTOYIzsxpQYSXFQqrPNwG3koxsuz1wG3BzKYMysxoRJCXFQpYyKSQp9o6I30VEc7r8HuhZ6sDMrDYUY97nYmqv7/PA9OPfJX0P+ANJXj8e+FsZYjOzWlBFr+Q8SRJua7n1y3n7Aji7VEGZWe1QhXXza6/v88hyBmJmNajMjSiFKKhHi6S9gD3Je5YYEb8tVVBmVivK24hSiA6ToqTzgXEkSfFvwNHAQ4CTopltuQorKRbS+vxJ4DBgbkR8ERgF9CtpVGZWOyrsPcVCqs8rI6JFUrOkvsA8YFiJ4zKzWlFhJcVCkuJkSf2Ba0hapJcDj5QyKDOrEVFFrc+tIuL09OPVku4B+kbE1NKGZWY1o1pKipL2aW9fRDxVmpDMzLLTXknx0nb2BXBokWMxsxqkaikpRsQh5QykmF6a2psjtx+ddRjWCZfN8GPqarP38CJdqAjvKUoaRvKa4GCSQtv4iPiFpB8CXwLmp4eeExHtdlMu6OVtM7OSKN7rNs3AtyLiKUlbA09Kujfdd3lEXFLohZwUzSxTatnya0TEHGBO+nmZpOkkswV0WiEvb5uZlU6RX96WNAIYAzyWbvqapKmSfi1pQEfnFzLytiR9TtIP0vUdJe1beIhmZu0oPCk2Spqct5y2/qUk9QHuAP4jIpYCVwE7k8wtNYf2G5CBwqrPVwItJK3NFwDL0pu+r4Bzzcw2SdGp1ucFETF2k9dKZhu9A7gxIv4IEBFv5+2/Bri7o5sUUn3eLyLOAFalN1kEdC/gPDOzjhVhOgIlE9NfB0yPiMvytg/JO+w4YFpH4RRSUmySVE9agJU0iKTkaGa2xYrR0AIcBHweeFbSlHTbOcBnJY0myV8zWHew7I0qJCn+N/AnYFtJF5GMmnNep0M2M9uYIrySExEP0TZLQL5OT51SSN/nGyU9STJ8mICPRcT0zt7IzGwDnXumWBaFDDK7I7ACuCt/W0TMKmVgZlYjqi0pAn+lbQKrnsBI4EXgvSWMy8xqRbUlxYj4l/z1dPSc0zdxuJlZp1Rd9Xl9ad/C/UoRjJnVoGpLipLOylutA/YBZpcsIjOrHdXY0AJsnfe5meQZ4x2lCcfMak41JcX0pe2tI+LbZYrHzGpNtSRFSd0iolnSQeUMyMxqh6iu6vPjJM8Pp0j6C3Ab8G7rztYO12ZmW6SKkmKrnsBCklFyWt9XDMBJ0cy2TBSt73PRtJcUt01bnqfRlgxbVVhuN7OqVWHZpL2kWA/0YeOdrCvsa5hZtaqmZ4pzIuKCskViZrWpipLils87aGbWnuLN5lc07SXFw8oWhZnVrKppaImId8oZiJnVpmp6pmhmVnpOimZmqSp7pmhmVlKi8lp0nRTNLFsuKZqZtama1mczs7JwSdHMLFWlI2+bmZWOk6KZWRuXFM3M8jkpmpmlKnCQ2bqsAzCzGhcFLu2QNEzS/ZKel/ScpG+k2wdKulfSy+k/B3QUjpOimWWmdeKqQpYONAPfiog9gf2BMyTtCXwPuC8idgXuS9fb5aRoZtkqQkkxIuZExFPp52XAdGAocCxwQ3rYDcDHOgrHzxTNLFOK4ra0SBoBjAEeAwZHxJx011xgcEfnOymaWXY619DSKGly3vr4iBiff4CkPsAdwH9ExFKpbbiJiAip44q4k6KZZavwguKCiBi7qZ2SGkgS4o1589K/LWlIRMyRNASY19FN/EzRzDJVjIYWJUXC64DpEXFZ3q6/ACeln08C7uwoHpcUzSxbxXmkeBDweeBZSVPSbecAPwZulXQKMBP4dEcXclI0s+wUaUCIiHiITY9X26lJ+JwUzSxb7uZnZpYQoJbKyopOimaWKY+SY2udddks9vvQMhYv6MaXD90NgA8cs5jPf2suw3ZdzZkf3pWXp/be6Lljxy3lKz+aTX1d8PebB3Lrr5J3UgcPW805V82i74BmXn62Fz/9+o40N/klg2JoWiV+dfxeNK8WLTkx6uiFHHXWm7z0cF/uung40SJ6bJXjM5e8yqARqzY4/x9XbM9jtw6mrj447vzX2f2DSwCYPqk/f75gBC05sf/xb3PY6bPL/dWyU4Gz+ZXl1yJpG0lT0mWupLfy1ruXI4ZKNPGWgZx74sh1ts14oScXnDqCZx/dapPn1dUFZ1z8FuedOJIvjduNQ45dzI67Jj/CU8+dwx+vaeSLB+3B8sXdOOqz75T0O9SSbj2C0296jv+8Zyrf/ttUXvjf/sx4qg93nLcTn/vFK3z771PZ59gF/OOXQzc4d+7LvXj6rka+O3EKp90wnTu+vxMtOWjJwR9/MJLTrp/Od++dwlN/aWTuy70y+HbZUUthS7mUJSlGxMKIGB0Ro4Grgctb1yNijaSaLLFOe6wPyxat+9XfeKUnb77as93zdhuzgtkzujN3Vg+am+qYdGd/DjhyCRCMev9yHry7PwD33jaAA45aUqLoa48EPbZKfp25ZpFrFkrn6Fy1rB6AlUvr6Tt4zQbnTps4gDEfXUC3HsE2w1bTOHwVs6b0YdaUPjQOX8U2O66mW/dgzEcXMG1ihwO5dC1F6PtcTJklI0nXA6tI+ig+LGkpsDwiLkn3TwOOiYgZkj4HnAl0J+nPeHpE5LKJPHvbbNfE/NltBewFcxrYfZ8V9B2Y490l9bTktHZ743bNWYXZJbXk4LJj9mbBzJ4c9Pm5DB+znON//CrXfHF3Gnq20LNPjm/8adoG5y15uwfDxyxbu95vyBqWvJ38Dftvv3rt9v5D1jBzytal/yKVIiqvoSXrh007AAdGxFmbOkDSHsDxwEFpSTMHnFie8MzWVVcP3/77VM5/5ElmPdOHOS/24n+vG8KXfvMC5z/6FO/71HzuvHB41mFWlSINHVY0WVdbbyugxHcY8K/AE2nn7l5spP+ipNOA0wB6svHGia5i4dwGBm3fVkVrHNLEgjkNLH2nnq365airD1pySrbPzfpP3DX16pdjlwOWMn3SAGZP34rhY5YDMOaYBYw/aY8Nju83eDWL80r3S+Z0p19azV48u8fa7YvndKff4NUbnN+lVVZBMfOS4rt5n5tZN57WB2sCbsh7BrlbRPxw/QtFxPiIGBsRYxvosf7uLuXFKb0ZOnINg4etpltDC+OOXcyjE/sB4pmH+/CBYxYDcPinFvHIhH6ZxtqVLF/YjZVLkmeHa1bV8dJD/Ri8ywpWLatn3mvJf64vPtSfbXdZucG5ex2+iKfvaqR5tVj4Rg/mz+jJjqOXM2zUcubP6MnCN3rQvEY8fVcjex2+qKzfK0tFHGS2aCqpGDEDOAZA0j5Aa7PsfcCdki6PiHmSBgJbR8TMbMIsnu9dOZO9D1hOv4HN/H7y8/zu0sEsW9SN0y98i37bNPOj373Oq8/15NwTdmbg4Ca+eckbfP/zO9GSE1ecO5SLb3qNunqY+IeBzHwp+VFed9EQzrlqJid/Zy6vTOvFhJsHZvwtu46l87pz87d2oaUFokWM+shC3nvYYj71/17j+q/uhhT07tfMZ372KgDT7h3AG8/24eiz3mC796xk9DEL+cnho6nrFnzigtepS/IrH7/gdcZ/YQ9acmLfT89ju/dsmFS7rIhkqSCKMgck6YfAcmAv4O6IuD3d3otkBIuhJI0pBwBHpw0txwNnk5Qkm4AzIuLRTd2jrwbGfupUd0fL2GUzHsk6BOukvYe/9WR7Q3kVYuv+O8SYD36joGMf/Mt3tvh+hSh7SXFjVd90+0rgiE3suwW4pYRhmVlGKm02v0qqPptZrQmgwl7JcVI0s2xVVk50UjSzbHlACDOzfBXW+uykaGaZcknRzCylCuz77KRoZtnyKzlmZm3kZ4pmZqkKHHnbSdHMMlR5fZ+dFM0sU25oMTNrFe77bGa2LlefzczyVFZOdFI0s2z5lRwzs3wVlhSznqPFzGqYIlCusKXDa0m/ljQvnR65ddsPJb0laUq6fLij6zgpmlm2Wudp6Wjp2PXAURvZfnnexHd/6+girj6bWbaKVH2OiAckjdjS67ikaGbZCZIBIQpZNt/XJE1Nq9cDOjrYSdHMMqWIghagUdLkvOW0Ai5/FbAzMBqYA1za0QmuPptZtgqvPi/o7BSnEfF262dJ1wB3d3SOk6KZZScCWkrXz0/SkIiYk64eB0xr73hwUjSzrBUpJ0q6GRhHUs1+EzgfGCdpNMnTyxnAlzu6jpOimWWqWD1aIuKzG9l8XWev46RoZtmqsB4tTopmlp0APJ6imVmr0ja0bA4nRTPLlqvPZmYpV5/NzPIFhKvPZmZtXH02M0u5+mxmth63PpuZtSp4ANmycVI0s+wELimama3DJUUzszxOimZmrcKtz2ZmawVELpd1FOtwUjSzbLn6bGaWKvF0BJvDSdHMsuWSoplZm3BJ0cwsFQE5J0UzszYeOszMLBFA+D1FM7NUeJBZM7N1uKRoZpavwkqKigp7R6gYJM0HZmYdR4k0AguyDsI6pav+zYZHxKAtuYCke0j+/RRiQUQctSX3K0SXTIpdmaTJETE26ziscP6bVZe6rAMwM6skTopmZnmcFKvP+KwDsE7z36yK+JmimVkelxTNzPI4KZqZ5XFSNCsiJT4n6Qfp+o6S9s06Liuck2IVkNRb0vclXZOu7yrpmKzjso26EjgA+Gy6vgy4IrtwrLOcFKvDb4DVJD82gLeAC7MLx9qxX0ScAawCiIhFQPdsQ7LOcFKsDjtHxE+BJoCIWAEo25BsE5ok1ZOMioWkQUBlde61djkpVoc1knrR9kPbmaTkaJXnv4E/AdtKugh4CLg425CsM/yeYhWQdDhwHrAnMBE4CDg5IiZlGZdtnKTdgcNISvP3RcT0jEOyTnBSrBKStgH2J/mhPRoRXXHUlaonaceNbY+IWeWOxTaPk2IVkHQQMCUi3pX0OWAf4BcR0VWHR6takp4lecwhoCcwEngxIt6baWBWMD9TrA5XASskjQLOAl4FfpttSLYxEfEvEbF3+s9dgX2BR7KOywrnpFgdmiMp0h8LXBERVwBbZxyTFSAingL2yzoOK5ynI6gOyySdDXwOOFhSHdCQcUy2EZLOylutI3nUMTujcGwzuKRYHY4neQXnlIiYC+wA/CzbkGwTts5begB/JSnhW5VwQ4tZkaQvbf8kIr6ddSy2+Vx9rmCSlpG+sL3+LiAiom+ZQ7JNkNQtIprTNwWsirmkaFYEkp6KiH0kXQUMBW4D3m3dHxF/zCw46xSXFKuIpG1J3n0D/EJwheoJLAQOpe19xQCcFKuEk2IVkPRvwKXA9sA8YDgwHfALwZVj27TleRptybCVq2NVxK3P1eFHJF38XoqIkST9ah/NNiRbTz3QJ122zvvculiVcEmxOjRFxEJJdZLqIuJ+ST/POihbx5yIuCDrIGzLOSlWh8WS+gAPADdKmkfeQ3yrCB7fsotw63MFk7RjRMyStBWwkuRxx4lAP+DGiFiYaYC2lqSBEfFO1nHYlnNSrGCtr3mkn++IiE9kHZNZV+eGlsqWXyXbKbMozGqIk2Jli018NrMScfW5gknKkTSoCOgFrGjdhbv5mZWEk6KZWR5Xn83M8jgpmpnlcVKsUZJykqZImibpNkm9t+Ba10v6ZPr5Wkl7tnPsOEkHbsY9ZkhqLHT7escs7+S9fijJYyLWKCfF2rUyIkZHxF7AGuAr+TslbVZvp4g4NSKeb+eQcUCnk6JZuTgpGsCDwC5pKe5BSX8BnpdUL+lnkp6QNFXSlwGU+JWkFyX9A9i29UKSJkkam34+StJTkp6RdJ+kESTJ95tpKfUDkgZJuiO9xxOtg7RK2kbSREnPSbqWArrRSfqzpCfTc05bb9/l6fb7JA1Kt+0s6Z70nAfTSeytxrnvc41LS4RHA/ekm/YB9oqI19PEsiQi3iepB/CwpInAGGA3YE9gMPA88Ov1rjsIuAY4OL3WwIh4R9LVwPKIuCQ97ibg8oh4KJ1IfgKwB3A+8FBEXCDpI8ApBXydf0/v0Qt4Iu0FtBDYCpgcEd+U9IP02l8DxgNfiYiXJe0HXEkyDqLVMCfF2tVL0pT084PAdSTV2scj4vV0+xHA3q3PC0n6XO8KHAzcHBE5YLakf27k+vsDD7Req51+wR8C9pTWFgT7poNfHAx8PD33r5IWFfCdzpR0XPp5WBrrQqAFuCXd/nvgj+k9DgRuy7t3jwLuYV2ck2LtWhkRo/M3pMkhf/QdAV+PiAnrHffhIsZRB+wfEas2EkvBJI0jSbAHRMQKSZPIG6V8PZHed/H6/w7M/EzR2jMB+KqkBgBJ70lH7HkAOD595jgEOGQj5z5KMkf1yPTcgen2ZSSDsLaaCHy9dUXS6PTjA8AJ6bajgQEdxNoPWJQmxN1JSqqt6oDW0u4JJNXypcDrkj6V3kOSRnVwD6sBTorWnmtJnhc+JWka8D8ktYs/AS+n+34LPLL+iRExHziNpKr6DG3V17uA41obWoAzgbFpQ87ztLWC/xdJUn2OpBrd0Xw09wDdJE0Hfsy6I5O/C+ybfodDgdbBYE8ETknjew7Pz2y4m5+Z2TpcUjQzy+OkaGaWx0nRzCyPk6KZWR4nRTOzPE6KZmZ5nBTNzPI4KZqZ5fk/zhlvHEqJskYAAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"y_predicted = model.predict(X_test)\n",
"print(\n",
" sklearn.metrics.classification_report(\n",
" [y > 0 for y in y_test], [y > 0 for y in y_predicted]\n",
" )\n",
")\n",
"sklearn.metrics.ConfusionMatrixDisplay.from_predictions(\n",
" [y > 0 for y in y_test],\n",
" [y > 0 for y in y_predicted],\n",
" xticks_rotation=\"vertical\",\n",
" values_format=\".2f\",\n",
")\n",
"None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We get an F1-score of 0.78 without any hyperparameter-optimisation; this task may be feasible to solve with AI.\n",
"\n",
"Next: [Part 2](/examples/scibert/train)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.4 ('.env': venv)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in a new issue View git blame Copy permalink