{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Explore data and feasibility of approach\n", "\n", "![screenshot fo the annotator tool](/media/annotator.png)\n", "\n", "We had asked our clients and in-house experts to annotate sentences using a rigorous guideline. The aim is to decide on which sentences they would like to see in a summary for a paper.\n", "\n", "The results are in JSON format, each annotator has a separate file. Let's load them." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data/evaluation-experiment-2-stage #1-sa6a0y.json\n", "data/evaluation-experiment-2-stage #1-2m6dmb.json\n" ] } ], "source": [ "from pathlib import Path\n", "import json\n", "\n", "annotations = []\n", "for p in Path(\"data\").glob(\"*.json\"):\n", " with open(p, encoding=\"utf-8\") as f:\n", " print(p)\n", " annotations.append(json.load(f))\n", "\n", "evaluations = {\n", " sentence: [\n", " annotation[sentence] for annotation in annotations if sentence in annotation\n", " ]\n", " for sentence in {\n", " sentence for annotation in annotations for sentence in annotation.keys()\n", " }\n", "}\n", "\n", "X = [s for s in evaluations.keys()]\n", "y = [int(sum(e) > 0) for e in evaluations.values()]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save the compiled and processed data for later use using LargeFileS3." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mCopying file for summary-train-dataset-small-0\u001b[0m\n", "\u001b[38;5;39mCompressing summary-train-dataset-small-0\u001b[0m\n", "\u001b[38;5;39mUploading summary-train-dataset-small-0 to S3 as summary-train-dataset-small/0\u001b[0m\n", "\u001b[38;5;39mUploading summary-train-dataset-small-0.tar.gz 0.04/0.04 MB (100.0%)\u001b[0m\n" ] } ], "source": [ "from great_ai.large_file import LargeFileS3\n", "import json\n", "\n", "LargeFileS3.configure_credentials_from_file(\"config.ini\")\n", "\n", "with LargeFileS3(\"summary-train-dataset-small\", \"w\", encoding=\"utf-8\") as f:\n", " json.dump((X, y), f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Filter out sentences which don't have enough annotations." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "y1 = [e[0] for e in evaluations.values() if len(e) == 2]\n", "y2 = [e[1] for e in evaluations.values() if len(e) == 2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Calculate [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa).\n", "\n", "It's a bit low but the task itself is pretty subjective so it's not all that surprising." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.3546448712421808" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import sklearn.metrics\n", "\n", "sklearn.metrics.cohen_kappa_score(y1, y2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Can we train anything on this data?\n", "\n", "Let's try with a trivial SVM." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "X = [s for s in evaluations.keys()]\n", "y = [int(sum(e) > 0) for e in evaluations.values()]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.79, 0.75, 0.77, 0.69, 0.77])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "from sklearn.svm import LinearSVC\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.model_selection import cross_val_score\n", "\n", "\n", "model = Pipeline(\n", " steps=[\n", " (\"vectorizer\", TfidfVectorizer(sublinear_tf=True, min_df=3, max_df=0.3)),\n", " (\"classifier\", LinearSVC()),\n", " ]\n", ") # baseline model\n", "\n", "cross_val_score(model, X, y, cv=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The cross-validation shows promising accuracies. But accuracy isn't everything, therefore, we should investigate the accuracy metrics." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('vectorizer',\n",
       "                 TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n",
       "                ('classifier', LinearSVC())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('vectorizer',\n", " TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),\n", " ('classifier', LinearSVC())])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n", "model.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " False 0.78 0.78 0.78 51\n", " True 0.78 0.78 0.78 49\n", "\n", " accuracy 0.78 100\n", " macro avg 0.78 0.78 0.78 100\n", "weighted avg 0.78 0.78 0.78 100\n", "\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEaCAYAAACGrEV/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfQElEQVR4nO3deZxcVZn/8c+3O52NkI0OIYSQhEUWGZIwkVUxgGyKg7ihoIIDooLiiI7KojgI/FxYdJRlAiiogGwqgkqCSIblxxYghEDYSQIkIQtZydZd/cwf93a6snVXJ1V1q7q+79frvqi7P0VTD+fcc885igjMzCxRl3UAZmaVxEnRzCyPk6KZWR4nRTOzPE6KZmZ5nBTNzPI4KZpZlyGpXtLTku5O10dKekzSK5JukdS9o2s4KZpZV/INYHre+k+AyyNiF2ARcEpHF3BSNLMuQdIOwEeAa9N1AYcCt6eH3AB8rKPrOCmaWVfxc+A7QEu6vg2wOCKa0/U3gaEdXaRbSULLWOPA+hgxrCHrMKwTXpraO+sQrJOWsWhBRAzakmscechWsfCdXEHHPjl19XPAqrxN4yNiPICkY4B5EfGkpHFbElOXTIojhjXw+IRhWYdhnXDk9qOzDsE66R9x+8wtvcaCd3I8NmGHgo5tGPLqqogYu4ndBwH/JunDQE+gL/ALoL+kbmlpcQfgrY7u4+qzmWUoyEVLQUu7V4k4OyJ2iIgRwGeAf0bEicD9wCfTw04C7uwoIidFM8tMAC1EQctm+i5wlqRXSJ4xXtfRCV2y+mxm1SEImqKwZ4oFXzNiEjAp/fwasG9nzndSNLNMbUEpsCScFM0sMwHknBTNzNq4pGhmlgogV2FTojgpmllmgqDJJUUzs1RArrJyopOimWUneU+xsjgpmlmGRA5lHcQ6nBTNLDMBtLj6bGbWxiVFM7NUAE1RWUMwOCmaWWaSHi0uKZqZARCIXIUN1uWkaGaZagmXFM3MAFefzczWEYimqKw0VFnRmFnNcUnRzCwVIXJ+JcfMrE2LS4pmZomkocUlRTOzlKvPZmZrJd386rMOYx1OimaWGfdoMTNbT4urz2ZmCTe0mJnlCUTOfZ/NzNq0uKRoZpaIkFufzcxaBfg9RTOzfMVoaJHUE3gA6EGS126PiPMlXQ98EFiSHnpyRExp71pOimaWmUDFGmR2NXBoRCyX1AA8JOnv6b7/jIjbC72Qk6KZZaoYJcWICGB5utqQLps1eWplVebNrKYkg8zWF7R0RFK9pCnAPODeiHgs3XWRpKmSLpfUo6PrOCmaWWaCpEdLIQvQKGly3nLaOteKyEXEaGAHYF9JewFnA7sD7wMGAt/tKCZXn80sU50YeXtBRIzt6KCIWCzpfuCoiLgk3bxa0m+Ab3d0vkuKZpaZCHWmpLhJkgZJ6p9+7gUcDrwgaUi6TcDHgGkdxeSSopllqkjvKQ4BbpBUT1LYuzUi7pb0T0mDAAFTgK90dCEnRTPLTFCc6QgiYiowZiPbD+3stZwUzSwzgWhqcTc/M7O1PHSYmVmqiD1aisZJ0cwy5aHDzMxSEXiQWTOzfK4+m5mlWvs+V5LKqszXoFwOTj/8PXz/CyMBmDurO2d+ZFdOPnAPLvrycJrWbPz/on/45bacfOAenPL+3Zk8aeu125+4f2tOef/unHzgHtzyy23L8h1qxVmXzeKWqc/xP/98ce22DxyzmPH3v8Df33yGXfdesclzx45byrUPvsBvHp7Op7/29trtg4et5hd3v8xvHp7OOVfPoFtDS0m/Q6VJ+j6roKVcSpYUJeUkTclbRrRz7PJN7evq/nztIIbtunrt+rUXDeHjX5rP9f9/On3657jn5oEbnDPzpR5MunMA4+9/gYtueo1fnb0DuVySYK84ZwcuvPE1rpn0AvffOYCZL3U4KIgVaOItAzn3xJHrbJvxQk8uOHUEzz661SbPq6sLzrj4Lc47cSRfGrcbhxy7mB13XQXAqefO4Y/XNPLFg/Zg+eJuHPXZd0r6HSpPcbr5FVMp77QyIkbnLTNKeK+qNH92A4/f15ejT1gIJA+dn3loaz5wzGIADv/UOzxyT78NzntkQj/GHbuI7j2C7XZcw/YjVvPi07158enebD9iNUOGr6GhezDu2EU8MmHD823zTHusD8sWrfvE6Y1XevLmqz3bPW+3MSuYPaM7c2f1oLmpjkl39ueAI5cAwaj3L+fBu/sDcO9tAzjgqCXtXqsrakEFLeVStvQrqY+k+yQ9JelZScdu5Jghkh5IS5bTJH0g3X6EpEfSc2+T1KdccZfS1ecP5dTzZqP0r7D0nXq26pejPv3dNQ5pYsHchg3OWzCngUHbN61dbxzSxMK5DSycu+H2BXM2PN/Ka5vtmpg/u/va9QVzGmgc0kTfgTneXVJPS05t27drzirMTLS2PheylEspk2KvvKrzn4BVwHERsQ9wCHBpOnJFvhOACemYaKOAKZIagfOAD6XnTgbOWv9mkk5rHWdt/sJcCb9WcTx6b1/6Nzaz694rsw7FLDOBaG6pL2gpl1K2Pq9MkxsA6bwJF0s6GGgBhgKDgbl55zwB/Do99s8RMUXSB4E9gYfTHNodeGT9m0XEeGA8wNhRPTdrGPJyev6JrXh0Yl+euG9P1qwWK5bVc9UPhvLuknpyzVDfrbXk0LTBuY1Dmpg/u60EuGBOA9ukx62/vXHIhudbeSUl+DVr11tL8K01g7r6oCWntGZQey+ElLNqXIhytj6fCAwC/jVNlm8D6zyMiYgHgIOBt4DrJX2BZMife/OeTe4ZEaeUMe6S+Pdz5nDjk8/z28ef5+yrZjLq/cv43hWzGHVQ/jOmgemzp3Xtf8RSJt05gDWrxdxZ3Xnr9R7sNmYFu41ewVuv92DurO40rRGT7hzA/kcsLfM3s/W9OKU3Q0euYfCw1XRraGHcsYt5dGI/QDzzcJ+8Z8i19wy4Elufy/m/pX7AvIhoknQIMHz9AyQNB96MiGvSuRT2AS4CrpC0S0S8ImkrYGhEvFTG2MvmlHNnc/FXh3P9T4ewy14rOTJtjXxkQl9eeqY3J31nLiN2W8XBH13MaeN2p74++NrFb1Kf1i7OuOhNzjlhJ1py4ojPvMOI3VZl+G26lu9dOZO9D1hOv4HN/H7y8/zu0sEsW9SN0y98i37bNPOj373Oq8/15NwTdmbg4Ca+eckbfP/zyd/iinOHcvFNr1FXDxP/MJCZLyXlgesuGsI5V83k5O/M5ZVpvZiwkbcNurpytiwXQskkWCW4sLQ8IvrkrTcCdwF9SJ4L7g8cHREzWo+VdBLwn0ATycxcX4iI1yUdCvyEZE5XgPMi4i+buvfYUT3j8QnDSvK9rDSO3H501iFYJ/0jbn+ykOkB2jNw923jsF9/oqBjbz/o6i2+XyFKVlLMT4jp+gLggPaOjYgbgBs2sv+fJBPPmFkXUqxBZoup9p7qmlnFCKC5pbKqz06KZpYpDwhhZpbyILNmZuvxM0Uzs1bh6rOZ2VqtL29XEidFM8tM0vfZrc9mZmuFS4pmZm3c0GJmlgo3tJiZravSqs+V9YTTzGqMyLXUFbS0exWpp6THJT0j6TlJ/5VuHynpMUmvSLpFUvd2L4STopllqIjjKa4GDo2IUcBo4ChJ+5OMrnV5ROwCLAI6HIvVSdHMshPJc8VClnYvk2idFbQhXQI4FLg93X4D8LGOQnJSNLNMFWs2P0n1kqYA84B7gVeBxRHROhvYmyTToLTLDS1mlpmgUw0tjZIm562PT+dmSq4VkQNGS+oP/AnYfXNiclI0swx1apScBYWMvB0RiyXdTzKodX9J3dLS4g4k8z+1y9VnM8tUS4sKWtojaVBaQkRSL+BwYDpwP/DJ9LCTgDs7isclRTPLTNKIUpT3FIcAN0iqJyns3RoRd0t6HviDpAuBp4HrOrqQk6KZZaoYPVoiYiowZiPbXwP27cy1nBTNLFMlmlB0szkpmlmmKq2bn5OimWXGc7SYmeUrXkNL0Tgpmlm2/EzRzKxN1ZQUJf2SdnJ4RJxZkojMrKZUU+vz5Hb2mZltsU72fS6LTSbFiLghf11S74hYUfqQzKxmBEQHXfjKrcO+z5IOSLvKvJCuj5J0ZckjM7PaEAUuZVLIgBA/B44EFgJExDPAwSWMycxqhogobCmXglqfI+INaZ2gcqUJx8xqThU1tLR6Q9KBQEhqAL5BMiSPmdmWqcCXtwupPn8FOINkGO/ZJJPCnFHCmMysllTYM8UOS4oRsQA4sQyxmFktqraSoqSdJN0lab6keZLulLRTOYIzsxpQYSXFQqrPNwG3koxsuz1wG3BzKYMysxoRJCXFQpYyKSQp9o6I30VEc7r8HuhZ6sDMrDYUY97nYmqv7/PA9OPfJX0P+ANJXj8e+FsZYjOzWlBFr+Q8SRJua7n1y3n7Aji7VEGZWe1QhXXza6/v88hyBmJmNajMjSiFKKhHi6S9gD3Je5YYEb8tVVBmVivK24hSiA6ToqTzgXEkSfFvwNHAQ4CTopltuQorKRbS+vxJ4DBgbkR8ERgF9CtpVGZWOyrsPcVCqs8rI6JFUrOkvsA8YFiJ4zKzWlFhJcVCkuJkSf2Ba0hapJcDj5QyKDOrEVFFrc+tIuL09OPVku4B+kbE1NKGZWY1o1pKipL2aW9fRDxVmpDMzLLTXknx0nb2BXBokWMxsxqkaikpRsQh5QykmF6a2psjtx+ddRjWCZfN8GPqarP38CJdqAjvKUoaRvKa4GCSQtv4iPiFpB8CXwLmp4eeExHtdlMu6OVtM7OSKN7rNs3AtyLiKUlbA09Kujfdd3lEXFLohZwUzSxTatnya0TEHGBO+nmZpOkkswV0WiEvb5uZlU6RX96WNAIYAzyWbvqapKmSfi1pQEfnFzLytiR9TtIP0vUdJe1beIhmZu0oPCk2Spqct5y2/qUk9QHuAP4jIpYCVwE7k8wtNYf2G5CBwqrPVwItJK3NFwDL0pu+r4Bzzcw2SdGp1ucFETF2k9dKZhu9A7gxIv4IEBFv5+2/Bri7o5sUUn3eLyLOAFalN1kEdC/gPDOzjhVhOgIlE9NfB0yPiMvytg/JO+w4YFpH4RRSUmySVE9agJU0iKTkaGa2xYrR0AIcBHweeFbSlHTbOcBnJY0myV8zWHew7I0qJCn+N/AnYFtJF5GMmnNep0M2M9uYIrySExEP0TZLQL5OT51SSN/nGyU9STJ8mICPRcT0zt7IzGwDnXumWBaFDDK7I7ACuCt/W0TMKmVgZlYjqi0pAn+lbQKrnsBI4EXgvSWMy8xqRbUlxYj4l/z1dPSc0zdxuJlZp1Rd9Xl9ad/C/UoRjJnVoGpLipLOylutA/YBZpcsIjOrHdXY0AJsnfe5meQZ4x2lCcfMak41JcX0pe2tI+LbZYrHzGpNtSRFSd0iolnSQeUMyMxqh6iu6vPjJM8Pp0j6C3Ab8G7rztYO12ZmW6SKkmKrnsBCklFyWt9XDMBJ0cy2TBSt73PRtJcUt01bnqfRlgxbVVhuN7OqVWHZpL2kWA/0YeOdrCvsa5hZtaqmZ4pzIuKCskViZrWpipLils87aGbWnuLN5lc07SXFw8oWhZnVrKppaImId8oZiJnVpmp6pmhmVnpOimZmqSp7pmhmVlKi8lp0nRTNLFsuKZqZtama1mczs7JwSdHMLFWlI2+bmZWOk6KZWRuXFM3M8jkpmpmlKnCQ2bqsAzCzGhcFLu2QNEzS/ZKel/ScpG+k2wdKulfSy+k/B3QUjpOimWWmdeKqQpYONAPfiog9gf2BMyTtCXwPuC8idgXuS9fb5aRoZtkqQkkxIuZExFPp52XAdGAocCxwQ3rYDcDHOgrHzxTNLFOK4ra0SBoBjAEeAwZHxJx011xgcEfnOymaWXY619DSKGly3vr4iBiff4CkPsAdwH9ExFKpbbiJiAip44q4k6KZZavwguKCiBi7qZ2SGkgS4o1589K/LWlIRMyRNASY19FN/EzRzDJVjIYWJUXC64DpEXFZ3q6/ACeln08C7uwoHpcUzSxbxXmkeBDweeBZSVPSbecAPwZulXQKMBP4dEcXclI0s+wUaUCIiHiITY9X26lJ+JwUzSxb7uZnZpYQoJbKyopOimaWKY+SY2udddks9vvQMhYv6MaXD90NgA8cs5jPf2suw3ZdzZkf3pWXp/be6Lljxy3lKz+aTX1d8PebB3Lrr5J3UgcPW805V82i74BmXn62Fz/9+o40N/klg2JoWiV+dfxeNK8WLTkx6uiFHHXWm7z0cF/uung40SJ6bJXjM5e8yqARqzY4/x9XbM9jtw6mrj447vzX2f2DSwCYPqk/f75gBC05sf/xb3PY6bPL/dWyU4Gz+ZXl1yJpG0lT0mWupLfy1ruXI4ZKNPGWgZx74sh1ts14oScXnDqCZx/dapPn1dUFZ1z8FuedOJIvjduNQ45dzI67Jj/CU8+dwx+vaeSLB+3B8sXdOOqz75T0O9SSbj2C0296jv+8Zyrf/ttUXvjf/sx4qg93nLcTn/vFK3z771PZ59gF/OOXQzc4d+7LvXj6rka+O3EKp90wnTu+vxMtOWjJwR9/MJLTrp/Od++dwlN/aWTuy70y+HbZUUthS7mUJSlGxMKIGB0Ro4Grgctb1yNijaSaLLFOe6wPyxat+9XfeKUnb77as93zdhuzgtkzujN3Vg+am+qYdGd/DjhyCRCMev9yHry7PwD33jaAA45aUqLoa48EPbZKfp25ZpFrFkrn6Fy1rB6AlUvr6Tt4zQbnTps4gDEfXUC3HsE2w1bTOHwVs6b0YdaUPjQOX8U2O66mW/dgzEcXMG1ihwO5dC1F6PtcTJklI0nXA6tI+ig+LGkpsDwiLkn3TwOOiYgZkj4HnAl0J+nPeHpE5LKJPHvbbNfE/NltBewFcxrYfZ8V9B2Y490l9bTktHZ743bNWYXZJbXk4LJj9mbBzJ4c9Pm5DB+znON//CrXfHF3Gnq20LNPjm/8adoG5y15uwfDxyxbu95vyBqWvJ38Dftvv3rt9v5D1jBzytal/yKVIiqvoSXrh007AAdGxFmbOkDSHsDxwEFpSTMHnFie8MzWVVcP3/77VM5/5ElmPdOHOS/24n+vG8KXfvMC5z/6FO/71HzuvHB41mFWlSINHVY0WVdbbyugxHcY8K/AE2nn7l5spP+ipNOA0wB6svHGia5i4dwGBm3fVkVrHNLEgjkNLH2nnq365airD1pySrbPzfpP3DX16pdjlwOWMn3SAGZP34rhY5YDMOaYBYw/aY8Nju83eDWL80r3S+Z0p19azV48u8fa7YvndKff4NUbnN+lVVZBMfOS4rt5n5tZN57WB2sCbsh7BrlbRPxw/QtFxPiIGBsRYxvosf7uLuXFKb0ZOnINg4etpltDC+OOXcyjE/sB4pmH+/CBYxYDcPinFvHIhH6ZxtqVLF/YjZVLkmeHa1bV8dJD/Ri8ywpWLatn3mvJf64vPtSfbXdZucG5ex2+iKfvaqR5tVj4Rg/mz+jJjqOXM2zUcubP6MnCN3rQvEY8fVcjex2+qKzfK0tFHGS2aCqpGDEDOAZA0j5Aa7PsfcCdki6PiHmSBgJbR8TMbMIsnu9dOZO9D1hOv4HN/H7y8/zu0sEsW9SN0y98i37bNPOj373Oq8/15NwTdmbg4Ca+eckbfP/zO9GSE1ecO5SLb3qNunqY+IeBzHwp+VFed9EQzrlqJid/Zy6vTOvFhJsHZvwtu46l87pz87d2oaUFokWM+shC3nvYYj71/17j+q/uhhT07tfMZ372KgDT7h3AG8/24eiz3mC796xk9DEL+cnho6nrFnzigtepS/IrH7/gdcZ/YQ9acmLfT89ju/dsmFS7rIhkqSCKMgck6YfAcmAv4O6IuD3d3otkBIuhJI0pBwBHpw0txwNnk5Qkm4AzIuLRTd2jrwbGfupUd0fL2GUzHsk6BOukvYe/9WR7Q3kVYuv+O8SYD36joGMf/Mt3tvh+hSh7SXFjVd90+0rgiE3suwW4pYRhmVlGKm02v0qqPptZrQmgwl7JcVI0s2xVVk50UjSzbHlACDOzfBXW+uykaGaZcknRzCylCuz77KRoZtnyKzlmZm3kZ4pmZqkKHHnbSdHMMlR5fZ+dFM0sU25oMTNrFe77bGa2LlefzczyVFZOdFI0s2z5lRwzs3wVlhSznqPFzGqYIlCusKXDa0m/ljQvnR65ddsPJb0laUq6fLij6zgpmlm2Wudp6Wjp2PXAURvZfnnexHd/6+girj6bWbaKVH2OiAckjdjS67ikaGbZCZIBIQpZNt/XJE1Nq9cDOjrYSdHMMqWIghagUdLkvOW0Ai5/FbAzMBqYA1za0QmuPptZtgqvPi/o7BSnEfF262dJ1wB3d3SOk6KZZScCWkrXz0/SkIiYk64eB0xr73hwUjSzrBUpJ0q6GRhHUs1+EzgfGCdpNMnTyxnAlzu6jpOimWWqWD1aIuKzG9l8XWev46RoZtmqsB4tTopmlp0APJ6imVmr0ja0bA4nRTPLlqvPZmYpV5/NzPIFhKvPZmZtXH02M0u5+mxmth63PpuZtSp4ANmycVI0s+wELimama3DJUUzszxOimZmrcKtz2ZmawVELpd1FOtwUjSzbLn6bGaWKvF0BJvDSdHMsuWSoplZm3BJ0cwsFQE5J0UzszYeOszMLBFA+D1FM7NUeJBZM7N1uKRoZpavwkqKigp7R6gYJM0HZmYdR4k0AguyDsI6pav+zYZHxKAtuYCke0j+/RRiQUQctSX3K0SXTIpdmaTJETE26ziscP6bVZe6rAMwM6skTopmZnmcFKvP+KwDsE7z36yK+JmimVkelxTNzPI4KZqZ5XFSNCsiJT4n6Qfp+o6S9s06Liuck2IVkNRb0vclXZOu7yrpmKzjso26EjgA+Gy6vgy4IrtwrLOcFKvDb4DVJD82gLeAC7MLx9qxX0ScAawCiIhFQPdsQ7LOcFKsDjtHxE+BJoCIWAEo25BsE5ok1ZOMioWkQUBlde61djkpVoc1knrR9kPbmaTkaJXnv4E/AdtKugh4CLg425CsM/yeYhWQdDhwHrAnMBE4CDg5IiZlGZdtnKTdgcNISvP3RcT0jEOyTnBSrBKStgH2J/mhPRoRXXHUlaonaceNbY+IWeWOxTaPk2IVkHQQMCUi3pX0OWAf4BcR0VWHR6takp4lecwhoCcwEngxIt6baWBWMD9TrA5XASskjQLOAl4FfpttSLYxEfEvEbF3+s9dgX2BR7KOywrnpFgdmiMp0h8LXBERVwBbZxyTFSAingL2yzoOK5ynI6gOyySdDXwOOFhSHdCQcUy2EZLOylutI3nUMTujcGwzuKRYHY4neQXnlIiYC+wA/CzbkGwTts5begB/JSnhW5VwQ4tZkaQvbf8kIr6ddSy2+Vx9rmCSlpG+sL3+LiAiom+ZQ7JNkNQtIprTNwWsirmkaFYEkp6KiH0kXQUMBW4D3m3dHxF/zCw46xSXFKuIpG1J3n0D/EJwheoJLAQOpe19xQCcFKuEk2IVkPRvwKXA9sA8YDgwHfALwZVj27TleRptybCVq2NVxK3P1eFHJF38XoqIkST9ah/NNiRbTz3QJ122zvvculiVcEmxOjRFxEJJdZLqIuJ+ST/POihbx5yIuCDrIGzLOSlWh8WS+gAPADdKmkfeQ3yrCB7fsotw63MFk7RjRMyStBWwkuRxx4lAP+DGiFiYaYC2lqSBEfFO1nHYlnNSrGCtr3mkn++IiE9kHZNZV+eGlsqWXyXbKbMozGqIk2Jli018NrMScfW5gknKkTSoCOgFrGjdhbv5mZWEk6KZWR5Xn83M8jgpmpnlcVKsUZJykqZImibpNkm9t+Ba10v6ZPr5Wkl7tnPsOEkHbsY9ZkhqLHT7escs7+S9fijJYyLWKCfF2rUyIkZHxF7AGuAr+TslbVZvp4g4NSKeb+eQcUCnk6JZuTgpGsCDwC5pKe5BSX8BnpdUL+lnkp6QNFXSlwGU+JWkFyX9A9i29UKSJkkam34+StJTkp6RdJ+kESTJ95tpKfUDkgZJuiO9xxOtg7RK2kbSREnPSbqWArrRSfqzpCfTc05bb9/l6fb7JA1Kt+0s6Z70nAfTSeytxrnvc41LS4RHA/ekm/YB9oqI19PEsiQi3iepB/CwpInAGGA3YE9gMPA88Ov1rjsIuAY4OL3WwIh4R9LVwPKIuCQ97ibg8oh4KJ1IfgKwB3A+8FBEXCDpI8ApBXydf0/v0Qt4Iu0FtBDYCpgcEd+U9IP02l8DxgNfiYiXJe0HXEkyDqLVMCfF2tVL0pT084PAdSTV2scj4vV0+xHA3q3PC0n6XO8KHAzcHBE5YLakf27k+vsDD7Req51+wR8C9pTWFgT7poNfHAx8PD33r5IWFfCdzpR0XPp5WBrrQqAFuCXd/nvgj+k9DgRuy7t3jwLuYV2ck2LtWhkRo/M3pMkhf/QdAV+PiAnrHffhIsZRB+wfEas2EkvBJI0jSbAHRMQKSZPIG6V8PZHed/H6/w7M/EzR2jMB+KqkBgBJ70lH7HkAOD595jgEOGQj5z5KMkf1yPTcgen2ZSSDsLaaCHy9dUXS6PTjA8AJ6bajgQEdxNoPWJQmxN1JSqqt6oDW0u4JJNXypcDrkj6V3kOSRnVwD6sBTorWnmtJnhc+JWka8D8ktYs/AS+n+34LPLL+iRExHziNpKr6DG3V17uA41obWoAzgbFpQ87ztLWC/xdJUn2OpBrd0Xw09wDdJE0Hfsy6I5O/C+ybfodDgdbBYE8ETknjew7Pz2y4m5+Z2TpcUjQzy+OkaGaWx0nRzCyPk6KZWR4nRTOzPE6KZmZ5nBTNzPI4KZqZ5fk/zhlvHEqJskYAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "y_predicted = model.predict(X_test)\n", "print(\n", " sklearn.metrics.classification_report(\n", " [y > 0 for y in y_test], [y > 0 for y in y_predicted]\n", " )\n", ")\n", "sklearn.metrics.ConfusionMatrixDisplay.from_predictions(\n", " [y > 0 for y in y_test],\n", " [y > 0 for y in y_predicted],\n", " xticks_rotation=\"vertical\",\n", " values_format=\".2f\",\n", ")\n", "None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We get an F1-score of 0.78 without any hyperparameter-optimisation; this task may be feasible to solve with AI.\n", "\n", "Next: [Part 2](/examples/scibert/train)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad" } } }, "nbformat": 4, "nbformat_minor": 2 }