{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Fine-tune SciBERT\n", "\n", "We are planning to do a simple classification task on scientific text. For that, [SciBERT](https://github.com/allenai/scibert) is an ideal model to fine-tune since it has been pretrained of academic publications.\n", "\n", "This notebook was updated so that it can run in [Google Colab](https://colab.research.google.com/).\n", "\n", "First, we need to install the dependencies." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "executionInfo": { "elapsed": 2529, "status": "ok", "timestamp": 1656596749103, "user_tz": -120 }, "id": "j7l0nD9hDQbB", "outputId": "88a9931b-396a-4cf1-c659-8a7b098b3cdd" }, "outputs": [], "source": [ "!pip install transformers datasets great-ai > /dev/null" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load the training data from S3. (We have uploaded this to S3 in the `data` notebook.)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;39mLatest version of summary-train-dataset-small is 0 (from versions: 0)\u001b[0m\n", "\u001b[38;5;39mFile summary-train-dataset-small-0 found in cache\u001b[0m\n" ] } ], "source": [ "from great_ai.large_file import LargeFileS3\n", "import json\n", "\n", "LargeFileS3.configure_credentials_from_file(\"config.ini\")\n", "\n", "with LargeFileS3(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n", " # splitting training and test data is done later by `datasets`\n", " X, y = json.load(f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finetune SciBERT, for more info about this step, check out [HuggingFace](https://huggingface.co/docs/transformers/training).\n", "If you're only here for `great-ai`, feel free to skip the next cell." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "executionInfo": { "elapsed": 118131, "status": "ok", "timestamp": 1656593941974, "user_tz": -120 }, "id": "AL3etUQ3LtKN", "outputId": "fe00589f-64dd-4b70-e612-3873b504c00a" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9c57de70e68a41ecbde5093bd671715a", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/1 [00:00\n", " \n", " \n", " [130/650 01:43 < 07:01, 1.23 it/s, Epoch 10/50]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation LossF1
10.5868000.5121380.719101
20.4116000.4166750.849057
30.2456000.4170700.864000
40.1478000.5758780.852459
50.0568000.4742590.896552
60.0225000.7542360.843137
70.0010000.8576360.834783
80.0005000.9202320.869565
90.0003000.9707900.877193
100.0003000.9486890.862385

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "...\n", "Deleting older checkpoint [models/checkpoint-39] due to args.save_total_limit\n", "***** Running Evaluation *****\n", " Num examples = 100\n", " Batch size = 32\n", "Saving model checkpoint to models/checkpoint-117\n", "Configuration saved in models/checkpoint-117/config.json\n", "Model weights saved in models/checkpoint-117/pytorch_model.bin\n", "Deleting older checkpoint [models/checkpoint-52] due to args.save_total_limit\n", "***** Running Evaluation *****\n", " Num examples = 100\n", " Batch size = 32\n", "Saving model checkpoint to models/checkpoint-130\n", "Configuration saved in models/checkpoint-130/config.json\n", "Model weights saved in models/checkpoint-130/pytorch_model.bin\n", "Deleting older checkpoint [models/checkpoint-78] due to args.save_total_limit\n", "\n", "\n", "Training completed. Do not forget to share your model on huggingface.co/models =)\n", "\n", "\n", "Loading best model from models/checkpoint-65 (score: 0.896551724137931).\n" ] } ], "source": [ "from transformers import (\n", " AutoModelForSequenceClassification,\n", " AutoTokenizer,\n", " DataCollatorWithPadding,\n", " Trainer,\n", " TrainingArguments,\n", " EarlyStoppingCallback,\n", ")\n", "from pathlib import Path\n", "import numpy as np\n", "from datasets import Dataset, load_metric\n", "\n", "MODEL = \"allenai/scibert_scivocab_uncased\"\n", "BATCH_SIZE = 32\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(MODEL)\n", "model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n", "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n", "\n", "\n", "def tokenize_function(v):\n", " return tokenizer(v[\"text\"])\n", "\n", "\n", "dataset = (\n", " Dataset.from_dict({\"text\": X, \"label\": y})\n", " .map(lambda v: tokenizer(v[\"text\"], truncation=True), batched=True)\n", " .remove_columns(\"text\")\n", " .train_test_split(test_size=0.2, shuffle=True)\n", ")\n", "\n", "f1_score = load_metric(\"f1\")\n", "\n", "\n", "def compute_metrics(p):\n", " pred, labels = p\n", " pred = np.argmax(pred, axis=1)\n", " return f1_score.compute(predictions=pred, references=labels)\n", "\n", "\n", "training_args = TrainingArguments(\n", " output_dir=Path(\"models\"),\n", " per_device_train_batch_size=BATCH_SIZE,\n", " per_device_eval_batch_size=BATCH_SIZE,\n", " save_total_limit=5,\n", " num_train_epochs=50,\n", " save_strategy=\"epoch\",\n", " evaluation_strategy=\"epoch\",\n", " logging_strategy=\"epoch\",\n", " weight_decay=0.01,\n", " metric_for_best_model=\"f1\",\n", " load_best_model_at_end=True,\n", ")\n", "\n", "result = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=dataset[\"train\"],\n", " eval_dataset=dataset[\"test\"],\n", " data_collator=data_collator,\n", " compute_metrics=compute_metrics,\n", " callbacks=[EarlyStoppingCallback(early_stopping_patience=5)],\n", ").train()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The best macro F1-score on the test set is **0.89** which is (not surprisingly) substantially more than the SVM achieved. We have a great model, it's time to deploy it. But first, we have to store it in a secure place." ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "executionInfo": { "elapsed": 25368, "status": "ok", "timestamp": 1656594537509, "user": { "displayName": "Schmelczer András", "userId": "08401926777942666437" }, "user_tz": -120 }, "id": "fyNKltdquZSP", "outputId": "e8c2cbb1-78e1-41a3-b7cf-b0cd573bc45d" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Configuration saved in pretrained/config.json\n", "Model weights saved in pretrained/pytorch_model.bin\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " adding: pretrained/ (stored 0%)\n", " adding: pretrained/config.json (deflated 49%)\n", " adding: pretrained/pytorch_model.bin (deflated 7%)\n" ] } ], "source": [ "from great_ai import save_model\n", "\n", "# save Torch model to local disk\n", "model.save_pretrained(\"pretrained\")\n", "\n", "# upload model from local disk to S3\n", "# (because the S3 credentials have been already set, `save_model` will use LargeFileS3)\n", "save_model(\"pretrained\", key=\"scibert-highlights\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next: [Part 3](/examples/scibert/deploy)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "vscode": { "interpreter": { "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad" } } }, "nbformat": 4, "nbformat_minor": 0 }