374 lines
11 KiB
Text
374 lines
11 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Fine-tune SciBERT\n",
|
|
"\n",
|
|
"We are planning to do a simple classification task on scientific text. For that, [SciBERT](https://github.com/allenai/scibert) is an ideal model to fine-tune since it has been pretrained of academic publications.\n",
|
|
"\n",
|
|
"This notebook was updated so that it can run in [Google Colab](https://colab.research.google.com/).\n",
|
|
"\n",
|
|
"First, we need to install the dependencies."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"executionInfo": {
|
|
"elapsed": 2529,
|
|
"status": "ok",
|
|
"timestamp": 1656596749103,
|
|
"user_tz": -120
|
|
},
|
|
"id": "j7l0nD9hDQbB",
|
|
"outputId": "88a9931b-396a-4cf1-c659-8a7b098b3cdd"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install transformers datasets great-ai > /dev/null"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Load the training data from S3. (We have uploaded this to S3 in the `data` notebook.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;39mLatest version of summary-train-dataset-small is 0 (from versions: 0)\u001b[0m\n",
|
|
"\u001b[38;5;39mFile summary-train-dataset-small-0 found in cache\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from great_ai.large_file import LargeFileS3\n",
|
|
"import json\n",
|
|
"\n",
|
|
"LargeFileS3.configure_credentials_from_file(\"config.ini\")\n",
|
|
"\n",
|
|
"with LargeFileS3(\"summary-train-dataset-small\", encoding=\"utf-8\") as f:\n",
|
|
" # splitting training and test data is done later by `datasets`\n",
|
|
" X, y = json.load(f)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Finetune SciBERT, for more info about this step, check out [HuggingFace](https://huggingface.co/docs/transformers/training).\n",
|
|
"If you're only here for `great-ai`, feel free to skip the next cell."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"metadata": {
|
|
"executionInfo": {
|
|
"elapsed": 118131,
|
|
"status": "ok",
|
|
"timestamp": 1656593941974,
|
|
"user_tz": -120
|
|
},
|
|
"id": "AL3etUQ3LtKN",
|
|
"outputId": "fe00589f-64dd-4b70-e612-3873b504c00a"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "9c57de70e68a41ecbde5093bd671715a",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/1 [00:00<?, ?ba/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"\n",
|
|
" <div>\n",
|
|
" \n",
|
|
" <progress value='130' max='650' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
|
|
" [130/650 01:43 < 07:01, 1.23 it/s, Epoch 10/50]\n",
|
|
" </div>\n",
|
|
" <table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: left;\">\n",
|
|
" <th>Epoch</th>\n",
|
|
" <th>Training Loss</th>\n",
|
|
" <th>Validation Loss</th>\n",
|
|
" <th>F1</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <td>1</td>\n",
|
|
" <td>0.586800</td>\n",
|
|
" <td>0.512138</td>\n",
|
|
" <td>0.719101</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>2</td>\n",
|
|
" <td>0.411600</td>\n",
|
|
" <td>0.416675</td>\n",
|
|
" <td>0.849057</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>3</td>\n",
|
|
" <td>0.245600</td>\n",
|
|
" <td>0.417070</td>\n",
|
|
" <td>0.864000</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>4</td>\n",
|
|
" <td>0.147800</td>\n",
|
|
" <td>0.575878</td>\n",
|
|
" <td>0.852459</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>5</td>\n",
|
|
" <td>0.056800</td>\n",
|
|
" <td>0.474259</td>\n",
|
|
" <td>0.896552</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>6</td>\n",
|
|
" <td>0.022500</td>\n",
|
|
" <td>0.754236</td>\n",
|
|
" <td>0.843137</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>7</td>\n",
|
|
" <td>0.001000</td>\n",
|
|
" <td>0.857636</td>\n",
|
|
" <td>0.834783</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>8</td>\n",
|
|
" <td>0.000500</td>\n",
|
|
" <td>0.920232</td>\n",
|
|
" <td>0.869565</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>9</td>\n",
|
|
" <td>0.000300</td>\n",
|
|
" <td>0.970790</td>\n",
|
|
" <td>0.877193</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>10</td>\n",
|
|
" <td>0.000300</td>\n",
|
|
" <td>0.948689</td>\n",
|
|
" <td>0.862385</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table><p>"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.HTML object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"...\n",
|
|
"Deleting older checkpoint [models/checkpoint-39] due to args.save_total_limit\n",
|
|
"***** Running Evaluation *****\n",
|
|
" Num examples = 100\n",
|
|
" Batch size = 32\n",
|
|
"Saving model checkpoint to models/checkpoint-117\n",
|
|
"Configuration saved in models/checkpoint-117/config.json\n",
|
|
"Model weights saved in models/checkpoint-117/pytorch_model.bin\n",
|
|
"Deleting older checkpoint [models/checkpoint-52] due to args.save_total_limit\n",
|
|
"***** Running Evaluation *****\n",
|
|
" Num examples = 100\n",
|
|
" Batch size = 32\n",
|
|
"Saving model checkpoint to models/checkpoint-130\n",
|
|
"Configuration saved in models/checkpoint-130/config.json\n",
|
|
"Model weights saved in models/checkpoint-130/pytorch_model.bin\n",
|
|
"Deleting older checkpoint [models/checkpoint-78] due to args.save_total_limit\n",
|
|
"\n",
|
|
"\n",
|
|
"Training completed. Do not forget to share your model on huggingface.co/models =)\n",
|
|
"\n",
|
|
"\n",
|
|
"Loading best model from models/checkpoint-65 (score: 0.896551724137931).\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers import (\n",
|
|
" AutoModelForSequenceClassification,\n",
|
|
" AutoTokenizer,\n",
|
|
" DataCollatorWithPadding,\n",
|
|
" Trainer,\n",
|
|
" TrainingArguments,\n",
|
|
" EarlyStoppingCallback,\n",
|
|
")\n",
|
|
"from pathlib import Path\n",
|
|
"import numpy as np\n",
|
|
"from datasets import Dataset, load_metric\n",
|
|
"\n",
|
|
"MODEL = \"allenai/scibert_scivocab_uncased\"\n",
|
|
"BATCH_SIZE = 32\n",
|
|
"\n",
|
|
"tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
|
|
"model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=2)\n",
|
|
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
|
|
"\n",
|
|
"\n",
|
|
"def tokenize_function(v):\n",
|
|
" return tokenizer(v[\"text\"])\n",
|
|
"\n",
|
|
"\n",
|
|
"dataset = (\n",
|
|
" Dataset.from_dict({\"text\": X, \"label\": y})\n",
|
|
" .map(lambda v: tokenizer(v[\"text\"], truncation=True), batched=True)\n",
|
|
" .remove_columns(\"text\")\n",
|
|
" .train_test_split(test_size=0.2, shuffle=True) # test is actually validation\n",
|
|
")\n",
|
|
"\n",
|
|
"f1_score = load_metric(\"f1\")\n",
|
|
"\n",
|
|
"\n",
|
|
"def compute_metrics(p):\n",
|
|
" pred, labels = p\n",
|
|
" pred = np.argmax(pred, axis=1)\n",
|
|
" return f1_score.compute(predictions=pred, references=labels)\n",
|
|
"\n",
|
|
"\n",
|
|
"training_args = TrainingArguments(\n",
|
|
" output_dir=Path(\"models\"),\n",
|
|
" per_device_train_batch_size=BATCH_SIZE,\n",
|
|
" per_device_eval_batch_size=BATCH_SIZE,\n",
|
|
" save_total_limit=5,\n",
|
|
" num_train_epochs=50,\n",
|
|
" save_strategy=\"epoch\",\n",
|
|
" evaluation_strategy=\"epoch\",\n",
|
|
" logging_strategy=\"epoch\",\n",
|
|
" weight_decay=0.01,\n",
|
|
" metric_for_best_model=\"f1\",\n",
|
|
" load_best_model_at_end=True,\n",
|
|
")\n",
|
|
"\n",
|
|
"result = Trainer(\n",
|
|
" model=model,\n",
|
|
" args=training_args,\n",
|
|
" train_dataset=dataset[\"train\"],\n",
|
|
" eval_dataset=dataset[\"test\"],\n",
|
|
" data_collator=data_collator,\n",
|
|
" compute_metrics=compute_metrics,\n",
|
|
" callbacks=[EarlyStoppingCallback(early_stopping_patience=5)],\n",
|
|
").train()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The best macro F1-score on the test set is **0.89** which is (not surprisingly) substantially more than the SVM achieved. We have a great model, it's time to deploy it. But first, we have to store it in a secure place."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 44,
|
|
"metadata": {
|
|
"executionInfo": {
|
|
"elapsed": 25368,
|
|
"status": "ok",
|
|
"timestamp": 1656594537509,
|
|
"user": {
|
|
"displayName": "Schmelczer András",
|
|
"userId": "08401926777942666437"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "fyNKltdquZSP",
|
|
"outputId": "e8c2cbb1-78e1-41a3-b7cf-b0cd573bc45d"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Configuration saved in pretrained/config.json\n",
|
|
"Model weights saved in pretrained/pytorch_model.bin\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" adding: pretrained/ (stored 0%)\n",
|
|
" adding: pretrained/config.json (deflated 49%)\n",
|
|
" adding: pretrained/pytorch_model.bin (deflated 7%)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from great_ai import save_model\n",
|
|
"\n",
|
|
"# save Torch model to local disk\n",
|
|
"model.save_pretrained(\"pretrained\")\n",
|
|
"\n",
|
|
"# upload model from local disk to S3\n",
|
|
"# (because the S3 credentials have been already set, `save_model` will use LargeFileS3)\n",
|
|
"save_model(\"pretrained\", key=\"scibert-highlights\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Next: [Part 3](/examples/scibert/deploy)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3.10.4 ('.env': venv)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
},
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|