{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train your model\n", "\n", "The first step is, of course, to install `great-ai` in your Python environment." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install great-ai > /dev/null" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we have to get some data. After downloading it [from here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag), we might notice that the dataset is in [JSON Lines](https://jsonlines.org/) format (each line is a seperate JSON document). \n", "\n", "Let's write a function which takes a single line, and returns the sentence and the corresponding label from it. Before returning, the sentence is also [cleaned](/reference/utilities/#great_ai.utilities.clean.clean) to remove any LaTeX, XML, unicode, PDF-extraction artifacts." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import json\n", "from great_ai.utilities import clean\n", "\n", "\n", "def preprocess(line):\n", " data_point = json.loads(line)\n", "\n", " sentence = data_point[\"text\"]\n", " label = data_point[\"label\"]\n", "\n", " return clean(sentence), label" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can load the dataset and extract the *training* samples from it. Since we're impatient, we can do it in parallel using the [`simple_parallel_map`](/reference/utilities/#great_ai.utilities.simple_parallel_map) function.\n", "\n", "> Open files in Python are iterable, with a line being return (in text mode) with each iteration." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 84000/84000 [00:09<00:00, 8960.63it/s] \n" ] } ], "source": [ "from great_ai.utilities import simple_parallel_map\n", "\n", "with open(\"data/train.txt\", encoding=\"utf-8\") as f:\n", " training_data = simple_parallel_map(preprocess, f)\n", "\n", "X_train = [d[0] for d in training_data]\n", "y_train = [d[1] for d in training_data]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's do the same for the *test* data." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 22399/22399 [00:03<00:00, 6078.02it/s]\n" ] } ], "source": [ "with open(\"data/test.txt\", encoding=\"utf-8\") as f:\n", " test_data = simple_parallel_map(preprocess, f)\n", "\n", "X_test = [d[0] for d in test_data]\n", "y_test = [d[1] for d in test_data]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After obtaining some clean data, it's time to create and train a model on it. We are going to use [scikit-learn](https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html) for this purpose.\n", "\n", "In this case, we opted for a [linear Support Vector Machine](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) because it's known to be an adequate baseline for simple classification tasks." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),\n",
" ('linearsvc', LinearSVC())])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(steps=[('tfidfvectorizer', TfidfVectorizer()),\n",
" ('linearsvc', LinearSVC())])TfidfVectorizer()
LinearSVC()