{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
    "> Part 1: obtain and clean data\n",
    "\n",
    "![position of this step in the lifecycle](diagrams/scope-data.svg)\n",
    "> The blue boxes show the steps implemented in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract\n",
    "\n",
    "This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
    "\n",
    "In this case, we download the semantic scholar dataset from a public S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_CHUNK_COUNT = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Processing 1 out of the 6002 available chunks'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "from random import shuffle\n",
    "\n",
    "manifest = (\n",
    "    urllib.request.urlopen(\n",
    "        \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n",
    "    )\n",
    "    .read()\n",
    "    .decode()\n",
    ")  # a list of available chunks separated by '\\n' characters\n",
    "\n",
    "lines = manifest.split()\n",
    "shuffle(lines)\n",
    "chunks = lines[:MAX_CHUNK_COUNT]\n",
    "\n",
    "f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform\n",
    "\n",
    "- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
    "- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
    "- We will speed up processing using `great_ai.utilities.parallel_map`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226m2022-06-25 11:20:01,955 |  WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:20:01,956 |     INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
      "\u001b[38;5;226m2022-06-25 11:20:01,956 |  WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
      "100%|██████████| 1/1 [04:02<00:00, 242.61s/it]\n"
     ]
    }
   ],
   "source": [
    "from typing import List, Tuple\n",
    "import json\n",
    "import gzip\n",
    "from great_ai import parallel_map, clean, is_english, predict_language\n",
    "\n",
    "\n",
    "def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
    "    # Extract\n",
    "    response = urllib.request.urlopen(\n",
    "        f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n",
    "    )  # a gzipped JSON Lines file\n",
    "\n",
    "    decompressed = gzip.decompress(response.read())\n",
    "    decoded = decompressed.decode()\n",
    "    chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n",
    "\n",
    "    # Transform\n",
    "    return [\n",
    "        (\n",
    "            clean(\n",
    "                f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n",
    "                convert_to_ascii=True,\n",
    "            ),  # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n",
    "            c[\"fieldsOfStudy\"],\n",
    "        )  # Create pairs of `(text, [...domains])`\n",
    "        for c in chunk\n",
    "        if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n",
    "    ]\n",
    "\n",
    "\n",
    "preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import chain\n",
    "\n",
    "preprocessed_data = list(chain(*preprocessed_chunks))\n",
    "X, y = zip(\n",
    "    *preprocessed_data\n",
    ")  # X is the input, y is the expected (ground truth) output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load\n",
    "\n",
    "Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n",
    "\n",
    "#### Production-ready backend\n",
    "\n",
    "The MongoDB driver is automatically configured if  `mongo.ini` exists with the following scheme:\n",
    "\n",
    "```ini\n",
    "mongo_connection_string=mongodb://localhost:27017/\n",
    "mongo_database=my_great_ai_db\n",
    "```\n",
    "> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226m2022-06-25 11:24:04,668 |  WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,669 |     INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising MongodbDriver\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,670 |     INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,671 |     INFO | Settings: configured ✅\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,672 |     INFO | 🔩 tracing_database: MongodbDriver\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,672 |     INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,673 |     INFO | 🔩 is_production: False\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,673 |     INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
      "\u001b[38;5;39m2022-06-25 11:24:04,674 |     INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
      "\u001b[38;5;226m2022-06-25 11:24:04,674 |  WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
      "\u001b[38;5;226m2022-06-25 11:24:04,674 |  WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from great_ai import add_ground_truth\n",
    "\n",
    "add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next: [Part 2](train.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 ('.env': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}