{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n", "> Part 1: obtain and clean data\n", "\n", "![position of this step in the lifecycle](diagrams/scope-data.svg)\n", "> The blue boxes show the steps implemented in this notebook." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "MAX_CHUNK_COUNT = 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract\n", "\n", "This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n", "\n", "In this case, we download the semantic scholar dataset from a public S3 bucket." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Processing 4 out of the 6002 available chunks'" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import urllib.request\n", "\n", "manifest = (\n", " urllib.request.urlopen(\n", " \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n", " )\n", " .read()\n", " .decode()\n", ") # a list of available chunks separated by '\\n' characters\n", "\n", "chunks = manifest.split()[:MAX_CHUNK_COUNT]\n", "\n", "f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Transform\n", "\n", "- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n", "- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n", "- We will speed up processing using `great_ai.utilities.parallel_map`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m2022-06-19 14:59:12,562 | WARNING | Limiting concurrency to 4 because there are only 4 chunks\u001b[0m\n", "\u001b[38;5;39m2022-06-19 14:59:12,563 | INFO | Starting parallel map (concurrency: 4, chunk size: 1)\u001b[0m\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ff8fc113515944cfa75127f4aba3d491", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4 [00:00 List[Tuple[str, List[str]]]:\n", " # Extract\n", " response = urllib.request.urlopen(\n", " f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n", " ) # a gzipped JSON Lines file\n", "\n", " decompressed = gzip.decompress(response.read())\n", " decoded = decompressed.decode()\n", " chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n", "\n", " # Transform\n", " return [\n", " # Create pairs of `(text, [...domains])`\n", " # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n", " (\n", " clean(\n", " f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n", " convert_to_ascii=True,\n", " ),\n", " c[\"fieldsOfStudy\"],\n", " )\n", " for c in chunk\n", " if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n", " ]\n", "\n", "\n", "preprocessed_chunks = parallel_map(preprocess_chunk, chunks)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from itertools import chain\n", "\n", "preprocessed_data = list(chain(*preprocessed_chunks))\n", "X, y = zip(\n", " *preprocessed_data\n", ") # X is the input, y is the expected (ground truth) output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load\n", "\n", "Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n", "\n", "#### Use a different repository\n", "\n", "For the sake of simplicity, the tutorial uses the local hard drive (`great_ai.ParallelTinyDbDriver`) as the central repository.\n", "This can be simply changed, for example, by the following snippet:\n", "\n", "```python\n", "from great_ai import configure, MongoDbDriver\n", "\n", "configure(tracing_database=MongoDbDriver('mongodb://localhost:27017_or_something_like_that'))\n", "```" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;5;226m2022-06-19 15:03:30,300 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n", "\u001b[38;5;226m2022-06-19 15:03:30,301 | WARNING | The selected persistence driver (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n", "\u001b[38;5;39m2022-06-19 15:03:30,301 | INFO | Options: configured ✅\u001b[0m\n" ] } ], "source": [ "from great_ai import add_ground_truth\n", "\n", "add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Next: [Part 2](train.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.4 ('.env': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad" } } }, "nbformat": 4, "nbformat_minor": 2 }