227 lines
7.7 KiB
Text
227 lines
7.7 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
|
"> Part 1: obtain and clean data\n",
|
|
"\n",
|
|
"\n",
|
|
"> The blue boxes show the steps implemented in this notebook."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Extract\n",
|
|
"\n",
|
|
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
|
"\n",
|
|
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"MAX_CHUNK_COUNT = 1"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Processing 1 out of the 6002 available chunks'"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import urllib.request\n",
|
|
"from random import shuffle\n",
|
|
"\n",
|
|
"manifest = (\n",
|
|
" urllib.request.urlopen(\n",
|
|
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n",
|
|
" )\n",
|
|
" .read()\n",
|
|
" .decode()\n",
|
|
") # a list of available chunks separated by '\\n' characters\n",
|
|
"\n",
|
|
"lines = manifest.split()\n",
|
|
"shuffle(lines)\n",
|
|
"chunks = lines[:MAX_CHUNK_COUNT]\n",
|
|
"\n",
|
|
"f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Transform\n",
|
|
"\n",
|
|
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
|
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
|
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226m2022-06-25 11:20:01,955 | WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:20:01,956 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
|
|
"\u001b[38;5;226m2022-06-25 11:20:01,956 | WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
|
|
"100%|██████████| 1/1 [04:02<00:00, 242.61s/it]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from typing import List, Tuple\n",
|
|
"import json\n",
|
|
"import gzip\n",
|
|
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
|
"\n",
|
|
"\n",
|
|
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
|
" # Extract\n",
|
|
" response = urllib.request.urlopen(\n",
|
|
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n",
|
|
" ) # a gzipped JSON Lines file\n",
|
|
"\n",
|
|
" decompressed = gzip.decompress(response.read())\n",
|
|
" decoded = decompressed.decode()\n",
|
|
" chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n",
|
|
"\n",
|
|
" # Transform\n",
|
|
" return [\n",
|
|
" (\n",
|
|
" clean(\n",
|
|
" f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n",
|
|
" convert_to_ascii=True,\n",
|
|
" ), # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n",
|
|
" c[\"fieldsOfStudy\"],\n",
|
|
" ) # Create pairs of `(text, [...domains])`\n",
|
|
" for c in chunk\n",
|
|
" if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n",
|
|
" ]\n",
|
|
"\n",
|
|
"\n",
|
|
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from itertools import chain\n",
|
|
"\n",
|
|
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
|
"X, y = zip(\n",
|
|
" *preprocessed_data\n",
|
|
") # X is the input, y is the expected (ground truth) output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load\n",
|
|
"\n",
|
|
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n",
|
|
"\n",
|
|
"#### Production-ready backend\n",
|
|
"\n",
|
|
"The MongoDB driver is automatically configured if `mongo.ini` exists with the following scheme:\n",
|
|
"\n",
|
|
"```ini\n",
|
|
"mongo_connection_string=mongodb://localhost:27017/\n",
|
|
"mongo_database=my_great_ai_db\n",
|
|
"```\n",
|
|
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226m2022-06-25 11:24:04,668 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,669 | INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising MongodbDriver\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,670 | INFO | Found credentials file (/data/projects/great-ai/examples/simple/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,671 | INFO | Settings: configured ✅\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,672 | INFO | 🔩 tracing_database: MongodbDriver\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,672 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,673 | INFO | 🔩 is_production: False\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,673 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-25 11:24:04,674 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
|
"\u001b[38;5;226m2022-06-25 11:24:04,674 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
|
"\u001b[38;5;226m2022-06-25 11:24:04,674 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from great_ai import add_ground_truth\n",
|
|
"\n",
|
|
"add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Next: [Part 2](train.ipynb)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3.10.4 ('.env': venv)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
},
|
|
"orig_nbformat": 4,
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|