231 lines
6.8 KiB
Text
231 lines
6.8 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
|
"> Part 1: obtain and clean data\n",
|
|
"\n",
|
|
"\n",
|
|
"> The blue boxes show the steps implemented in this notebook."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"MAX_CHUNK_COUNT = 4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Extract\n",
|
|
"\n",
|
|
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
|
"\n",
|
|
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Processing 4 out of the 6002 available chunks'"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import urllib.request\n",
|
|
"\n",
|
|
"manifest = (\n",
|
|
" urllib.request.urlopen(\n",
|
|
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n",
|
|
" )\n",
|
|
" .read()\n",
|
|
" .decode()\n",
|
|
") # a list of available chunks separated by '\\n' characters\n",
|
|
"\n",
|
|
"chunks = manifest.split()[:MAX_CHUNK_COUNT]\n",
|
|
"\n",
|
|
"f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Transform\n",
|
|
"\n",
|
|
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
|
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
|
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226m2022-06-19 14:59:12,562 | WARNING | Limiting concurrency to 4 because there are only 4 chunks\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-19 14:59:12,563 | INFO | Starting parallel map (concurrency: 4, chunk size: 1)\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "ff8fc113515944cfa75127f4aba3d491",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/4 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"from typing import List, Tuple\n",
|
|
"import json\n",
|
|
"import gzip\n",
|
|
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
|
"\n",
|
|
"\n",
|
|
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
|
" # Extract\n",
|
|
" response = urllib.request.urlopen(\n",
|
|
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n",
|
|
" ) # a gzipped JSON Lines file\n",
|
|
"\n",
|
|
" decompressed = gzip.decompress(response.read())\n",
|
|
" decoded = decompressed.decode()\n",
|
|
" chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n",
|
|
"\n",
|
|
" # Transform\n",
|
|
" return [\n",
|
|
" # Create pairs of `(text, [...domains])`\n",
|
|
" # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n",
|
|
" (\n",
|
|
" clean(\n",
|
|
" f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n",
|
|
" convert_to_ascii=True,\n",
|
|
" ),\n",
|
|
" c[\"fieldsOfStudy\"],\n",
|
|
" )\n",
|
|
" for c in chunk\n",
|
|
" if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n",
|
|
" ]\n",
|
|
"\n",
|
|
"\n",
|
|
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from itertools import chain\n",
|
|
"\n",
|
|
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
|
"X, y = zip(\n",
|
|
" *preprocessed_data\n",
|
|
") # X is the input, y is the expected (ground truth) output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load\n",
|
|
"\n",
|
|
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n",
|
|
"\n",
|
|
"#### Use a different repository\n",
|
|
"\n",
|
|
"For the sake of simplicity, the tutorial uses the local hard drive (`great_ai.ParallelTinyDbDriver`) as the central repository.\n",
|
|
"This can be simply changed, for example, by the following snippet:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"from great_ai import configure, MongoDbDriver\n",
|
|
"\n",
|
|
"configure(tracing_database=MongoDbDriver('mongodb://localhost:27017_or_something_like_that'))\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[38;5;226m2022-06-19 15:03:30,300 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
|
"\u001b[38;5;226m2022-06-19 15:03:30,301 | WARNING | The selected persistence driver (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
|
"\u001b[38;5;39m2022-06-19 15:03:30,301 | INFO | Options: configured ✅\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from great_ai import add_ground_truth\n",
|
|
"\n",
|
|
"add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Next: [Part 2](train.ipynb)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3.10.4 ('.env': venv)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
},
|
|
"orig_nbformat": 4,
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|