Deployed b1a66cb with MkDocs version: 1.3.0
This commit is contained in:
parent
0683061983
commit
7305d2ead3
38 changed files with 3566 additions and 2820 deletions
|
|
@ -4,17 +4,20 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
||||
"> Part 1: obtain and clean data\n",
|
||||
"# Simple example: data engineering\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps implemented in this notebook.\n",
|
||||
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
||||
"\n",
|
||||
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
|
||||
"\n",
|
||||
"### Extract\n",
|
||||
"\n",
|
||||
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
||||
"\n",
|
||||
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
||||
"In this example, we download the semantic scholar dataset from a public S3 bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -23,7 +26,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MAX_CHUNK_COUNT = 1"
|
||||
"MAX_CHUNK_COUNT = 4"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -34,7 +37,7 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Processing 1 out of the 6002 available chunks'"
|
||||
"'Processing 4 out of the 6002 available chunks'"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
|
|
@ -69,7 +72,7 @@
|
|||
"\n",
|
||||
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
||||
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
||||
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
||||
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -77,56 +80,11 @@
|
|||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Spacy model en_core_web_sm not found locally, downloading...\n",
|
||||
"Collecting en-core-web-sm==3.3.0\n",
|
||||
" Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)\n",
|
||||
" ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00\n",
|
||||
"Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)\n",
|
||||
"Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)\n",
|
||||
"Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)\n",
|
||||
"Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)\n",
|
||||
"Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)\n",
|
||||
"Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)\n",
|
||||
"Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)\n",
|
||||
"Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)\n",
|
||||
"Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)\n",
|
||||
"Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)\n",
|
||||
"Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)\n",
|
||||
"Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)\n",
|
||||
"Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)\n",
|
||||
"Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)\n",
|
||||
"Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)\n",
|
||||
"Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)\n",
|
||||
"Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)\n",
|
||||
"Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)\n",
|
||||
"Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)\n",
|
||||
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)\n",
|
||||
"Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)\n",
|
||||
"Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)\n",
|
||||
"Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)\n",
|
||||
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)\n",
|
||||
"Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)\n",
|
||||
"Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)\n",
|
||||
"Installing collected packages: en-core-web-sm\n",
|
||||
"Successfully installed en-core-web-sm-3.3.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
|
||||
"100%|██████████| 1/1 [03:26<00:00, 206.86s/it]\n"
|
||||
"100%|██████████| 4/4 [04:42<00:00, 70.62s/it] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -134,7 +92,13 @@
|
|||
"from typing import List, Tuple\n",
|
||||
"import json\n",
|
||||
"import gzip\n",
|
||||
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
||||
"from great_ai.utilities import (\n",
|
||||
" simple_parallel_map,\n",
|
||||
" clean,\n",
|
||||
" is_english,\n",
|
||||
" predict_language,\n",
|
||||
" unchunk,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
||||
|
|
@ -161,7 +125,9 @@
|
|||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
||||
"preprocessed_data = unchunk(\n",
|
||||
" simple_parallel_map(preprocess_chunk, chunks, concurrency=4)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -170,12 +136,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import chain\n",
|
||||
"\n",
|
||||
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
||||
"X, y = zip(\n",
|
||||
" *preprocessed_data\n",
|
||||
") # X is the input, y is the expected (ground truth) output"
|
||||
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -194,33 +155,16 @@
|
|||
"mongo_connection_string=mongodb://localhost:27017/\n",
|
||||
"mongo_database=my_great_ai_db\n",
|
||||
"```\n",
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
|
||||
"\n",
|
||||
"Otherwise, TinyDB is used which is just a local JSON file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,992 | INFO | Settings: configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from great_ai import add_ground_truth\n",
|
||||
"\n",
|
||||
|
|
@ -231,7 +175,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Next: [Part 2](train.ipynb)"
|
||||
"### Next: [Part 2](/examples/simple/train)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -256,7 +200,7 @@
|
|||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "c1f394f9662881005685eeb18d8f9f77079b1b8b9a5ece1f825bfa01fcb7f52f"
|
||||
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
|
||||
|
||||
|
||||
<title>Train a domain classifier on the semantic scholar dataset - GreatAI documentation</title>
|
||||
<title>Simple example: data engineering - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -70,7 +83,7 @@
|
|||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#train-a-domain-classifier-on-the-semantic-scholar-dataset" class="md-skip">
|
||||
<a href="#simple-example-data-engineering" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
|
|
@ -103,7 +116,7 @@
|
|||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -614,12 +641,12 @@
|
|||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
|
||||
|
||||
|
|
@ -694,7 +721,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -708,7 +735,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -822,6 +849,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="data.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -1871,13 +1899,13 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h1 id="train-a-domain-classifier-on-the-semantic-scholar-dataset">Train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a><a class="anchor-link" href="#train-a-domain-classifier-on-the-semantic-scholar-dataset">¶</a></h1><blockquote><p>Part 1: obtain and clean data</p>
|
||||
</blockquote>
|
||||
<p><img src="../diagrams/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps implemented in this notebook.</p>
|
||||
<h1 id="simple-example-data-engineering">Simple example: data engineering<a class="anchor-link" href="#simple-example-data-engineering">¶</a></h1><p>Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a> by taking full advantage of <code>great-ai</code>. Subsequently, we create a production-ready deployment.</p>
|
||||
<p><img src="/media/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.</p>
|
||||
</blockquote>
|
||||
<p>Since the true scope of <code>great-ai</code> is the phase between proof-of-concept code and production-ready service, it is predominantly used in the <a href="/examples/simple/deploy">deployment notebook</a>. Feel free to skip there, or continue reading if you'd like to see the full picture.</p>
|
||||
<h3 id="extract">Extract<a class="anchor-link" href="#extract">¶</a></h3><p>This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.</p>
|
||||
<p>In this case, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
<p>In this example, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1903,9 +1931,9 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">1</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">4</span>
|
||||
</pre></div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 1</div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 4</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1991,7 +2019,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output jp-OutputArea-executeResult" data-mime-type="text/plain">
|
||||
<pre>'Processing 1 out of the 6002 available chunks'</pre>
|
||||
<pre>'Processing 4 out of the 6002 available chunks'</pre>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
|
@ -2011,7 +2039,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<h3 id="transform">Transform<a class="anchor-link" href="#transform">¶</a></h3><ul>
|
||||
<li>Filter out non-English abstracts using <code>great_ai.utilities.predict_language</code></li>
|
||||
<li>Project it to only keep the necessary components (text and labels), clean the textual content using <code>great_ai.utilities.clean</code></li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.parallel_map</code>.</li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.simple_parallel_map</code>.</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
|
@ -2041,7 +2069,13 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Tuple</span>
|
||||
<span class="kn">import</span> <span class="nn">json</span>
|
||||
<span class="kn">import</span> <span class="nn">gzip</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">parallel_map</span><span class="p">,</span> <span class="n">clean</span><span class="p">,</span> <span class="n">is_english</span><span class="p">,</span> <span class="n">predict_language</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai.utilities</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">,</span>
|
||||
<span class="n">clean</span><span class="p">,</span>
|
||||
<span class="n">is_english</span><span class="p">,</span>
|
||||
<span class="n">predict_language</span><span class="p">,</span>
|
||||
<span class="n">unchunk</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">preprocess_chunk</span><span class="p">(</span><span class="n">chunk_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]]:</span>
|
||||
|
|
@ -2068,12 +2102,20 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<span class="p">]</span>
|
||||
|
||||
|
||||
<span class="n">preprocessed_chunks</span> <span class="o">=</span> <span class="n">parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">)</span>
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="n">unchunk</span><span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">concurrency</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
|
||||
<span class="p">)</span>
|
||||
</pre></div>
|
||||
<div id="cell-3" class="clipboard-copy-txt">from typing import List, Tuple
|
||||
import json
|
||||
import gzip
|
||||
from great_ai import parallel_map, clean, is_english, predict_language
|
||||
from great_ai.utilities import (
|
||||
simple_parallel_map,
|
||||
clean,
|
||||
is_english,
|
||||
predict_language,
|
||||
unchunk,
|
||||
)
|
||||
|
||||
|
||||
def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
||||
|
|
@ -2100,7 +2142,9 @@ def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
|||
]
|
||||
|
||||
|
||||
preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
||||
preprocessed_data = unchunk(
|
||||
simple_parallel_map(preprocess_chunk, chunks, concurrency=4)
|
||||
)</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2121,58 +2165,8 @@ preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
|||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="text/plain">
|
||||
<pre>Spacy model en_core_web_sm not found locally, downloading...
|
||||
Collecting en-core-web-sm==3.3.0
|
||||
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00
|
||||
Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)
|
||||
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)
|
||||
Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)
|
||||
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)
|
||||
Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)
|
||||
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)
|
||||
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)
|
||||
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)
|
||||
Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)
|
||||
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)
|
||||
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)
|
||||
Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)
|
||||
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)
|
||||
Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)
|
||||
Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)
|
||||
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)
|
||||
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)
|
||||
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)
|
||||
Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)
|
||||
Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)
|
||||
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)
|
||||
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)
|
||||
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)
|
||||
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)
|
||||
Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)
|
||||
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)
|
||||
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)
|
||||
Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)
|
||||
Installing collected packages: en-core-web-sm
|
||||
Successfully installed en-core-web-sm-3.3.0
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism</span>
|
||||
100%|██████████| 1/1 [03:26<00:00, 206.86s/it]
|
||||
<pre>100%|██████████| 4/4 [04:42<00:00, 70.62s/it]
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2203,19 +2197,9 @@ Successfully installed en-core-web-sm-3.3.0
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">chain</span>
|
||||
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">chain</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_chunks</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span>
|
||||
<span class="o">*</span><span class="n">preprocessed_data</span>
|
||||
<span class="p">)</span> <span class="c1"># X is the input, y is the expected (ground truth) output</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_data</span><span class="p">)</span> <span class="c1"># X is the input, y is the expected output</span>
|
||||
</pre></div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">from itertools import chain
|
||||
|
||||
preprocessed_data = list(chain(*preprocessed_chunks))
|
||||
X, y = zip(
|
||||
*preprocessed_data
|
||||
) # X is the input, y is the expected (ground truth) output</div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">X, y = zip(*preprocessed_data) # X is the input, y is the expected output</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2238,17 +2222,18 @@ X, y = zip(
|
|||
</pre></div>
|
||||
<blockquote><p>You can install MongoDB from <a href="https://www.mongodb.com/docs/manual/installation">here</a> or <a href="https://www.mongodb.com/cloud/atlas/register">use it as a service</a></p>
|
||||
</blockquote>
|
||||
<p>Otherwise, TinyDB is used which is just a local JSON file.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [5]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [8]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
|
||||
|
||||
<div class="CodeMirror cm-s-jupyter">
|
||||
|
|
@ -2277,40 +2262,6 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-Cell-outputWrapper">
|
||||
<div class="jp-Collapser jp-OutputCollapser jp-Cell-outputCollapser">
|
||||
</div>
|
||||
|
||||
|
||||
<div class="jp-OutputArea jp-Cell-outputArea">
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,992 | INFO | Settings: configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
|
|
@ -2319,7 +2270,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h3 id="next-part-2">Next: <a href="train.ipynb">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
<h3 id="next-part-2">Next: <a href="/examples/simple/train">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2333,7 +2284,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -2370,13 +2321,13 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
|
||||
|
||||
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Train a domain classifier on the semantic scholar dataset" rel="next">
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Optimise and train a model" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue