Deployed b1a66cb with MkDocs version: 1.3.0
This commit is contained in:
parent
0683061983
commit
7305d2ead3
38 changed files with 3566 additions and 2820 deletions
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -694,6 +721,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="train.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -4,17 +4,20 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
||||
"> Part 1: obtain and clean data\n",
|
||||
"# Simple example: data engineering\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps implemented in this notebook.\n",
|
||||
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
||||
"\n",
|
||||
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
|
||||
"\n",
|
||||
"### Extract\n",
|
||||
"\n",
|
||||
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
||||
"\n",
|
||||
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
||||
"In this example, we download the semantic scholar dataset from a public S3 bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -23,7 +26,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MAX_CHUNK_COUNT = 1"
|
||||
"MAX_CHUNK_COUNT = 4"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -34,7 +37,7 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Processing 1 out of the 6002 available chunks'"
|
||||
"'Processing 4 out of the 6002 available chunks'"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
|
|
@ -69,7 +72,7 @@
|
|||
"\n",
|
||||
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
||||
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
||||
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
||||
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -77,56 +80,11 @@
|
|||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Spacy model en_core_web_sm not found locally, downloading...\n",
|
||||
"Collecting en-core-web-sm==3.3.0\n",
|
||||
" Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)\n",
|
||||
" ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00\n",
|
||||
"Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)\n",
|
||||
"Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)\n",
|
||||
"Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)\n",
|
||||
"Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)\n",
|
||||
"Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)\n",
|
||||
"Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)\n",
|
||||
"Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)\n",
|
||||
"Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)\n",
|
||||
"Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)\n",
|
||||
"Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)\n",
|
||||
"Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)\n",
|
||||
"Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)\n",
|
||||
"Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)\n",
|
||||
"Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)\n",
|
||||
"Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)\n",
|
||||
"Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)\n",
|
||||
"Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)\n",
|
||||
"Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)\n",
|
||||
"Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)\n",
|
||||
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)\n",
|
||||
"Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)\n",
|
||||
"Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)\n",
|
||||
"Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)\n",
|
||||
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)\n",
|
||||
"Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)\n",
|
||||
"Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)\n",
|
||||
"Installing collected packages: en-core-web-sm\n",
|
||||
"Successfully installed en-core-web-sm-3.3.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
|
||||
"100%|██████████| 1/1 [03:26<00:00, 206.86s/it]\n"
|
||||
"100%|██████████| 4/4 [04:42<00:00, 70.62s/it] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -134,7 +92,13 @@
|
|||
"from typing import List, Tuple\n",
|
||||
"import json\n",
|
||||
"import gzip\n",
|
||||
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
||||
"from great_ai.utilities import (\n",
|
||||
" simple_parallel_map,\n",
|
||||
" clean,\n",
|
||||
" is_english,\n",
|
||||
" predict_language,\n",
|
||||
" unchunk,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
||||
|
|
@ -161,7 +125,9 @@
|
|||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
||||
"preprocessed_data = unchunk(\n",
|
||||
" simple_parallel_map(preprocess_chunk, chunks, concurrency=4)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -170,12 +136,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import chain\n",
|
||||
"\n",
|
||||
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
||||
"X, y = zip(\n",
|
||||
" *preprocessed_data\n",
|
||||
") # X is the input, y is the expected (ground truth) output"
|
||||
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -194,33 +155,16 @@
|
|||
"mongo_connection_string=mongodb://localhost:27017/\n",
|
||||
"mongo_database=my_great_ai_db\n",
|
||||
"```\n",
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
|
||||
"\n",
|
||||
"Otherwise, TinyDB is used which is just a local JSON file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,992 | INFO | Settings: configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from great_ai import add_ground_truth\n",
|
||||
"\n",
|
||||
|
|
@ -231,7 +175,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Next: [Part 2](train.ipynb)"
|
||||
"### Next: [Part 2](/examples/simple/train)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -256,7 +200,7 @@
|
|||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "c1f394f9662881005685eeb18d8f9f77079b1b8b9a5ece1f825bfa01fcb7f52f"
|
||||
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
|
||||
|
||||
|
||||
<title>Train a domain classifier on the semantic scholar dataset - GreatAI documentation</title>
|
||||
<title>Simple example: data engineering - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -70,7 +83,7 @@
|
|||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#train-a-domain-classifier-on-the-semantic-scholar-dataset" class="md-skip">
|
||||
<a href="#simple-example-data-engineering" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
|
|
@ -103,7 +116,7 @@
|
|||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -614,12 +641,12 @@
|
|||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
|
||||
|
||||
|
|
@ -694,7 +721,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -708,7 +735,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -822,6 +849,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="data.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -1871,13 +1899,13 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h1 id="train-a-domain-classifier-on-the-semantic-scholar-dataset">Train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a><a class="anchor-link" href="#train-a-domain-classifier-on-the-semantic-scholar-dataset">¶</a></h1><blockquote><p>Part 1: obtain and clean data</p>
|
||||
</blockquote>
|
||||
<p><img src="../diagrams/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps implemented in this notebook.</p>
|
||||
<h1 id="simple-example-data-engineering">Simple example: data engineering<a class="anchor-link" href="#simple-example-data-engineering">¶</a></h1><p>Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a> by taking full advantage of <code>great-ai</code>. Subsequently, we create a production-ready deployment.</p>
|
||||
<p><img src="/media/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.</p>
|
||||
</blockquote>
|
||||
<p>Since the true scope of <code>great-ai</code> is the phase between proof-of-concept code and production-ready service, it is predominantly used in the <a href="/examples/simple/deploy">deployment notebook</a>. Feel free to skip there, or continue reading if you'd like to see the full picture.</p>
|
||||
<h3 id="extract">Extract<a class="anchor-link" href="#extract">¶</a></h3><p>This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.</p>
|
||||
<p>In this case, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
<p>In this example, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1903,9 +1931,9 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">1</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">4</span>
|
||||
</pre></div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 1</div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 4</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1991,7 +2019,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output jp-OutputArea-executeResult" data-mime-type="text/plain">
|
||||
<pre>'Processing 1 out of the 6002 available chunks'</pre>
|
||||
<pre>'Processing 4 out of the 6002 available chunks'</pre>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
|
@ -2011,7 +2039,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<h3 id="transform">Transform<a class="anchor-link" href="#transform">¶</a></h3><ul>
|
||||
<li>Filter out non-English abstracts using <code>great_ai.utilities.predict_language</code></li>
|
||||
<li>Project it to only keep the necessary components (text and labels), clean the textual content using <code>great_ai.utilities.clean</code></li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.parallel_map</code>.</li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.simple_parallel_map</code>.</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
|
@ -2041,7 +2069,13 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Tuple</span>
|
||||
<span class="kn">import</span> <span class="nn">json</span>
|
||||
<span class="kn">import</span> <span class="nn">gzip</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">parallel_map</span><span class="p">,</span> <span class="n">clean</span><span class="p">,</span> <span class="n">is_english</span><span class="p">,</span> <span class="n">predict_language</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai.utilities</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">,</span>
|
||||
<span class="n">clean</span><span class="p">,</span>
|
||||
<span class="n">is_english</span><span class="p">,</span>
|
||||
<span class="n">predict_language</span><span class="p">,</span>
|
||||
<span class="n">unchunk</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">preprocess_chunk</span><span class="p">(</span><span class="n">chunk_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]]:</span>
|
||||
|
|
@ -2068,12 +2102,20 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<span class="p">]</span>
|
||||
|
||||
|
||||
<span class="n">preprocessed_chunks</span> <span class="o">=</span> <span class="n">parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">)</span>
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="n">unchunk</span><span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">concurrency</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
|
||||
<span class="p">)</span>
|
||||
</pre></div>
|
||||
<div id="cell-3" class="clipboard-copy-txt">from typing import List, Tuple
|
||||
import json
|
||||
import gzip
|
||||
from great_ai import parallel_map, clean, is_english, predict_language
|
||||
from great_ai.utilities import (
|
||||
simple_parallel_map,
|
||||
clean,
|
||||
is_english,
|
||||
predict_language,
|
||||
unchunk,
|
||||
)
|
||||
|
||||
|
||||
def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
||||
|
|
@ -2100,7 +2142,9 @@ def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
|||
]
|
||||
|
||||
|
||||
preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
||||
preprocessed_data = unchunk(
|
||||
simple_parallel_map(preprocess_chunk, chunks, concurrency=4)
|
||||
)</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2121,58 +2165,8 @@ preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
|||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="text/plain">
|
||||
<pre>Spacy model en_core_web_sm not found locally, downloading...
|
||||
Collecting en-core-web-sm==3.3.0
|
||||
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00
|
||||
Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)
|
||||
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)
|
||||
Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)
|
||||
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)
|
||||
Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)
|
||||
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)
|
||||
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)
|
||||
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)
|
||||
Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)
|
||||
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)
|
||||
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)
|
||||
Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)
|
||||
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)
|
||||
Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)
|
||||
Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)
|
||||
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)
|
||||
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)
|
||||
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)
|
||||
Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)
|
||||
Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)
|
||||
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)
|
||||
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)
|
||||
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)
|
||||
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)
|
||||
Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)
|
||||
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)
|
||||
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)
|
||||
Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)
|
||||
Installing collected packages: en-core-web-sm
|
||||
Successfully installed en-core-web-sm-3.3.0
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism</span>
|
||||
100%|██████████| 1/1 [03:26<00:00, 206.86s/it]
|
||||
<pre>100%|██████████| 4/4 [04:42<00:00, 70.62s/it]
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2203,19 +2197,9 @@ Successfully installed en-core-web-sm-3.3.0
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">chain</span>
|
||||
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">chain</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_chunks</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span>
|
||||
<span class="o">*</span><span class="n">preprocessed_data</span>
|
||||
<span class="p">)</span> <span class="c1"># X is the input, y is the expected (ground truth) output</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_data</span><span class="p">)</span> <span class="c1"># X is the input, y is the expected output</span>
|
||||
</pre></div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">from itertools import chain
|
||||
|
||||
preprocessed_data = list(chain(*preprocessed_chunks))
|
||||
X, y = zip(
|
||||
*preprocessed_data
|
||||
) # X is the input, y is the expected (ground truth) output</div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">X, y = zip(*preprocessed_data) # X is the input, y is the expected output</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2238,17 +2222,18 @@ X, y = zip(
|
|||
</pre></div>
|
||||
<blockquote><p>You can install MongoDB from <a href="https://www.mongodb.com/docs/manual/installation">here</a> or <a href="https://www.mongodb.com/cloud/atlas/register">use it as a service</a></p>
|
||||
</blockquote>
|
||||
<p>Otherwise, TinyDB is used which is just a local JSON file.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [5]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [8]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
|
||||
|
||||
<div class="CodeMirror cm-s-jupyter">
|
||||
|
|
@ -2277,40 +2262,6 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-Cell-outputWrapper">
|
||||
<div class="jp-Collapser jp-OutputCollapser jp-Cell-outputCollapser">
|
||||
</div>
|
||||
|
||||
|
||||
<div class="jp-OutputArea jp-Cell-outputArea">
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,992 | INFO | Settings: configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
|
|
@ -2319,7 +2270,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h3 id="next-part-2">Next: <a href="train.ipynb">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
<h3 id="next-part-2">Next: <a href="/examples/simple/train">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2333,7 +2284,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -2370,13 +2321,13 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
|
||||
|
||||
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Train a domain classifier on the semantic scholar dataset" rel="next">
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Optimise and train a model" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Loading…
Add table
Add a link
Reference in a new issue