201 lines
6.4 KiB
Text
201 lines
6.4 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Simple example: data engineering\n",
|
|
"\n",
|
|
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
|
"\n",
|
|
"\n",
|
|
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
|
"\n",
|
|
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
|
|
"\n",
|
|
"### Extract\n",
|
|
"\n",
|
|
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
|
"\n",
|
|
"In this example, we download the semantic scholar dataset from a public S3 bucket."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"MAX_CHUNK_COUNT = 4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Processing 4 out of the 6002 available chunks'"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import urllib.request\n",
|
|
"from random import shuffle\n",
|
|
"\n",
|
|
"manifest = (\n",
|
|
" urllib.request.urlopen(\n",
|
|
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n",
|
|
" )\n",
|
|
" .read()\n",
|
|
" .decode()\n",
|
|
") # a list of available chunks separated by '\\n' characters\n",
|
|
"\n",
|
|
"lines = manifest.split()\n",
|
|
"shuffle(lines)\n",
|
|
"chunks = lines[:MAX_CHUNK_COUNT]\n",
|
|
"\n",
|
|
"f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Transform\n",
|
|
"\n",
|
|
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
|
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
|
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 4/4 [04:42<00:00, 70.62s/it] \n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from typing import List, Tuple\n",
|
|
"import json\n",
|
|
"import gzip\n",
|
|
"from great_ai.utilities import simple_parallel_map, clean, is_english, predict_language, unchunk\n",
|
|
"\n",
|
|
"\n",
|
|
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
|
" # Extract\n",
|
|
" response = urllib.request.urlopen(\n",
|
|
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n",
|
|
" ) # a gzipped JSON Lines file\n",
|
|
"\n",
|
|
" decompressed = gzip.decompress(response.read())\n",
|
|
" decoded = decompressed.decode()\n",
|
|
" chunk = [json.loads(line) for line in decoded.split(\"\\n\") if line]\n",
|
|
"\n",
|
|
" # Transform\n",
|
|
" return [\n",
|
|
" (\n",
|
|
" clean(\n",
|
|
" f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n",
|
|
" convert_to_ascii=True,\n",
|
|
" ), # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n",
|
|
" c[\"fieldsOfStudy\"],\n",
|
|
" ) # Create pairs of `(text, [...domains])`\n",
|
|
" for c in chunk\n",
|
|
" if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n",
|
|
" ]\n",
|
|
"\n",
|
|
"\n",
|
|
"preprocessed_data = unchunk(simple_parallel_map(preprocess_chunk, chunks, concurrency=4))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Load\n",
|
|
"\n",
|
|
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n",
|
|
"\n",
|
|
"#### Production-ready backend\n",
|
|
"\n",
|
|
"The MongoDB driver is automatically configured if `mongo.ini` exists with the following scheme:\n",
|
|
"\n",
|
|
"```ini\n",
|
|
"mongo_connection_string=mongodb://localhost:27017/\n",
|
|
"mongo_database=my_great_ai_db\n",
|
|
"```\n",
|
|
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
|
|
"\n",
|
|
"Otherwise, TinyDB is used which is just a local JSON file."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from great_ai import add_ground_truth\n",
|
|
"\n",
|
|
"add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Next: [Part 2](/examples/simple/train)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3.10.4 ('.env': venv)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
},
|
|
"orig_nbformat": 4,
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|