Deployed b1a66cb with MkDocs version: 1.3.0
This commit is contained in:
parent
0683061983
commit
7305d2ead3
38 changed files with 3566 additions and 2820 deletions
33
404.html
33
404.html
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -346,6 +359,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -596,7 +623,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -610,7 +637,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -624,7 +651,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="/examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -694,6 +721,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="train.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -4,17 +4,20 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus)\n",
|
||||
"> Part 1: obtain and clean data\n",
|
||||
"# Simple example: data engineering\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps implemented in this notebook.\n",
|
||||
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
||||
"\n",
|
||||
"Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the [deployment notebook](/examples/simple/deploy). Feel free to skip there, or continue reading if you'd like to see the full picture.\n",
|
||||
"\n",
|
||||
"### Extract\n",
|
||||
"\n",
|
||||
"This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.\n",
|
||||
"\n",
|
||||
"In this case, we download the semantic scholar dataset from a public S3 bucket."
|
||||
"In this example, we download the semantic scholar dataset from a public S3 bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -23,7 +26,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MAX_CHUNK_COUNT = 1"
|
||||
"MAX_CHUNK_COUNT = 4"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -34,7 +37,7 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'Processing 1 out of the 6002 available chunks'"
|
||||
"'Processing 4 out of the 6002 available chunks'"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
|
|
@ -69,7 +72,7 @@
|
|||
"\n",
|
||||
"- Filter out non-English abstracts using `great_ai.utilities.predict_language`\n",
|
||||
"- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`\n",
|
||||
"- We will speed up processing using `great_ai.utilities.parallel_map`."
|
||||
"- We will speed up processing using `great_ai.utilities.simple_parallel_map`."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -77,56 +80,11 @@
|
|||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Spacy model en_core_web_sm not found locally, downloading...\n",
|
||||
"Collecting en-core-web-sm==3.3.0\n",
|
||||
" Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)\n",
|
||||
" ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00\n",
|
||||
"Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)\n",
|
||||
"Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)\n",
|
||||
"Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)\n",
|
||||
"Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)\n",
|
||||
"Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)\n",
|
||||
"Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)\n",
|
||||
"Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)\n",
|
||||
"Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)\n",
|
||||
"Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)\n",
|
||||
"Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)\n",
|
||||
"Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)\n",
|
||||
"Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)\n",
|
||||
"Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)\n",
|
||||
"Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)\n",
|
||||
"Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)\n",
|
||||
"Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)\n",
|
||||
"Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)\n",
|
||||
"Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)\n",
|
||||
"Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)\n",
|
||||
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)\n",
|
||||
"Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)\n",
|
||||
"Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)\n",
|
||||
"Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)\n",
|
||||
"Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)\n",
|
||||
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)\n",
|
||||
"Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)\n",
|
||||
"Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)\n",
|
||||
"Installing collected packages: en-core-web-sm\n",
|
||||
"Successfully installed en-core-web-sm-3.3.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism\u001b[0m\n",
|
||||
"100%|██████████| 1/1 [03:26<00:00, 206.86s/it]\n"
|
||||
"100%|██████████| 4/4 [04:42<00:00, 70.62s/it] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -134,7 +92,13 @@
|
|||
"from typing import List, Tuple\n",
|
||||
"import json\n",
|
||||
"import gzip\n",
|
||||
"from great_ai import parallel_map, clean, is_english, predict_language\n",
|
||||
"from great_ai.utilities import (\n",
|
||||
" simple_parallel_map,\n",
|
||||
" clean,\n",
|
||||
" is_english,\n",
|
||||
" predict_language,\n",
|
||||
" unchunk,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
|
||||
|
|
@ -161,7 +125,9 @@
|
|||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"preprocessed_chunks = parallel_map(preprocess_chunk, chunks)"
|
||||
"preprocessed_data = unchunk(\n",
|
||||
" simple_parallel_map(preprocess_chunk, chunks, concurrency=4)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -170,12 +136,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from itertools import chain\n",
|
||||
"\n",
|
||||
"preprocessed_data = list(chain(*preprocessed_chunks))\n",
|
||||
"X, y = zip(\n",
|
||||
" *preprocessed_data\n",
|
||||
") # X is the input, y is the expected (ground truth) output"
|
||||
"X, y = zip(*preprocessed_data) # X is the input, y is the expected output"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -194,33 +155,16 @@
|
|||
"mongo_connection_string=mongodb://localhost:27017/\n",
|
||||
"mongo_database=my_great_ai_db\n",
|
||||
"```\n",
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n"
|
||||
"> You can install MongoDB from [here](https://www.mongodb.com/docs/manual/installation) or [use it as a service](https://www.mongodb.com/cloud/atlas/register)\n",
|
||||
"\n",
|
||||
"Otherwise, TinyDB is used which is just a local JSON file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,992 | INFO | Settings: configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from great_ai import add_ground_truth\n",
|
||||
"\n",
|
||||
|
|
@ -231,7 +175,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Next: [Part 2](train.ipynb)"
|
||||
"### Next: [Part 2](/examples/simple/train)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -256,7 +200,7 @@
|
|||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "c1f394f9662881005685eeb18d8f9f77079b1b8b9a5ece1f825bfa01fcb7f52f"
|
||||
"hash": "02dd6d3afbfa9fbbe1037d64ad9014965528a1ccad21929d6e72f466389a68ad"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
|
||||
|
||||
|
||||
<title>Train a domain classifier on the semantic scholar dataset - GreatAI documentation</title>
|
||||
<title>Simple example: data engineering - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -70,7 +83,7 @@
|
|||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#train-a-domain-classifier-on-the-semantic-scholar-dataset" class="md-skip">
|
||||
<a href="#simple-example-data-engineering" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
|
|
@ -103,7 +116,7 @@
|
|||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -614,12 +641,12 @@
|
|||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
|
||||
|
||||
|
|
@ -694,7 +721,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -708,7 +735,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -822,6 +849,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="data.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -1871,13 +1899,13 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h1 id="train-a-domain-classifier-on-the-semantic-scholar-dataset">Train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a><a class="anchor-link" href="#train-a-domain-classifier-on-the-semantic-scholar-dataset">¶</a></h1><blockquote><p>Part 1: obtain and clean data</p>
|
||||
</blockquote>
|
||||
<p><img src="../diagrams/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps implemented in this notebook.</p>
|
||||
<h1 id="simple-example-data-engineering">Simple example: data engineering<a class="anchor-link" href="#simple-example-data-engineering">¶</a></h1><p>Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the <a href="https://api.semanticscholar.org/corpus">semantic scholar dataset</a> by taking full advantage of <code>great-ai</code>. Subsequently, we create a production-ready deployment.</p>
|
||||
<p><img src="/media/scope-data.svg" alt="position of this step in the lifecycle"></p>
|
||||
<blockquote><p>The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.</p>
|
||||
</blockquote>
|
||||
<p>Since the true scope of <code>great-ai</code> is the phase between proof-of-concept code and production-ready service, it is predominantly used in the <a href="/examples/simple/deploy">deployment notebook</a>. Feel free to skip there, or continue reading if you'd like to see the full picture.</p>
|
||||
<h3 id="extract">Extract<a class="anchor-link" href="#extract">¶</a></h3><p>This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.</p>
|
||||
<p>In this case, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
<p>In this example, we download the semantic scholar dataset from a public S3 bucket.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1903,9 +1931,9 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">1</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">MAX_CHUNK_COUNT</span> <span class="o">=</span> <span class="mi">4</span>
|
||||
</pre></div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 1</div>
|
||||
<div id="cell-1" class="clipboard-copy-txt">MAX_CHUNK_COUNT = 4</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1991,7 +2019,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output jp-OutputArea-executeResult" data-mime-type="text/plain">
|
||||
<pre>'Processing 1 out of the 6002 available chunks'</pre>
|
||||
<pre>'Processing 4 out of the 6002 available chunks'</pre>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
|
@ -2011,7 +2039,7 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<h3 id="transform">Transform<a class="anchor-link" href="#transform">¶</a></h3><ul>
|
||||
<li>Filter out non-English abstracts using <code>great_ai.utilities.predict_language</code></li>
|
||||
<li>Project it to only keep the necessary components (text and labels), clean the textual content using <code>great_ai.utilities.clean</code></li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.parallel_map</code>.</li>
|
||||
<li>We will speed up processing using <code>great_ai.utilities.simple_parallel_map</code>.</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
|
@ -2041,7 +2069,13 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Tuple</span>
|
||||
<span class="kn">import</span> <span class="nn">json</span>
|
||||
<span class="kn">import</span> <span class="nn">gzip</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">parallel_map</span><span class="p">,</span> <span class="n">clean</span><span class="p">,</span> <span class="n">is_english</span><span class="p">,</span> <span class="n">predict_language</span>
|
||||
<span class="kn">from</span> <span class="nn">great_ai.utilities</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">,</span>
|
||||
<span class="n">clean</span><span class="p">,</span>
|
||||
<span class="n">is_english</span><span class="p">,</span>
|
||||
<span class="n">predict_language</span><span class="p">,</span>
|
||||
<span class="n">unchunk</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">preprocess_chunk</span><span class="p">(</span><span class="n">chunk_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]]:</span>
|
||||
|
|
@ -2068,12 +2102,20 @@ f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"<
|
|||
<span class="p">]</span>
|
||||
|
||||
|
||||
<span class="n">preprocessed_chunks</span> <span class="o">=</span> <span class="n">parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">)</span>
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="n">unchunk</span><span class="p">(</span>
|
||||
<span class="n">simple_parallel_map</span><span class="p">(</span><span class="n">preprocess_chunk</span><span class="p">,</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">concurrency</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
|
||||
<span class="p">)</span>
|
||||
</pre></div>
|
||||
<div id="cell-3" class="clipboard-copy-txt">from typing import List, Tuple
|
||||
import json
|
||||
import gzip
|
||||
from great_ai import parallel_map, clean, is_english, predict_language
|
||||
from great_ai.utilities import (
|
||||
simple_parallel_map,
|
||||
clean,
|
||||
is_english,
|
||||
predict_language,
|
||||
unchunk,
|
||||
)
|
||||
|
||||
|
||||
def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
||||
|
|
@ -2100,7 +2142,9 @@ def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
|
|||
]
|
||||
|
||||
|
||||
preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
||||
preprocessed_data = unchunk(
|
||||
simple_parallel_map(preprocess_chunk, chunks, concurrency=4)
|
||||
)</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2121,58 +2165,8 @@ preprocessed_chunks = parallel_map(preprocess_chunk, chunks)</div>
|
|||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="text/plain">
|
||||
<pre>Spacy model en_core_web_sm not found locally, downloading...
|
||||
Collecting en-core-web-sm==3.3.0
|
||||
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00
|
||||
Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)
|
||||
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)
|
||||
Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)
|
||||
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)
|
||||
Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)
|
||||
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)
|
||||
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)
|
||||
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)
|
||||
Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)
|
||||
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)
|
||||
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)
|
||||
Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)
|
||||
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)
|
||||
Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)
|
||||
Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)
|
||||
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)
|
||||
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)
|
||||
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)
|
||||
Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)
|
||||
Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)
|
||||
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
|
||||
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)
|
||||
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)
|
||||
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)
|
||||
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)
|
||||
Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)
|
||||
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)
|
||||
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)
|
||||
Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)
|
||||
Installing collected packages: en-core-web-sm
|
||||
Successfully installed en-core-web-sm-3.3.0
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:21:57,983 | WARNING | Limiting concurrency to 1 because there are only 1 chunks</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:21:57,984 | INFO | Starting parallel map (concurrency: 1, chunk size: 1)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:21:57,984 | WARNING | Running in series, there is no reason for parallelism</span>
|
||||
100%|██████████| 1/1 [03:26<00:00, 206.86s/it]
|
||||
<pre>100%|██████████| 4/4 [04:42<00:00, 70.62s/it]
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2203,19 +2197,9 @@ Successfully installed en-core-web-sm-3.3.0
|
|||
</div>
|
||||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">chain</span>
|
||||
|
||||
<span class="n">preprocessed_data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">chain</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_chunks</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span>
|
||||
<span class="o">*</span><span class="n">preprocessed_data</span>
|
||||
<span class="p">)</span> <span class="c1"># X is the input, y is the expected (ground truth) output</span>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">X</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">preprocessed_data</span><span class="p">)</span> <span class="c1"># X is the input, y is the expected output</span>
|
||||
</pre></div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">from itertools import chain
|
||||
|
||||
preprocessed_data = list(chain(*preprocessed_chunks))
|
||||
X, y = zip(
|
||||
*preprocessed_data
|
||||
) # X is the input, y is the expected (ground truth) output</div>
|
||||
<div id="cell-4" class="clipboard-copy-txt">X, y = zip(*preprocessed_data) # X is the input, y is the expected output</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2238,17 +2222,18 @@ X, y = zip(
|
|||
</pre></div>
|
||||
<blockquote><p>You can install MongoDB from <a href="https://www.mongodb.com/docs/manual/installation">here</a> or <a href="https://www.mongodb.com/cloud/atlas/register">use it as a service</a></p>
|
||||
</blockquote>
|
||||
<p>Otherwise, TinyDB is used which is just a local JSON file.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell jp-mod-noOutputs ">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [5]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
<div class="jp-InputPrompt jp-InputArea-prompt">In [8]:</div><div class="jp-CodeMirrorEditor jp-Editor jp-InputArea-editor" data-type="inline">
|
||||
|
||||
|
||||
<div class="CodeMirror cm-s-jupyter">
|
||||
|
|
@ -2277,40 +2262,6 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="jp-Cell-outputWrapper">
|
||||
<div class="jp-Collapser jp-OutputCollapser jp-Cell-outputCollapser">
|
||||
</div>
|
||||
|
||||
|
||||
<div class="jp-OutputArea jp-Cell-outputArea">
|
||||
|
||||
<div class="jp-OutputArea-child">
|
||||
|
||||
|
||||
<div class="jp-OutputPrompt jp-OutputArea-prompt"></div>
|
||||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-06-25 14:25:24,989 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,990 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,991 | INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,992 | INFO | Settings: configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,993 | INFO | 🔩 tracing_database: MongoDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 large_file_implementation: LargeFileMongo</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,994 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,995 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,996 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-06-25 14:25:24,997 | INFO | 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-06-25 14:25:24,998 | WARNING | > Find out more at https://se-ml.github.io/practices/</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
|
|
@ -2319,7 +2270,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h3 id="next-part-2">Next: <a href="train.ipynb">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
<h3 id="next-part-2">Next: <a href="/examples/simple/train">Part 2</a><a class="anchor-link" href="#next-part-2">¶</a></h3>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2333,7 +2284,7 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -2370,13 +2321,13 @@ add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)</div>
|
|||
|
||||
|
||||
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Train a domain classifier on the semantic scholar dataset" rel="next">
|
||||
<a href="../train/" class="md-footer__link md-footer__link--next" aria-label="Next: Optimise and train a model" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -707,6 +734,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/explanation.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -745,7 +773,7 @@
|
|||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../examples/simple/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Train a domain classifier on the semantic scholar dataset" rel="prev">
|
||||
<a href="../examples/simple/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Hardening and deployment" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
|
|
@ -754,7 +782,7 @@
|
|||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -601,7 +628,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -615,7 +642,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -629,7 +656,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -697,6 +724,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/call-remote.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -35,9 +35,9 @@ configure(
|
|||
|
||||
The only aspect that cannot be automated is choosing the backing storage for the database and file storage.
|
||||
|
||||
Right now, you have 3 options for storing the models and large datasets: [great_ai.large_file.LargeFileLocal][], [great_ai.large_file.LargeFileMongo][], and [great_ai.large_file.LargeFileS3][].
|
||||
Right now, you have 3 options for storing the models and large datasets: [LargeFileLocal][great_ai.large_file.LargeFileLocal], [LargeFileMongo][great_ai.large_file.LargeFileMongo], and [LargeFileS3][great_ai.large_file.LargeFileS3].
|
||||
|
||||
Without explicit configuration, [great_ai.large_file.LargeFileLocal][] is selected by default. This one still version-controls your files but it only stores them in a local path.
|
||||
Without explicit configuration, [LargeFileLocal][great_ai.large_file.LargeFileLocal] is selected by default. This one still version-controls your files but it only stores them in a local path.
|
||||
|
||||
!!! important
|
||||
If your working directory contains a `mongo.ini` or `s3.ini` file, an attempt is made to auto-configure [LargeFileMongo][great_ai.large_file.LargeFileMongo] or [LargeFileS3][great_ai.large_file.LargeFileS3] respectively.
|
||||
|
|
@ -76,6 +76,7 @@ save_model(model, 'my-model')
|
|||
MONGO_CONNECTION_STRING=mongodb://localhost:27017 # this is the default value
|
||||
# if `MONGO_CONNECTION_STRING` is specified, this default is overridden
|
||||
MONGO_CONNECTION_STRING=ENV:MONGO_CONNECTION_STRING
|
||||
|
||||
MONGO_DATABASE=my-database # it is automatically created if doesn't exist
|
||||
```
|
||||
|
||||
|
|
@ -98,4 +99,4 @@ By default, a thread-safe version of [TinyDB](https://tinydb.readthedocs.io/en/l
|
|||
|
||||
### MongoDB
|
||||
|
||||
At the moment, only MongoDB is supported as a production-ready `TracingDatabase`. In order to use it, you have to either place a file named `mongo.ini` in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver.configure_credentials_from_file] or [MongoDbDriver.configure_credentials][great_ai.MongoDbDriver.configure_credentials].
|
||||
At the moment, only MongoDB is supported as a production-ready `TracingDatabase`. In order to use it, you have to either place a file named `mongo.ini` in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver] or [MongoDbDriver.configure_credentials][great_ai.MongoDbDriver.configure_credentials].
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -691,7 +718,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -705,7 +732,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -719,7 +746,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -849,6 +876,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/configure-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -886,8 +914,8 @@
|
|||
</ol>
|
||||
<h2 id="using-remote-storage">Using remote storage<a class="headerlink" href="#using-remote-storage" title="Permanent link">#</a></h2>
|
||||
<p>The only aspect that cannot be automated is choosing the backing storage for the database and file storage.</p>
|
||||
<p>Right now, you have 3 options for storing the models and large datasets: <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">great_ai.large_file.LargeFileLocal</a>, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">great_ai.large_file.LargeFileMongo</a>, and <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">great_ai.large_file.LargeFileS3</a>.</p>
|
||||
<p>Without explicit configuration, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">great_ai.large_file.LargeFileLocal</a> is selected by default. This one still version-controls your files but it only stores them in a local path.</p>
|
||||
<p>Right now, you have 3 options for storing the models and large datasets: <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">LargeFileLocal</a>, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">LargeFileMongo</a>, and <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">LargeFileS3</a>.</p>
|
||||
<p>Without explicit configuration, <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileLocal">LargeFileLocal</a> is selected by default. This one still version-controls your files but it only stores them in a local path.</p>
|
||||
<div class="admonition important">
|
||||
<p class="admonition-title">Important</p>
|
||||
<p>If your working directory contains a <code>mongo.ini</code> or <code>s3.ini</code> file, an attempt is made to auto-configure <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileMongo">LargeFileMongo</a> or <a class="autorefs autorefs-internal" href="../../reference/large-file/#great_ai.large_file.LargeFileS3">LargeFileS3</a> respectively.</p>
|
||||
|
|
@ -919,7 +947,8 @@
|
|||
<div class="highlight"><span class="filename">mongo.ini</span><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="n">MONGO_CONNECTION_STRING</span><span class="o">=</span><span class="n">mongodb</span><span class="p">:</span><span class="err">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">27017</span><span class="w"> </span><span class="c1"># this is the default value</span><span class="w"></span>
|
||||
<a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a><span class="c1"># if `MONGO_CONNECTION_STRING` is specified, this default is overridden</span><span class="w"></span>
|
||||
<a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="n">MONGO_CONNECTION_STRING</span><span class="o">=</span><span class="n">ENV</span><span class="p">:</span><span class="n">MONGO_CONNECTION_STRING</span><span class="w"></span>
|
||||
<a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a><span class="n">MONGO_DATABASE</span><span class="o">=</span><span class="n">my-database</span><span class="w"> </span><span class="c1"># it is automatically created if doesn't exist</span><span class="w"></span>
|
||||
<a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a>
|
||||
<a id="__codelineno-3-5" name="__codelineno-3-5" href="#__codelineno-3-5"></a><span class="n">MONGO_DATABASE</span><span class="o">=</span><span class="n">my-database</span><span class="w"> </span><span class="c1"># it is automatically created if doesn't exist</span><span class="w"></span>
|
||||
</code></pre></div>
|
||||
<div class="highlight"><span class="filename">use-mongo.py</span><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileMongo</span>
|
||||
<a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span>
|
||||
|
|
@ -936,14 +965,14 @@
|
|||
<h2 id="using-a-database">Using a database<a class="headerlink" href="#using-a-database" title="Permanent link">#</a></h2>
|
||||
<p>By default, a thread-safe version of <a href="https://tinydb.readthedocs.io/en/latest/" target="_blank">TinyDB</a> is utilised for saving the prediction traces into a local file. Unfortunately, for most production needs, this method is not suitable.</p>
|
||||
<h3 id="mongodb">MongoDB<a class="headerlink" href="#mongodb" title="Permanent link">#</a></h3>
|
||||
<p>At the moment, only MongoDB is supported as a production-ready <code>TracingDatabase</code>. In order to use it, you have to either place a file named <code>mongo.ini</code> in your working directory, or explicitly call [MongoDbDriver.configure_credentials_from_file][great_ai.MongoDbDriver.configure_credentials_from_file] or <a class="autorefs autorefs-internal" href="../../reference/#great_ai.persistence.mongodb_driver.MongoDbDriver.configure_credentials">MongoDbDriver.configure_credentials</a>.</p>
|
||||
<p>At the moment, only MongoDB is supported as a production-ready <code>TracingDatabase</code>. In order to use it, you have to either place a file named <code>mongo.ini</code> in your working directory, or explicitly call <a class="autorefs autorefs-internal" href="../../reference/#great_ai.MongoDbDriver">MongoDbDriver.configure_credentials_from_file</a> or <a class="autorefs autorefs-internal" href="../../reference/#great_ai.persistence.mongodb_driver.MongoDbDriver.configure_credentials">MongoDbDriver.configure_credentials</a>.</p>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ def greeter(your_name):
|
|||
|
||||
## With types
|
||||
|
||||
[Type annotating your codebase](https://realpython.com/python-type-checking/){ target=_blank } can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.
|
||||
Even though it's not required by GreatAI, [type annotating your codebase](https://realpython.com/python-type-checking/){ target=_blank } can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.
|
||||
|
||||
```python title="type_safe_greeter.py"
|
||||
from great_ai import GreatAI
|
||||
|
|
@ -100,7 +100,7 @@ assert type_safe_greeter('Andras').output == 'Hi Andras'
|
|||
Refer to the following example summarising the options you have when instantiating a GreatAI service.
|
||||
|
||||
```python title="complex.py"
|
||||
from great_ai import save_model, GreatAI, parameter, use_model
|
||||
from great_ai import save_model, GreatAI, parameter, use_model, log_metric
|
||||
|
||||
save_model(4, 'secret-number') #(1)
|
||||
|
||||
|
|
@ -108,6 +108,10 @@ save_model(4, 'secret-number') #(1)
|
|||
@parameter('positive_number', validator=lambda n: n > 0, disable_logging=True)
|
||||
@use_model('secret-number', version='latest', model_kwarg_name='secret')
|
||||
def add_number(positive_number: int, secret: int) -> int:
|
||||
log_metric(
|
||||
'log directly into the returned Trace',
|
||||
positive_number * 2
|
||||
)
|
||||
return positive_number + secret
|
||||
|
||||
assert add_number(1).output == 5
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--active">
|
||||
|
|
@ -685,7 +712,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -699,7 +726,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -713,7 +740,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -837,6 +864,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/create-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -861,7 +889,7 @@
|
|||
<p>The purpose of the <a class="autorefs autorefs-internal" href="../../reference/#great_ai.deploy.great_ai.GreatAI.create">@GreatAI.create</a> is simply to provide you with type-checking through MyPy, Pylance, and similar libraries. However, the overloading support for <code>__new__</code> is lacking in MyPy, thus, a static factory method is used instead.</p>
|
||||
</details>
|
||||
<h2 id="with-types">With types<a class="headerlink" href="#with-types" title="Permanent link">#</a></h2>
|
||||
<p><a href="https://realpython.com/python-type-checking/" target="_blank">Type annotating your codebase</a> can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.</p>
|
||||
<p>Even though it's not required by GreatAI, <a href="https://realpython.com/python-type-checking/" target="_blank">type annotating your codebase</a> can save you from lots of trivial mistakes, that's why it's highly advised. Simply add the expected types to your function's signature.</p>
|
||||
<div class="highlight"><span class="filename">type_safe_greeter.py</span><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">GreatAI</span>
|
||||
<a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a>
|
||||
<a id="__codelineno-2-3" name="__codelineno-2-3" href="#__codelineno-2-3"></a><span class="nd">@GreatAI</span><span class="o">.</span><span class="n">create</span>
|
||||
|
|
@ -920,7 +948,7 @@
|
|||
</div>
|
||||
<h2 id="complex-example">Complex example<a class="headerlink" href="#complex-example" title="Permanent link">#</a></h2>
|
||||
<p>Refer to the following example summarising the options you have when instantiating a GreatAI service.</p>
|
||||
<div class="highlight"><span class="filename">complex.py</span><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span><span class="p">,</span> <span class="n">GreatAI</span><span class="p">,</span> <span class="n">parameter</span><span class="p">,</span> <span class="n">use_model</span>
|
||||
<div class="highlight"><span class="filename">complex.py</span><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">save_model</span><span class="p">,</span> <span class="n">GreatAI</span><span class="p">,</span> <span class="n">parameter</span><span class="p">,</span> <span class="n">use_model</span><span class="p">,</span> <span class="n">log_metric</span>
|
||||
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>
|
||||
<a id="__codelineno-6-3" name="__codelineno-6-3" href="#__codelineno-6-3"></a><span class="n">save_model</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="s1">'secret-number'</span><span class="p">)</span> <span class="c1">#(1)</span>
|
||||
<a id="__codelineno-6-4" name="__codelineno-6-4" href="#__codelineno-6-4"></a>
|
||||
|
|
@ -928,9 +956,13 @@
|
|||
<a id="__codelineno-6-6" name="__codelineno-6-6" href="#__codelineno-6-6"></a><span class="nd">@parameter</span><span class="p">(</span><span class="s1">'positive_number'</span><span class="p">,</span> <span class="n">validator</span><span class="o">=</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span> <span class="o">></span> <span class="mi">0</span><span class="p">,</span> <span class="n">disable_logging</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
|
||||
<a id="__codelineno-6-7" name="__codelineno-6-7" href="#__codelineno-6-7"></a><span class="nd">@use_model</span><span class="p">(</span><span class="s1">'secret-number'</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s1">'latest'</span><span class="p">,</span> <span class="n">model_kwarg_name</span><span class="o">=</span><span class="s1">'secret'</span><span class="p">)</span>
|
||||
<a id="__codelineno-6-8" name="__codelineno-6-8" href="#__codelineno-6-8"></a><span class="k">def</span> <span class="nf">add_number</span><span class="p">(</span><span class="n">positive_number</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">secret</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
|
||||
<a id="__codelineno-6-9" name="__codelineno-6-9" href="#__codelineno-6-9"></a> <span class="k">return</span> <span class="n">positive_number</span> <span class="o">+</span> <span class="n">secret</span>
|
||||
<a id="__codelineno-6-10" name="__codelineno-6-10" href="#__codelineno-6-10"></a>
|
||||
<a id="__codelineno-6-11" name="__codelineno-6-11" href="#__codelineno-6-11"></a><span class="k">assert</span> <span class="n">add_number</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">output</span> <span class="o">==</span> <span class="mi">5</span>
|
||||
<a id="__codelineno-6-9" name="__codelineno-6-9" href="#__codelineno-6-9"></a> <span class="n">log_metric</span><span class="p">(</span>
|
||||
<a id="__codelineno-6-10" name="__codelineno-6-10" href="#__codelineno-6-10"></a> <span class="s1">'log directly into the returned Trace'</span><span class="p">,</span>
|
||||
<a id="__codelineno-6-11" name="__codelineno-6-11" href="#__codelineno-6-11"></a> <span class="n">positive_number</span> <span class="o">*</span> <span class="mi">2</span>
|
||||
<a id="__codelineno-6-12" name="__codelineno-6-12" href="#__codelineno-6-12"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-6-13" name="__codelineno-6-13" href="#__codelineno-6-13"></a> <span class="k">return</span> <span class="n">positive_number</span> <span class="o">+</span> <span class="n">secret</span>
|
||||
<a id="__codelineno-6-14" name="__codelineno-6-14" href="#__codelineno-6-14"></a>
|
||||
<a id="__codelineno-6-15" name="__codelineno-6-15" href="#__codelineno-6-15"></a><span class="k">assert</span> <span class="n">add_number</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">output</span> <span class="o">==</span> <span class="mi">5</span>
|
||||
</code></pre></div>
|
||||
<ol>
|
||||
<li>Refer to <a href="/how-to-guides/store-models">storing models</a> for specifying where to store your models. </li>
|
||||
|
|
@ -941,7 +973,7 @@
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -962,7 +994,7 @@
|
|||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../../tutorial/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Harden and deploy your app" rel="prev">
|
||||
<a href="../install/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Installation guide" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
|
|
@ -971,7 +1003,7 @@
|
|||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Harden and deploy your app
|
||||
Installation guide
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -651,7 +678,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -665,7 +692,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -679,7 +706,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -769,6 +796,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/handle-training-data.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
924
how-to-guides/install/index.html
Normal file
924
how-to-guides/install/index.html
Normal file
|
|
@ -0,0 +1,924 @@
|
|||
|
||||
<!doctype html>
|
||||
<html lang="en" class="no-js">
|
||||
<head>
|
||||
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
|
||||
<meta name="description" content="GreatAI helps you easily transform your prototype AI code into production-ready software.">
|
||||
|
||||
|
||||
<meta name="author" content="András Schmelczer">
|
||||
|
||||
|
||||
<link rel="icon" href="../../media/favicon.ico">
|
||||
<meta name="generator" content="mkdocs-1.3.0, mkdocs-material-8.3.9">
|
||||
|
||||
|
||||
|
||||
<title>Installation guide - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/stylesheets/main.1d29e8d0.min.css">
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/stylesheets/palette.cbb835fc.min.css">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
|
||||
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
|
||||
|
||||
|
||||
|
||||
<link rel="stylesheet" href="../../assets/_mkdocstrings.css">
|
||||
|
||||
<script>__md_scope=new URL("../..",location),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<body dir="ltr" data-md-color-scheme="default" data-md-color-primary="light-blue" data-md-color-accent="">
|
||||
|
||||
|
||||
|
||||
<script>var palette=__md_get("__palette");if(palette&&"object"==typeof palette.color)for(var key of Object.keys(palette.color))document.body.setAttribute("data-md-color-"+key,palette.color[key])</script>
|
||||
|
||||
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
|
||||
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
|
||||
<label class="md-overlay" for="__drawer"></label>
|
||||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#installation-guide" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
</div>
|
||||
<div data-md-component="announce">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
<header class="md-header" data-md-component="header">
|
||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||
<a href="../.." title="GreatAI documentation" class="md-header__button md-logo" aria-label="GreatAI documentation" data-md-component="logo">
|
||||
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
|
||||
|
||||
</a>
|
||||
<label class="md-header__button md-icon" for="__drawer">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2Z"/></svg>
|
||||
</label>
|
||||
<div class="md-header__title" data-md-component="header-title">
|
||||
<div class="md-header__ellipsis">
|
||||
<div class="md-header__topic">
|
||||
<span class="md-ellipsis">
|
||||
GreatAI documentation
|
||||
</span>
|
||||
</div>
|
||||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Installation guide
|
||||
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form class="md-header__option" data-md-component="palette">
|
||||
|
||||
|
||||
|
||||
<input class="md-option" data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme="default" data-md-color-primary="light-blue" data-md-color-accent="" aria-label="Switch to dark mode" type="radio" name="__palette" id="__palette_1">
|
||||
|
||||
<label class="md-header__button md-icon" title="Switch to dark mode" for="__palette_2" hidden>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 2a7 7 0 0 0-7 7c0 2.38 1.19 4.47 3 5.74V17a1 1 0 0 0 1 1h6a1 1 0 0 0 1-1v-2.26c1.81-1.27 3-3.36 3-5.74a7 7 0 0 0-7-7M9 21a1 1 0 0 0 1 1h4a1 1 0 0 0 1-1v-1H9v1Z"/></svg>
|
||||
</label>
|
||||
|
||||
|
||||
|
||||
|
||||
<input class="md-option" data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme="slate" data-md-color-primary="light-blue" data-md-color-accent="" aria-label="Switch to light mode" type="radio" name="__palette" id="__palette_2">
|
||||
|
||||
<label class="md-header__button md-icon" title="Switch to light mode" for="__palette_1" hidden>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 2a7 7 0 0 1 7 7c0 2.38-1.19 4.47-3 5.74V17a1 1 0 0 1-1 1H9a1 1 0 0 1-1-1v-2.26C6.19 13.47 5 11.38 5 9a7 7 0 0 1 7-7M9 21v-1h6v1a1 1 0 0 1-1 1h-4a1 1 0 0 1-1-1m3-17a5 5 0 0 0-5 5c0 2.05 1.23 3.81 3 4.58V16h4v-2.42c1.77-.77 3-2.53 3-4.58a5 5 0 0 0-5-5Z"/></svg>
|
||||
</label>
|
||||
|
||||
|
||||
</form>
|
||||
|
||||
|
||||
|
||||
<label class="md-header__button md-icon" for="__search">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
|
||||
</label>
|
||||
<div class="md-search" data-md-component="search" role="dialog">
|
||||
<label class="md-search__overlay" for="__search"></label>
|
||||
<div class="md-search__inner" role="search">
|
||||
<form class="md-search__form" name="search">
|
||||
<input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
|
||||
<label class="md-search__icon md-icon" for="__search">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</label>
|
||||
<nav class="md-search__options" aria-label="Search">
|
||||
|
||||
<button type="reset" class="md-search__icon md-icon" aria-label="Clear" tabindex="-1">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41Z"/></svg>
|
||||
</button>
|
||||
</nav>
|
||||
|
||||
</form>
|
||||
<div class="md-search__output">
|
||||
<div class="md-search__scrollwrap" data-md-scrollfix>
|
||||
<div class="md-search-result" data-md-component="search-result">
|
||||
<div class="md-search-result__meta">
|
||||
Initializing search
|
||||
</div>
|
||||
<ol class="md-search-result__list"></ol>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="md-header__source">
|
||||
<a href="https://github.com/schmelczer/great-ai" title="Go to repository" class="md-source" data-md-component="source">
|
||||
<div class="md-source__icon md-icon">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.55 236.05 244 40.45a28.87 28.87 0 0 0-40.81 0l-40.66 40.63 51.52 51.52c27.06-9.14 52.68 16.77 43.39 43.68l49.66 49.66c34.23-11.8 61.18 31 35.47 56.69-26.49 26.49-70.21-2.87-56-37.34L240.22 199v121.85c25.3 12.54 22.26 41.85 9.08 55a34.34 34.34 0 0 1-48.55 0c-17.57-17.6-11.07-46.91 11.25-56v-123c-20.8-8.51-24.6-30.74-18.64-45L142.57 101 8.45 235.14a28.86 28.86 0 0 0 0 40.81l195.61 195.6a28.86 28.86 0 0 0 40.8 0l194.69-194.69a28.86 28.86 0 0 0 0-40.81z"/></svg>
|
||||
</div>
|
||||
<div class="md-source__repository">
|
||||
schmelczer/great-ai
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</nav>
|
||||
|
||||
</header>
|
||||
|
||||
<div class="md-container" data-md-component="container">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<main class="md-main" data-md-component="main">
|
||||
<div class="md-main__inner md-grid">
|
||||
|
||||
|
||||
|
||||
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
|
||||
<div class="md-sidebar__scrollwrap">
|
||||
<div class="md-sidebar__inner">
|
||||
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||
<label class="md-nav__title" for="__drawer">
|
||||
<a href="../.." title="GreatAI documentation" class="md-nav__button md-logo" aria-label="GreatAI documentation" data-md-component="logo">
|
||||
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
|
||||
|
||||
</a>
|
||||
GreatAI documentation
|
||||
</label>
|
||||
|
||||
<div class="md-nav__source">
|
||||
<a href="https://github.com/schmelczer/great-ai" title="Go to repository" class="md-source" data-md-component="source">
|
||||
<div class="md-source__icon md-icon">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.55 236.05 244 40.45a28.87 28.87 0 0 0-40.81 0l-40.66 40.63 51.52 51.52c27.06-9.14 52.68 16.77 43.39 43.68l49.66 49.66c34.23-11.8 61.18 31 35.47 56.69-26.49 26.49-70.21-2.87-56-37.34L240.22 199v121.85c25.3 12.54 22.26 41.85 9.08 55a34.34 34.34 0 0 1-48.55 0c-17.57-17.6-11.07-46.91 11.25-56v-123c-20.8-8.51-24.6-30.74-18.64-45L142.57 101 8.45 235.14a28.86 28.86 0 0 0 0 40.81l195.61 195.6a28.86 28.86 0 0 0 40.8 0l194.69-194.69a28.86 28.86 0 0 0 0-40.81z"/></svg>
|
||||
</div>
|
||||
<div class="md-source__repository">
|
||||
schmelczer/great-ai
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../.." class="md-nav__link">
|
||||
Overview
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_2" type="checkbox" id="__nav_2" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_2">
|
||||
Tutorial
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Tutorial" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Tutorial
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/" class="md-nav__link">
|
||||
Tutorial overview
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/train/" class="md-nav__link">
|
||||
Train your model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../tutorial/deploy/" class="md-nav__link">
|
||||
Harden and deploy your app
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_3" type="checkbox" id="__nav_3" checked>
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3">
|
||||
User Guides
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="User Guides" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_3">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
User Guides
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--active">
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="toc" type="checkbox" id="__toc">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
Installation guide
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Installation guide
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#command-line-tools" class="md-nav__link">
|
||||
Command-line tools
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../configure-service/" class="md-nav__link">
|
||||
How to configure GreatAI
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../use-service/" class="md-nav__link">
|
||||
How to use a GreatAI service
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../handle-training-data/" class="md-nav__link">
|
||||
How to handle training data
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../large_file/" class="md-nav__link">
|
||||
How to use LargeFile
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../call_remote.md" class="md-nav__link">
|
||||
None
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../scraping/" class="md-nav__link">
|
||||
Scraping
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_4" type="checkbox" id="__nav_4" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_4">
|
||||
Reference
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Reference" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_4">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Reference
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/" class="md-nav__link">
|
||||
GreatAI reference
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/utilities/" class="md-nav__link">
|
||||
Utilities
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/large-file/" class="md-nav__link">
|
||||
LargeFile
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../reference/views/" class="md-nav__link">
|
||||
View models
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_5" type="checkbox" id="__nav_5" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_5">
|
||||
Examples
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Examples" data-md-level="1">
|
||||
<label class="md-nav__title" for="__nav_5">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Examples
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_5_1" type="checkbox" id="__nav_5_1" >
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_5_1">
|
||||
Explainable Naive Bayes
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" aria-label="Explainable Naive Bayes" data-md-level="2">
|
||||
<label class="md-nav__title" for="__nav_5_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Explainable Naive Bayes
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../explanation/" class="md-nav__link">
|
||||
Explanation
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" >
|
||||
<div class="md-sidebar__scrollwrap">
|
||||
<div class="md-sidebar__inner">
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#command-line-tools" class="md-nav__link">
|
||||
Command-line tools
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="md-content" data-md-component="content">
|
||||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/install.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<h1 id="installation-guide">Installation guide<a class="headerlink" href="#installation-guide" title="Permanent link">#</a></h1>
|
||||
<p>Provided you already have <a href="https://www.python.org/downloads/" target="_blank">Python3</a> (and pip) installed, simply execute:</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>pip install great-ai
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>Python 3.8 or later is required.</p>
|
||||
</blockquote>
|
||||
<p>This will work on all major operating systems.</p>
|
||||
<h2 id="command-line-tools">Command-line tools<a class="headerlink" href="#command-line-tools" title="Permanent link">#</a></h2>
|
||||
<p>After installation, <code>great-ai</code> and <code>large-file</code> are available as commands. The former is required for deploying your application while the latter lets you manage models and datasets from your terminal.</p>
|
||||
<div class="admonition warning">
|
||||
<p class="admonition-title">Windows</p>
|
||||
<p>On Windows, you might encounter a similar <mark>warning</mark> from <code>pip</code>:</p>
|
||||
<blockquote>
|
||||
<p>WARNING: The scripts great-ai.exe, great_ai.exe, large-file.exe and large_file.exe are installed in 'C:\Users...\Scripts' which is not on PATH.
|
||||
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</p>
|
||||
</blockquote>
|
||||
<p>This means that <code>great-ai.exe</code> and <code>large-file.exe</code> are not in your <code>PATH</code>. Either add their containing directory ('C:\Users...\Scripts' in this case) to your <code>PATH</code> or use <code>python3 -m great_ai</code> and <code>python3 -m great_ai.large_file</code> instead of the exe-s.</p>
|
||||
</div>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</article>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</main>
|
||||
|
||||
<footer class="md-footer">
|
||||
|
||||
|
||||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href="../../tutorial/deploy/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Harden and deploy your app" rel="prev">
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
|
||||
</div>
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Previous
|
||||
</span>
|
||||
Harden and deploy your app
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="../create-service/" class="md-footer__link md-footer__link--next" aria-label="Next: How to create a GreatAI service" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
How to create a GreatAI service
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4Z"/></svg>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
</nav>
|
||||
|
||||
<div class="md-footer-meta md-typeset">
|
||||
<div class="md-footer-meta__inner md-grid">
|
||||
<div class="md-copyright">
|
||||
|
||||
<div class="md-copyright__highlight">
|
||||
GNU General Public License v3
|
||||
</div>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<div class="md-social">
|
||||
|
||||
|
||||
|
||||
<a href="https://schmelczer.dev" target="_blank" rel="noopener" title="about the author" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="m414.8 40.79-128 448.01c-4.9 17-22.6 26.8-39.6 22-17-4.9-26.8-22.6-22-39.6l128-447.99c4.9-16.994 22.6-26.834 39.6-21.978 17 4.855 26.8 22.568 22 39.558zm103.8 80.61 112 112c12.5 12.5 12.5 32.7 0 45.2l-112 112c-12.5 12.5-32.7 12.5-45.2 0s-12.5-32.7 0-45.2l89.3-89.4-89.3-89.4c-12.5-12.5-12.5-32.7 0-45.2s32.7-12.5 45.2 0zm-352 45.2L77.25 256l89.35 89.4c12.5 12.5 12.5 32.7 0 45.2s-32.7 12.5-45.2 0L9.372 278.6c-12.496-12.5-12.496-32.7 0-45.2l112.028-112c12.5-12.5 32.7-12.5 45.2 0s12.5 32.7 0 45.2z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai" target="_blank" rel="noopener" title="great-ai on Github" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 496 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://hub.docker.com/repository/docker/schmelczera/great-ai" target="_blank" rel="noopener" title="great-ai on DockerHub" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M349.9 236.3h-66.1v-59.4h66.1v59.4zm0-204.3h-66.1v60.7h66.1V32zm78.2 144.8H362v59.4h66.1v-59.4zm-156.3-72.1h-66.1v60.1h66.1v-60.1zm78.1 0h-66.1v60.1h66.1v-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7H2.4c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4.4 67.6.1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zm-511.1-27.9h-66v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm-78.1-72.1h-66.1v60.1h66.1v-60.1z"/></svg>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="https://pypi.org/project/great-ai" target="_blank" rel="noopener" title="great-ai on PyPI" class="md-social__link">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.8 200.5c-7.7-30.9-22.3-54.2-53.4-54.2h-40.1v47.4c0 36.8-31.2 67.8-66.8 67.8H172.7c-29.2 0-53.4 25-53.4 54.3v101.8c0 29 25.2 46 53.4 54.3 33.8 9.9 66.3 11.7 106.8 0 26.9-7.8 53.4-23.5 53.4-54.3v-40.7H226.2v-13.6h160.2c31.1 0 42.6-21.7 53.4-54.2 11.2-33.5 10.7-65.7 0-108.6zM286.2 404c11.1 0 20.1 9.1 20.1 20.3 0 11.3-9 20.4-20.1 20.4-11 0-20.1-9.2-20.1-20.4.1-11.3 9.1-20.3 20.1-20.3zM167.8 248.1h106.8c29.7 0 53.4-24.5 53.4-54.3V91.9c0-29-24.4-50.7-53.4-55.6-35.8-5.9-74.7-5.6-106.8.1-45.2 8-53.4 24.7-53.4 55.6v40.7h106.9v13.6h-147c-31.1 0-58.3 18.7-66.8 54.2-9.8 40.7-10.2 66.1 0 108.6 7.6 31.6 25.7 54.2 56.8 54.2H101v-48.8c0-35.3 30.5-66.4 66.8-66.4zm-6.7-142.6c-11.1 0-20.1-9.1-20.1-20.3.1-11.3 9-20.4 20.1-20.4 11 0 20.1 9.2 20.1 20.4s-9 20.3-20.1 20.3z"/></svg>
|
||||
</a>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
<div class="md-dialog" data-md-component="dialog">
|
||||
<div class="md-dialog__inner md-typeset"></div>
|
||||
</div>
|
||||
<script id="__config" type="application/json">{"base": "../..", "features": ["content.code.annotate", "content.tooltips"], "search": "../../assets/javascripts/workers/search.b97dbffb.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version.title": "Select version"}}</script>
|
||||
|
||||
|
||||
<script src="../../assets/javascripts/bundle.6c7ad80a.min.js"></script>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
21
how-to-guides/install/install.md
Normal file
21
how-to-guides/install/install.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# Installation guide
|
||||
|
||||
Provided you already have [Python3](https://www.python.org/downloads/){ target=_blank } (and pip) installed, simply execute:
|
||||
|
||||
```sh
|
||||
pip install great-ai
|
||||
```
|
||||
> Python 3.8 or later is required.
|
||||
|
||||
This will work on all major operating systems.
|
||||
|
||||
## Command-line tools
|
||||
|
||||
After installation, `great-ai` and `large-file` are available as commands. The former is required for deploying your application while the latter lets you manage models and datasets from your terminal.
|
||||
|
||||
!!! warning "Windows"
|
||||
On Windows, you might encounter a similar ==warning== from `pip`:
|
||||
> WARNING: The scripts great-ai.exe, great_ai.exe, large-file.exe and large_file.exe are installed in 'C:\Users\...\Scripts' which is not on PATH.
|
||||
> Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
|
||||
|
||||
This means that `great-ai.exe` and `large-file.exe` are not in your `PATH`. Either add their containing directory ('C:\Users\...\Scripts' in this case) to your `PATH` or use `python3 -m great_ai` and `python3 -m great_ai.large_file` instead of the exe-s.
|
||||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -732,7 +759,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -746,7 +773,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -760,7 +787,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -931,6 +958,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/large_file.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -348,6 +361,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -606,7 +633,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -620,7 +647,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -634,7 +661,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -697,6 +724,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="scraping.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -353,6 +366,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -705,7 +732,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -719,7 +746,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -733,7 +760,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -877,6 +904,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/how-to-guides/use-service.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -919,9 +947,10 @@
|
|||
<p>Some configuration options are also supported.</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a>great-ai greeter.py --port <span class="m">8000</span> --host <span class="m">127</span>.0.0.1 --timeout_keep_alive <span class="m">10</span>
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>For more options (but no Notebook support, use <a href="https://www.uvicorn.org/" target="_blank">uvicorn</a>)</p>
|
||||
</blockquote>
|
||||
<details class="note">
|
||||
<summary>More options</summary>
|
||||
<p>For more options (but no Notebook support), simply use <a href="https://www.uvicorn.org/" target="_blank">uvicorn</a> for starting your app (available at <code>greeter.app</code>).</p>
|
||||
</details>
|
||||
<h3 id="in-production">In production<a class="headerlink" href="#in-production" title="Permanent link">#</a></h3>
|
||||
<p>There are three main approaches for deploying a GreatAI service.</p>
|
||||
<h4 id="manual-deployment">Manual deployment<a class="headerlink" href="#manual-deployment" title="Permanent link">#</a></h4>
|
||||
|
|
@ -941,7 +970,7 @@
|
|||
<p>You can replace <code>pwd</code> with the path to your code's folder.</p>
|
||||
</blockquote>
|
||||
<h4 id="use-a-platform-as-a-service">Use a Platform-as-a-Service<a class="headerlink" href="#use-a-platform-as-a-service" title="Permanent link">#</a></h4>
|
||||
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a> or <a href="https://mlem.ai/" target="_blank">MLEM</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
|
||||
<p>Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as <a href="https://aws.amazon.com/ecs/" target="_blank">AWS ECS</a>, <a href="https://www.digitalocean.com/products/app-platform" target="_blank">DO App platform</a>, <a href="https://mlem.ai/" target="_blank">MLEM</a>) that take a Docker image as a source and handle the rest of the deployment.</p>
|
||||
<p>To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as <a href="https://pytorch.org/" target="_blank">PyTorch</a> or <a href="https://www.tensorflow.org/" target="_blank">TensorFlow</a>.</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="k">FROM</span><span class="w"> </span><span class="s">schmelczera/great-ai:latest</span>
|
||||
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a>
|
||||
|
|
@ -988,7 +1017,7 @@
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
|
|
@ -51,7 +51,8 @@ Some configuration options are also supported.
|
|||
```sh
|
||||
great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10
|
||||
```
|
||||
> For more options (but no Notebook support, use [uvicorn](https://www.uvicorn.org/){ target=_blank })
|
||||
??? note "More options"
|
||||
For more options (but no Notebook support), simply use [uvicorn](https://www.uvicorn.org/){ target=_blank } for starting your app (available at `greeter.app`).
|
||||
|
||||
### In production
|
||||
|
||||
|
|
@ -80,7 +81,7 @@ docker run -p 6060:6060 --volume `pwd`:/app --rm \
|
|||
|
||||
#### Use a Platform-as-a-Service
|
||||
|
||||
Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank } or [MLEM](https://mlem.ai/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.
|
||||
Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [AWS ECS](https://aws.amazon.com/ecs/){ target=_blank }, [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank }, [MLEM](https://mlem.ai/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.
|
||||
|
||||
To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as [PyTorch](https://pytorch.org/){ target=_blank } or [TensorFlow](https://www.tensorflow.org/){ target=_blank }.
|
||||
|
||||
|
|
|
|||
78
index.html
78
index.html
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -269,6 +282,13 @@
|
|||
Features
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#roadmap" class="md-nav__link">
|
||||
Roadmap
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
|
|
@ -416,6 +436,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -666,7 +700,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -680,7 +714,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -694,7 +728,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -758,6 +792,13 @@
|
|||
Features
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#roadmap" class="md-nav__link">
|
||||
Roadmap
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
|
|
@ -803,6 +844,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/index.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -814,12 +856,12 @@
|
|||
<img src="media/logo.png" width=80>
|
||||
</div>
|
||||
|
||||
<p><a href="https://sonar.scoutinscience.com/dashboard?id=great-ai"><img alt="Sonar line coverage" src="https://sonar.scoutinscience.com/api/project_badges/measure?project=great-ai&metric=coverage" /></a>
|
||||
<a href="https://sonar.scoutinscience.com/dashboard?id=great-ai"><img alt="Sonar LoC" src="https://sonar.scoutinscience.com/api/project_badges/measure?project=great-ai&metric=ncloc" /></a>
|
||||
<a href="https://github.com/schmelczer/great-ai/actions/workflows/test.yml"><img alt="Test" src="https://github.com/schmelczer/great-ai/actions/workflows/test.yml/badge.svg" /></a>
|
||||
<a href="https://badge.fury.io/py/great-ai"><img alt="PyPI version" src="https://badge.fury.io/py/great-ai.svg" /></a>
|
||||
<p><a href="https://badge.fury.io/py/great-ai"><img alt="PyPI version" src="https://badge.fury.io/py/great-ai.svg" /></a>
|
||||
<a href="https://pepy.tech/project/great-ai"><img alt="Downloads" src="https://pepy.tech/badge/great-ai/month" /></a>
|
||||
<img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/schmelczera/great-ai" /></p>
|
||||
<img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/schmelczera/great-ai" />
|
||||
<a href="https://github.com/schmelczer/great-ai/actions/workflows/test.yml"><img alt="Test" src="https://github.com/schmelczer/great-ai/actions/workflows/test.yml/badge.svg" /></a>
|
||||
<a href="https://sonar.scoutinscience.com/dashboard?id=great-ai"><img alt="Sonar line coverage" src="https://sonar.scoutinscience.com/api/project_badges/measure?project=great-ai&metric=coverage" /></a>
|
||||
<a href="https://sonar.scoutinscience.com/dashboard?id=great-ai"><img alt="Sonar LoC" src="https://sonar.scoutinscience.com/api/project_badges/measure?project=great-ai&metric=ncloc" /></a></p>
|
||||
<p>Applying AI is becoming increasingly easier but many case studies have shown that these applications are often deployed poorly. This may lead to suboptimal performance and to introducing <a href="https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction" target="_blank">unintended biases</a>. GreatAI helps fixing this by allowing you to <mark>easily transform your prototype AI code into production-ready software</mark>.</p>
|
||||
<details class="quote">
|
||||
<summary>Case studies</summary>
|
||||
|
|
@ -838,7 +880,7 @@
|
|||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Input validation</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Sensible cache-policy</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Seamless support for both synchronous and <code>async</code> inference methods</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Easy integration with other remote GreatAI instances</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Easy integration with remote GreatAI instances</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Built-in parallelisation (with support for multiprocessing, async, and mixed modes) for batch processing</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Well-tested utilities for common NLP tasks (cleaning, language-tagging, sentence-segmentation, etc.)</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> A simple, unified configuration interface</li>
|
||||
|
|
@ -847,6 +889,11 @@
|
|||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Docker support for deployment</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Deployable Jupyter Notebooks</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled checked/><span class="task-list-indicator"></span></label> Dashboard for high-level overview and analysing traces</li>
|
||||
</ul>
|
||||
<h2 id="roadmap">Roadmap<a class="headerlink" href="#roadmap" title="Permanent link">#</a></h2>
|
||||
<ul class="task-list">
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled/><span class="task-list-indicator"></span></label> Prometheus & Grafana integration</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled/><span class="task-list-indicator"></span></label> Well-tested feature extraction code for non-NLP data</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled/><span class="task-list-indicator"></span></label> Support for direct file input</li>
|
||||
<li class="task-list-item"><label class="task-list-control"><input type="checkbox" disabled/><span class="task-list-indicator"></span></label> Support for PostgreSQL</li>
|
||||
</ul>
|
||||
|
|
@ -874,7 +921,7 @@
|
|||
</ol>
|
||||
<details class="note">
|
||||
<summary>Note</summary>
|
||||
<p>In practice, <code>hello_world</code> could be an inference function of some AI/ML application. But it could also just wrap a black-box solution of some SaaS. Either ways, it is imperative to have continuos oversight of the services you provide and data you process.</p>
|
||||
<p>In practice, <code>hello_world</code> could be an inference function of some AI/ML application. But it could also just wrap a black-box solution of some SaaS. Either ways, it is <a href="https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai" target="_blank">imperative to have continuos oversight</a> of the services you provide and data you process especially in the context of AI/ML applications.</p>
|
||||
</details>
|
||||
<div class="highlight"><span class="filename">terminal</span><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a>great-ai hello-world.py
|
||||
</code></pre></div>
|
||||
|
|
@ -901,22 +948,23 @@
|
|||
</ul>
|
||||
<h2 id="why-greatai">Why GreatAI?<a class="headerlink" href="#why-greatai" title="Permanent link">#</a></h2>
|
||||
<p>There are other, existing solutions aiming to facilitate this phase. <a href="https://aws.amazon.com/sagemaker" target="_blank">Amazon SageMaker</a> and <a href="https://www.seldon.io/solutions/open-source-projects/core" target="_blank">Seldon Core</a> provide the most comprehensive suite of features. If you have the opportunity use those, do that because they're great.</p>
|
||||
<p>However, research indicates that professionals rarely use them. This may be due to their inherent setup and operating complexity. GreatAI is designed to be as simple to use as possible. Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the <a href="https://se-ml.github.io" target="_blank">SE4ML best-practices</a>, and thus, can meaningfully improve your deployment without requiring prohibitively large effort.</p>
|
||||
<div style="display: flex; justify-content: space-evenly;">
|
||||
<p>However, research indicates that professionals rarely use them. This may be due to their inherent setup and operating complexity. <mark>GreatAI is designed to be as simple to use as possible.</mark> Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the <a href="https://se-ml.github.io" target="_blank">SE4ML best-practices</a>, and thus, can meaningfully improve your deployment without requiring prohibitively large effort.</p>
|
||||
<div style="display: flex; justify-content: space-evenly; flex-wrap: wrap;">
|
||||
<p><a class="md-button md-button--primary" href="https://pypi.org/project/great-ai"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M439.8 200.5c-7.7-30.9-22.3-54.2-53.4-54.2h-40.1v47.4c0 36.8-31.2 67.8-66.8 67.8H172.7c-29.2 0-53.4 25-53.4 54.3v101.8c0 29 25.2 46 53.4 54.3 33.8 9.9 66.3 11.7 106.8 0 26.9-7.8 53.4-23.5 53.4-54.3v-40.7H226.2v-13.6h160.2c31.1 0 42.6-21.7 53.4-54.2 11.2-33.5 10.7-65.7 0-108.6zM286.2 404c11.1 0 20.1 9.1 20.1 20.3 0 11.3-9 20.4-20.1 20.4-11 0-20.1-9.2-20.1-20.4.1-11.3 9.1-20.3 20.1-20.3zM167.8 248.1h106.8c29.7 0 53.4-24.5 53.4-54.3V91.9c0-29-24.4-50.7-53.4-55.6-35.8-5.9-74.7-5.6-106.8.1-45.2 8-53.4 24.7-53.4 55.6v40.7h106.9v13.6h-147c-31.1 0-58.3 18.7-66.8 54.2-9.8 40.7-10.2 66.1 0 108.6 7.6 31.6 25.7 54.2 56.8 54.2H101v-48.8c0-35.3 30.5-66.4 66.8-66.4zm-6.7-142.6c-11.1 0-20.1-9.1-20.1-20.3.1-11.3 9-20.4 20.1-20.4 11 0 20.1 9.2 20.1 20.4s-9 20.3-20.1 20.3z"/></svg></span> Find it on PyPI</a></p>
|
||||
<p><a class="md-button md-button--primary" href="https://hub.docker.com/repository/docker/schmelczera/great-ai"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M349.9 236.3h-66.1v-59.4h66.1v59.4zm0-204.3h-66.1v60.7h66.1V32zm78.2 144.8H362v59.4h66.1v-59.4zm-156.3-72.1h-66.1v60.1h66.1v-60.1zm78.1 0h-66.1v60.1h66.1v-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7H2.4c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4.4 67.6.1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zm-511.1-27.9h-66v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm-78.1-72.1h-66.1v60.1h66.1v-60.1z"/></svg></span> Find it on DockerHub</a></p>
|
||||
<p><a class="md-button md-button--primary" href="/tutorial"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 6.1.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2022 Fonticons, Inc.--><path d="M128 96h384v256h64V80c0-26.37-21.6-48-48-48H112c-26.37 0-48 21.63-48 48v272h64V96zm496 288H16c-8.75 0-16 7.3-16 16v16c0 35.25 28.75 64 64 64h512c35.25 0 64-28.75 64-64v-16c0-8.7-7.2-16-16-16zm-258.1-97.8c3.9 3.9 9 5.8 14.1 5.8s10.23-1.938 14.14-5.844l48-48c7.812-7.813 7.812-20.5 0-28.31l-48-48c-7.812-7.813-20.47-7.813-28.28 0-7.812 7.813-7.812 20.5 0 28.31l33.86 33.84-33.86 33.84C358 265.7 358 278.4 365.9 286.2zm-91.8-124.3c-7.812-7.813-20.47-7.813-28.28 0l-48 48c-7.812 7.813-7.812 20.5 0 28.31l48 48C249.8 290.1 254.9 292 260 292s10.23-1.938 14.14-5.844c7.812-7.813 7.812-20.5 0-28.31L240.3 224l33.86-33.84c6.94-7.76 6.94-20.46-.06-28.26z"/></svg></span> Check out the tutorial</a></p>
|
||||
</div>
|
||||
<h2 id="production-use">Production use<a class="headerlink" href="#production-use" title="Permanent link">#</a></h2>
|
||||
<p>GreatAI has been battle-tested on the core platform services of <a href="https://www.scoutinscience.com/" target="_blank">ScoutinScience</a>.</p>
|
||||
<p><img alt="ScoutinScience logo" loading="lazy" src="media/scoutinscience.svg#only-light" />
|
||||
<img alt="ScoutinScience logo" loading="lazy" src="media/scoutinscience-white.svg#only-dark" /></p>
|
||||
<p><a href="https://www.scoutinscience.com/" target="_blank"><img alt="ScoutinScience logo" loading="lazy" src="media/scoutinscience.svg#only-light" />
|
||||
<img alt="ScoutinScience logo" loading="lazy" src="media/scoutinscience-white.svg#only-dark" /></a></p>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
26
index.md
26
index.md
|
|
@ -3,13 +3,12 @@
|
|||
<img src="media/logo.png" width=80>
|
||||
</div>
|
||||
|
||||
[](https://sonar.scoutinscience.com/dashboard?id=great-ai)
|
||||
[](https://sonar.scoutinscience.com/dashboard?id=great-ai)
|
||||
[](https://github.com/schmelczer/great-ai/actions/workflows/test.yml)
|
||||
[](https://badge.fury.io/py/great-ai)
|
||||
[](https://pepy.tech/project/great-ai)
|
||||
|
||||
|
||||
[](https://github.com/schmelczer/great-ai/actions/workflows/test.yml)
|
||||
[](https://sonar.scoutinscience.com/dashboard?id=great-ai)
|
||||
[](https://sonar.scoutinscience.com/dashboard?id=great-ai)
|
||||
|
||||
Applying AI is becoming increasingly easier but many case studies have shown that these applications are often deployed poorly. This may lead to suboptimal performance and to introducing [unintended biases](https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction){ target=_blank }. GreatAI helps fixing this by allowing you to ==easily transform your prototype AI code into production-ready software==.
|
||||
|
||||
|
|
@ -32,7 +31,7 @@ Applying AI is becoming increasingly easier but many case studies have shown tha
|
|||
- [x] Input validation
|
||||
- [x] Sensible cache-policy
|
||||
- [x] Seamless support for both synchronous and `async` inference methods
|
||||
- [x] Easy integration with other remote GreatAI instances
|
||||
- [x] Easy integration with remote GreatAI instances
|
||||
- [x] Built-in parallelisation (with support for multiprocessing, async, and mixed modes) for batch processing
|
||||
- [x] Well-tested utilities for common NLP tasks (cleaning, language-tagging, sentence-segmentation, etc.)
|
||||
- [x] A simple, unified configuration interface
|
||||
|
|
@ -41,6 +40,11 @@ Applying AI is becoming increasingly easier but many case studies have shown tha
|
|||
- [x] Docker support for deployment
|
||||
- [x] Deployable Jupyter Notebooks
|
||||
- [x] Dashboard for high-level overview and analysing traces
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [ ] Prometheus & Grafana integration
|
||||
- [ ] Well-tested feature extraction code for non-NLP data
|
||||
- [ ] Support for direct file input
|
||||
- [ ] Support for PostgreSQL
|
||||
|
||||
|
|
@ -66,7 +70,7 @@ def hello_world(name: str) -> str: #(2)
|
|||
2. [Typing functions](https://docs.python.org/3/library/typing.html){ target=_blank } is recommended in general, however, not required for GreatAI to work.
|
||||
|
||||
??? note
|
||||
In practice, `hello_world` could be an inference function of some AI/ML application. But it could also just wrap a black-box solution of some SaaS. Either ways, it is imperative to have continuos oversight of the services you provide and data you process.
|
||||
In practice, `hello_world` could be an inference function of some AI/ML application. But it could also just wrap a black-box solution of some SaaS. Either ways, it is [imperative to have continuos oversight](https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai){ target=_blank } of the services you provide and data you process especially in the context of AI/ML applications.
|
||||
|
||||
```sh title="terminal"
|
||||
great-ai hello-world.py
|
||||
|
|
@ -98,18 +102,20 @@ GreatAI fits between the prototype and deployment phases of your (or your organi
|
|||
|
||||
There are other, existing solutions aiming to facilitate this phase. [Amazon SageMaker](https://aws.amazon.com/sagemaker){ target=_blank } and [Seldon Core](https://www.seldon.io/solutions/open-source-projects/core){ target=_blank } provide the most comprehensive suite of features. If you have the opportunity use those, do that because they're great.
|
||||
|
||||
However, research indicates that professionals rarely use them. This may be due to their inherent setup and operating complexity. GreatAI is designed to be as simple to use as possible. Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the [SE4ML best-practices](https://se-ml.github.io){ target=_blank }, and thus, can meaningfully improve your deployment without requiring prohibitively large effort.
|
||||
However, research indicates that professionals rarely use them. This may be due to their inherent setup and operating complexity. ==GreatAI is designed to be as simple to use as possible.== Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the [SE4ML best-practices](https://se-ml.github.io){ target=_blank }, and thus, can meaningfully improve your deployment without requiring prohibitively large effort.
|
||||
|
||||
|
||||
<div style="display: flex; justify-content: space-evenly;" markdown>
|
||||
<div style="display: flex; justify-content: space-evenly; flex-wrap: wrap;" markdown>
|
||||
[:fontawesome-brands-python: Find it on PyPI](https://pypi.org/project/great-ai){ .md-button .md-button--primary }
|
||||
|
||||
[:fontawesome-brands-docker: Find it on DockerHub](https://hub.docker.com/repository/docker/schmelczera/great-ai){ .md-button .md-button--primary }
|
||||
|
||||
[:fontawesome-solid-laptop-code: Check out the tutorial](/tutorial){ .md-button .md-button--primary }
|
||||
</div>
|
||||
|
||||
## Production use
|
||||
|
||||
GreatAI has been battle-tested on the core platform services of [ScoutinScience](https://www.scoutinscience.com/){ target=_blank }.
|
||||
|
||||
{ loading=lazy }
|
||||
{ loading=lazy }
|
||||
[{ loading=lazy }
|
||||
{ loading=lazy }](https://www.scoutinscience.com/){ target=_blank }
|
||||
|
|
|
|||
BIN
media/og-image.png
Normal file
BIN
media/og-image.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 112 KiB |
|
|
@ -1,6 +1,22 @@
|
|||
{% extends "base.html" %}
|
||||
|
||||
{% block extrahead %}
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
|
||||
{% if page.nb_url %}
|
||||
<a href="{{ page.nb_url }}" title="Download Notebook" class="md-content__button md-icon">
|
||||
{% include ".icons/material/download.svg" %}
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -820,7 +847,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -834,7 +861,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -848,7 +875,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -1107,6 +1134,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/reference/index.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -3270,6 +3298,101 @@ Returns:
|
|||
|
||||
<div class="doc doc-contents first">
|
||||
|
||||
<p>Add training data (with optional train-test splitting).</p>
|
||||
<p>Add and tag datapoints, wrap them into traces. The <code>inputs</code> are available via the
|
||||
<code>.input</code> property, while <code>expected_outputs</code> under both the <code>.output</code> and <code>.feedback</code>
|
||||
properties.</p>
|
||||
<p>All generated traces are tagged with <code>ground_truth</code> by default. Additional tags can
|
||||
be also provided. Using the <code>split_ratio</code> arguments, tags can be given randomly with
|
||||
a user-defined distribution. Only the ratio of the splits matter, they don't have to
|
||||
add up to 1.</p>
|
||||
|
||||
<p><strong>Examples:</strong></p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="gp">>>> </span><span class="n">add_ground_truth</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a><span class="gp">... </span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a><span class="gp">... </span> <span class="p">[</span><span class="s1">'odd'</span><span class="p">,</span> <span class="s1">'even'</span><span class="p">,</span> <span class="s1">'odd'</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a><span class="gp">... </span> <span class="n">tags</span><span class="o">=</span><span class="s1">'my_tag'</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a><span class="gp">... </span> <span class="n">train_split_ratio</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a><span class="gp">... </span> <span class="n">test_split_ratio</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-7" name="__codelineno-0-7" href="#__codelineno-0-7"></a><span class="gp">... </span> <span class="n">validation_split_ratio</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-8" name="__codelineno-0-8" href="#__codelineno-0-8"></a><span class="gp">... </span><span class="p">)</span>
|
||||
</code></pre></div>
|
||||
|
||||
<p><strong>Parameters:</strong></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
<th>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>inputs</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Iterable">Iterable</span>[<span title="typing.Any">Any</span>]</code>
|
||||
</td>
|
||||
<td><p>The inputs. (X in scikit-learn)</p></td>
|
||||
<td>
|
||||
<em>required</em>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>expected_outputs</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Iterable">Iterable</span>[<span title="great_ai.tracing.add_ground_truth.T">T</span>]</code>
|
||||
</td>
|
||||
<td><p>The ground-truth values corresponding to the inputs. (y in
|
||||
scikit-learn)</p></td>
|
||||
<td>
|
||||
<em>required</em>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>tags</code></td>
|
||||
<td>
|
||||
<code><span title="typing.List">List</span>[str]</code>
|
||||
</td>
|
||||
<td><p>A single tag or a list of tags to append to each generated trace's tags.</p></td>
|
||||
<td>
|
||||
<code>[]</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>train_split_ratio</code></td>
|
||||
<td>
|
||||
<code>float</code>
|
||||
</td>
|
||||
<td><p>The probability-weight of giving each trace the <code>train</code> tag.</p></td>
|
||||
<td>
|
||||
<code>1</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>test_split_ratio</code></td>
|
||||
<td>
|
||||
<code>float</code>
|
||||
</td>
|
||||
<td><p>The probability-weight of giving each trace the <code>test</code> tag.</p></td>
|
||||
<td>
|
||||
<code>0</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>validation_split_ratio</code></td>
|
||||
<td>
|
||||
<code>float</code>
|
||||
</td>
|
||||
<td><p>The probability-weight of giving each trace the</p></td>
|
||||
<td>
|
||||
<code>0</code>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<details class="quote">
|
||||
<summary>Source code in <code>great_ai/tracing/add_ground_truth.py</code></summary>
|
||||
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-0-19"> 19</a></span>
|
||||
|
|
@ -3325,7 +3448,37 @@ Returns:
|
|||
<span class="normal"><a href="#__codelineno-0-69"> 69</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-70"> 70</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-71"> 71</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-72">72</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-19" name="__codelineno-0-19"></a><span class="k">def</span> <span class="nf">add_ground_truth</span><span class="p">(</span>
|
||||
<span class="normal"><a href="#__codelineno-0-72"> 72</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-73"> 73</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-74"> 74</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-75"> 75</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-76"> 76</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-77"> 77</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-78"> 78</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-79"> 79</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-80"> 80</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-81"> 81</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-82"> 82</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-83"> 83</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-84"> 84</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-85"> 85</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-86"> 86</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-87"> 87</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-88"> 88</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-89"> 89</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-90"> 90</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-91"> 91</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-92"> 92</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-93"> 93</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-94"> 94</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-95"> 95</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-96"> 96</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-97"> 97</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-98"> 98</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-99"> 99</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-100">100</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-101">101</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-102">102</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-19" name="__codelineno-0-19"></a><span class="k">def</span> <span class="nf">add_ground_truth</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a> <span class="n">inputs</span><span class="p">:</span> <span class="n">Iterable</span><span class="p">[</span><span class="n">Any</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a> <span class="n">expected_outputs</span><span class="p">:</span> <span class="n">Iterable</span><span class="p">[</span><span class="n">T</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a> <span class="o">*</span><span class="p">,</span>
|
||||
|
|
@ -3334,51 +3487,81 @@ Returns:
|
|||
<a id="__codelineno-0-25" name="__codelineno-0-25"></a> <span class="n">test_split_ratio</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-26" name="__codelineno-0-26"></a> <span class="n">validation_split_ratio</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mi">0</span>
|
||||
<a id="__codelineno-0-27" name="__codelineno-0-27"></a><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-28" name="__codelineno-0-28"></a> <span class="n">get_context</span><span class="p">()</span> <span class="c1"># this resets the seed</span>
|
||||
<a id="__codelineno-0-28" name="__codelineno-0-28"></a> <span class="sd">"""Add training data (with optional train-test splitting).</span>
|
||||
<a id="__codelineno-0-29" name="__codelineno-0-29"></a>
|
||||
<a id="__codelineno-0-30" name="__codelineno-0-30"></a> <span class="n">inputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-31" name="__codelineno-0-31"></a> <span class="n">expected_outputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">expected_outputs</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-32" name="__codelineno-0-32"></a> <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-33" name="__codelineno-0-33"></a> <span class="n">expected_outputs</span>
|
||||
<a id="__codelineno-0-34" name="__codelineno-0-34"></a> <span class="p">),</span> <span class="s2">"The length of the inputs and expected_outputs must be equal"</span>
|
||||
<a id="__codelineno-0-35" name="__codelineno-0-35"></a>
|
||||
<a id="__codelineno-0-36" name="__codelineno-0-36"></a> <span class="n">sum_ratio</span> <span class="o">=</span> <span class="n">train_split_ratio</span> <span class="o">+</span> <span class="n">test_split_ratio</span> <span class="o">+</span> <span class="n">validation_split_ratio</span>
|
||||
<a id="__codelineno-0-37" name="__codelineno-0-37"></a> <span class="k">assert</span> <span class="n">sum_ratio</span> <span class="o">></span> <span class="mi">0</span><span class="p">,</span> <span class="s2">"The sum of the split ratios must be a positive number"</span>
|
||||
<a id="__codelineno-0-30" name="__codelineno-0-30"></a><span class="sd"> Add and tag datapoints, wrap them into traces. The `inputs` are available via the</span>
|
||||
<a id="__codelineno-0-31" name="__codelineno-0-31"></a><span class="sd"> `.input` property, while `expected_outputs` under both the `.output` and `.feedback`</span>
|
||||
<a id="__codelineno-0-32" name="__codelineno-0-32"></a><span class="sd"> properties.</span>
|
||||
<a id="__codelineno-0-33" name="__codelineno-0-33"></a>
|
||||
<a id="__codelineno-0-34" name="__codelineno-0-34"></a><span class="sd"> All generated traces are tagged with `ground_truth` by default. Additional tags can</span>
|
||||
<a id="__codelineno-0-35" name="__codelineno-0-35"></a><span class="sd"> be also provided. Using the `split_ratio` arguments, tags can be given randomly with</span>
|
||||
<a id="__codelineno-0-36" name="__codelineno-0-36"></a><span class="sd"> a user-defined distribution. Only the ratio of the splits matter, they don't have to</span>
|
||||
<a id="__codelineno-0-37" name="__codelineno-0-37"></a><span class="sd"> add up to 1.</span>
|
||||
<a id="__codelineno-0-38" name="__codelineno-0-38"></a>
|
||||
<a id="__codelineno-0-39" name="__codelineno-0-39"></a> <span class="n">train_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-40" name="__codelineno-0-40"></a> <span class="n">test_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-41" name="__codelineno-0-41"></a> <span class="n">validation_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-42" name="__codelineno-0-42"></a>
|
||||
<a id="__codelineno-0-43" name="__codelineno-0-43"></a> <span class="n">values</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">expected_outputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-44" name="__codelineno-0-44"></a> <span class="n">shuffle</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-45" name="__codelineno-0-45"></a>
|
||||
<a id="__codelineno-0-46" name="__codelineno-0-46"></a> <span class="n">split_tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-47" name="__codelineno-0-47"></a> <span class="p">[</span><span class="n">TRAIN_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">train_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-48" name="__codelineno-0-48"></a> <span class="o">+</span> <span class="p">[</span><span class="n">TEST_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">test_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-49" name="__codelineno-0-49"></a> <span class="o">+</span> <span class="p">[</span><span class="n">VALIDATION_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">validation_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-50" name="__codelineno-0-50"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-51" name="__codelineno-0-51"></a> <span class="n">shuffle</span><span class="p">(</span><span class="n">split_tags</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-52" name="__codelineno-0-52"></a>
|
||||
<a id="__codelineno-0-53" name="__codelineno-0-53"></a> <span class="n">created</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-54" name="__codelineno-0-54"></a> <span class="n">traces</span> <span class="o">=</span> <span class="p">[</span>
|
||||
<a id="__codelineno-0-55" name="__codelineno-0-55"></a> <span class="n">cast</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-56" name="__codelineno-0-56"></a> <span class="n">Trace</span><span class="p">[</span><span class="n">T</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-57" name="__codelineno-0-57"></a> <span class="n">Trace</span><span class="p">(</span> <span class="c1"># avoid ValueError: "Trace" object has no field "__orig_class__"</span>
|
||||
<a id="__codelineno-0-58" name="__codelineno-0-58"></a> <span class="n">trace_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">uuid4</span><span class="p">()),</span>
|
||||
<a id="__codelineno-0-59" name="__codelineno-0-59"></a> <span class="n">created</span><span class="o">=</span><span class="n">created</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-60" name="__codelineno-0-60"></a> <span class="n">original_execution_time_ms</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-61" name="__codelineno-0-61"></a> <span class="n">logged_values</span><span class="o">=</span><span class="n">X</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">else</span> <span class="p">{</span><span class="s2">"input"</span><span class="p">:</span> <span class="n">X</span><span class="p">},</span>
|
||||
<a id="__codelineno-0-62" name="__codelineno-0-62"></a> <span class="n">models</span><span class="o">=</span><span class="p">[],</span>
|
||||
<a id="__codelineno-0-63" name="__codelineno-0-63"></a> <span class="n">output</span><span class="o">=</span><span class="n">y</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-64" name="__codelineno-0-64"></a> <span class="n">feedback</span><span class="o">=</span><span class="n">y</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-65" name="__codelineno-0-65"></a> <span class="n">exception</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-66" name="__codelineno-0-66"></a> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="n">GROUND_TRUTH_TAG_NAME</span><span class="p">,</span> <span class="n">split_tag</span><span class="p">,</span> <span class="o">*</span><span class="n">tags</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-67" name="__codelineno-0-67"></a> <span class="p">),</span>
|
||||
<a id="__codelineno-0-68" name="__codelineno-0-68"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-69" name="__codelineno-0-69"></a> <span class="k">for</span> <span class="p">((</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">),</span> <span class="n">split_tag</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">split_tags</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-70" name="__codelineno-0-70"></a> <span class="p">]</span>
|
||||
<a id="__codelineno-0-71" name="__codelineno-0-71"></a>
|
||||
<a id="__codelineno-0-72" name="__codelineno-0-72"></a> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span><span class="o">.</span><span class="n">save_batch</span><span class="p">(</span><span class="n">traces</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-39" name="__codelineno-0-39"></a><span class="sd"> Examples:</span>
|
||||
<a id="__codelineno-0-40" name="__codelineno-0-40"></a><span class="sd"> >>> add_ground_truth(</span>
|
||||
<a id="__codelineno-0-41" name="__codelineno-0-41"></a><span class="sd"> ... [1, 2, 3],</span>
|
||||
<a id="__codelineno-0-42" name="__codelineno-0-42"></a><span class="sd"> ... ['odd', 'even', 'odd'],</span>
|
||||
<a id="__codelineno-0-43" name="__codelineno-0-43"></a><span class="sd"> ... tags='my_tag',</span>
|
||||
<a id="__codelineno-0-44" name="__codelineno-0-44"></a><span class="sd"> ... train_split_ratio=1,</span>
|
||||
<a id="__codelineno-0-45" name="__codelineno-0-45"></a><span class="sd"> ... test_split_ratio=1,</span>
|
||||
<a id="__codelineno-0-46" name="__codelineno-0-46"></a><span class="sd"> ... validation_split_ratio=0.5,</span>
|
||||
<a id="__codelineno-0-47" name="__codelineno-0-47"></a><span class="sd"> ... )</span>
|
||||
<a id="__codelineno-0-48" name="__codelineno-0-48"></a>
|
||||
<a id="__codelineno-0-49" name="__codelineno-0-49"></a><span class="sd"> Args:</span>
|
||||
<a id="__codelineno-0-50" name="__codelineno-0-50"></a><span class="sd"> inputs: The inputs. (X in scikit-learn)</span>
|
||||
<a id="__codelineno-0-51" name="__codelineno-0-51"></a><span class="sd"> expected_outputs: The ground-truth values corresponding to the inputs. (y in</span>
|
||||
<a id="__codelineno-0-52" name="__codelineno-0-52"></a><span class="sd"> scikit-learn)</span>
|
||||
<a id="__codelineno-0-53" name="__codelineno-0-53"></a><span class="sd"> tags: A single tag or a list of tags to append to each generated trace's tags.</span>
|
||||
<a id="__codelineno-0-54" name="__codelineno-0-54"></a><span class="sd"> train_split_ratio: The probability-weight of giving each trace the `train` tag.</span>
|
||||
<a id="__codelineno-0-55" name="__codelineno-0-55"></a><span class="sd"> test_split_ratio: The probability-weight of giving each trace the `test` tag.</span>
|
||||
<a id="__codelineno-0-56" name="__codelineno-0-56"></a><span class="sd"> validation_split_ratio: The probability-weight of giving each trace the</span>
|
||||
<a id="__codelineno-0-57" name="__codelineno-0-57"></a><span class="sd"> `validation` tag.</span>
|
||||
<a id="__codelineno-0-58" name="__codelineno-0-58"></a><span class="sd"> """</span>
|
||||
<a id="__codelineno-0-59" name="__codelineno-0-59"></a>
|
||||
<a id="__codelineno-0-60" name="__codelineno-0-60"></a> <span class="n">inputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-61" name="__codelineno-0-61"></a> <span class="n">expected_outputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">expected_outputs</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-62" name="__codelineno-0-62"></a> <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-63" name="__codelineno-0-63"></a> <span class="n">expected_outputs</span>
|
||||
<a id="__codelineno-0-64" name="__codelineno-0-64"></a> <span class="p">),</span> <span class="s2">"The length of the inputs and expected_outputs must be equal"</span>
|
||||
<a id="__codelineno-0-65" name="__codelineno-0-65"></a>
|
||||
<a id="__codelineno-0-66" name="__codelineno-0-66"></a> <span class="n">sum_ratio</span> <span class="o">=</span> <span class="n">train_split_ratio</span> <span class="o">+</span> <span class="n">test_split_ratio</span> <span class="o">+</span> <span class="n">validation_split_ratio</span>
|
||||
<a id="__codelineno-0-67" name="__codelineno-0-67"></a> <span class="k">assert</span> <span class="n">sum_ratio</span> <span class="o">></span> <span class="mi">0</span><span class="p">,</span> <span class="s2">"The sum of the split ratios must be a positive number"</span>
|
||||
<a id="__codelineno-0-68" name="__codelineno-0-68"></a>
|
||||
<a id="__codelineno-0-69" name="__codelineno-0-69"></a> <span class="n">train_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-70" name="__codelineno-0-70"></a> <span class="n">test_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-71" name="__codelineno-0-71"></a> <span class="n">validation_split_ratio</span> <span class="o">/=</span> <span class="n">sum_ratio</span>
|
||||
<a id="__codelineno-0-72" name="__codelineno-0-72"></a>
|
||||
<a id="__codelineno-0-73" name="__codelineno-0-73"></a> <span class="n">values</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">expected_outputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-74" name="__codelineno-0-74"></a> <span class="n">shuffle</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-75" name="__codelineno-0-75"></a>
|
||||
<a id="__codelineno-0-76" name="__codelineno-0-76"></a> <span class="n">split_tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-77" name="__codelineno-0-77"></a> <span class="p">[</span><span class="n">TRAIN_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">train_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-78" name="__codelineno-0-78"></a> <span class="o">+</span> <span class="p">[</span><span class="n">TEST_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">test_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-79" name="__codelineno-0-79"></a> <span class="o">+</span> <span class="p">[</span><span class="n">VALIDATION_SPLIT_TAG_NAME</span><span class="p">]</span> <span class="o">*</span> <span class="n">ceil</span><span class="p">(</span><span class="n">validation_split_ratio</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">))</span>
|
||||
<a id="__codelineno-0-80" name="__codelineno-0-80"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-81" name="__codelineno-0-81"></a> <span class="n">shuffle</span><span class="p">(</span><span class="n">split_tags</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-82" name="__codelineno-0-82"></a>
|
||||
<a id="__codelineno-0-83" name="__codelineno-0-83"></a> <span class="n">created</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-84" name="__codelineno-0-84"></a> <span class="n">traces</span> <span class="o">=</span> <span class="p">[</span>
|
||||
<a id="__codelineno-0-85" name="__codelineno-0-85"></a> <span class="n">cast</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-86" name="__codelineno-0-86"></a> <span class="n">Trace</span><span class="p">[</span><span class="n">T</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-87" name="__codelineno-0-87"></a> <span class="n">Trace</span><span class="p">(</span> <span class="c1"># avoid ValueError: "Trace" object has no field "__orig_class__"</span>
|
||||
<a id="__codelineno-0-88" name="__codelineno-0-88"></a> <span class="n">trace_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">uuid4</span><span class="p">()),</span>
|
||||
<a id="__codelineno-0-89" name="__codelineno-0-89"></a> <span class="n">created</span><span class="o">=</span><span class="n">created</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-90" name="__codelineno-0-90"></a> <span class="n">original_execution_time_ms</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-91" name="__codelineno-0-91"></a> <span class="n">logged_values</span><span class="o">=</span><span class="n">X</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">else</span> <span class="p">{</span><span class="s2">"input"</span><span class="p">:</span> <span class="n">X</span><span class="p">},</span>
|
||||
<a id="__codelineno-0-92" name="__codelineno-0-92"></a> <span class="n">models</span><span class="o">=</span><span class="p">[],</span>
|
||||
<a id="__codelineno-0-93" name="__codelineno-0-93"></a> <span class="n">output</span><span class="o">=</span><span class="n">y</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-94" name="__codelineno-0-94"></a> <span class="n">feedback</span><span class="o">=</span><span class="n">y</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-95" name="__codelineno-0-95"></a> <span class="n">exception</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-96" name="__codelineno-0-96"></a> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="n">GROUND_TRUTH_TAG_NAME</span><span class="p">,</span> <span class="n">split_tag</span><span class="p">,</span> <span class="o">*</span><span class="n">tags</span><span class="p">],</span>
|
||||
<a id="__codelineno-0-97" name="__codelineno-0-97"></a> <span class="p">),</span>
|
||||
<a id="__codelineno-0-98" name="__codelineno-0-98"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-99" name="__codelineno-0-99"></a> <span class="k">for</span> <span class="p">((</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">),</span> <span class="n">split_tag</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">split_tags</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-100" name="__codelineno-0-100"></a> <span class="p">]</span>
|
||||
<a id="__codelineno-0-101" name="__codelineno-0-101"></a>
|
||||
<a id="__codelineno-0-102" name="__codelineno-0-102"></a> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span><span class="o">.</span><span class="n">save_batch</span><span class="p">(</span><span class="n">traces</span><span class="p">)</span>
|
||||
</code></pre></div></td></tr></table></div>
|
||||
</details>
|
||||
</div>
|
||||
|
|
@ -3390,13 +3573,83 @@ Returns:
|
|||
|
||||
|
||||
<h3 id="great_ai.query_ground_truth" class="doc doc-heading">
|
||||
<code class="highlight language-python"><span class="n">query_ground_truth</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="o">=</span><span class="p">[],</span> <span class="o">*</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">return_max_count</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span></code>
|
||||
<code class="highlight language-python"><span class="n">query_ground_truth</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="o">=</span><span class="p">[],</span> <span class="o">*</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">return_max_count</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span></code>
|
||||
|
||||
<a href="#great_ai.query_ground_truth" class="headerlink" title="Permanent link">#</a></h3>
|
||||
|
||||
|
||||
<div class="doc doc-contents first">
|
||||
|
||||
<p>Return training samples.</p>
|
||||
<p>Combines, filters, and returns datapoints that have been either added by
|
||||
<code>add_ground_truth</code> or were the result of a prediction after which their trace got
|
||||
a feedback through the RESP API-s <code>/traces/{trace_id}/feedback</code> endpoint
|
||||
(end-to-end feedback).</p>
|
||||
<p>Filtering can be used to only return points matching all given tags (or the single
|
||||
given tag) and by time of creation.</p>
|
||||
|
||||
<p><strong>Examples:</strong></p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="gp">>>> </span><span class="n">query_ground_truth</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a><span class="go">[...]</span>
|
||||
</code></pre></div>
|
||||
|
||||
<p><strong>Parameters:</strong></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
<th>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>conjunctive_tags</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Union">Union</span>[<span title="typing.List">List</span>[str], str]</code>
|
||||
</td>
|
||||
<td><p>Single tag or a list of tags which the returned traces have to
|
||||
match. The relationship between the tags is conjunctive (AND).</p></td>
|
||||
<td>
|
||||
<code>[]</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>since</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Optional">Optional</span>[<span title="datetime.datetime">datetime</span>]</code>
|
||||
</td>
|
||||
<td><p>Only return traces created after the given timestamp. <code>None</code> means no
|
||||
filtering.</p></td>
|
||||
<td>
|
||||
<code>None</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>until</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Optional">Optional</span>[<span title="datetime.datetime">datetime</span>]</code>
|
||||
</td>
|
||||
<td><p>Only return traces created before the given timestamp. <code>None</code> means no
|
||||
filtering.</p></td>
|
||||
<td>
|
||||
<code>None</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>return_max_count</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Optional">Optional</span>[int]</code>
|
||||
</td>
|
||||
<td><p>Return at-most this many traces. (take, limit)</p></td>
|
||||
<td>
|
||||
<code>None</code>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<details class="quote">
|
||||
<summary>Source code in <code>great_ai/tracing/query_ground_truth.py</code></summary>
|
||||
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-0-8"> 8</a></span>
|
||||
|
|
@ -3413,21 +3666,79 @@ Returns:
|
|||
<span class="normal"><a href="#__codelineno-0-19">19</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-20">20</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-21">21</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-22">22</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-8" name="__codelineno-0-8"></a><span class="k">def</span> <span class="nf">query_ground_truth</span><span class="p">(</span>
|
||||
<span class="normal"><a href="#__codelineno-0-22">22</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-23">23</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-24">24</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-25">25</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-26">26</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-27">27</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-28">28</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-29">29</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-30">30</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-31">31</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-32">32</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-33">33</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-34">34</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-35">35</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-36">36</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-37">37</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-38">38</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-39">39</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-40">40</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-41">41</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-42">42</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-43">43</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-44">44</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-45">45</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-46">46</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-47">47</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-48">48</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-49">49</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-50">50</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-51">51</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-8" name="__codelineno-0-8"></a><span class="k">def</span> <span class="nf">query_ground_truth</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-9" name="__codelineno-0-9"></a> <span class="n">conjunctive_tags</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[],</span>
|
||||
<a id="__codelineno-0-10" name="__codelineno-0-10"></a> <span class="o">*</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-11" name="__codelineno-0-11"></a> <span class="n">since</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12"></a> <span class="n">return_max_count</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Trace</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a> <span class="n">tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a> <span class="n">conjunctive_tags</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">conjunctive_tags</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-17" name="__codelineno-0-17"></a> <span class="n">db</span> <span class="o">=</span> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a> <span class="n">items</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a> <span class="n">conjunctive_tags</span><span class="o">=</span><span class="n">tags</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="n">since</span><span class="p">,</span> <span class="n">take</span><span class="o">=</span><span class="n">return_max_count</span><span class="p">,</span> <span class="n">has_feedback</span><span class="o">=</span><span class="kc">True</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a> <span class="k">return</span> <span class="n">items</span>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12"></a> <span class="n">until</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a> <span class="n">return_max_count</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="n">Trace</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a> <span class="sd">"""Return training samples.</span>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a>
|
||||
<a id="__codelineno-0-17" name="__codelineno-0-17"></a><span class="sd"> Combines, filters, and returns datapoints that have been either added by</span>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a><span class="sd"> `add_ground_truth` or were the result of a prediction after which their trace got</span>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a><span class="sd"> a feedback through the RESP API-s `/traces/{trace_id}/feedback` endpoint</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a><span class="sd"> (end-to-end feedback).</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a><span class="sd"> Filtering can be used to only return points matching all given tags (or the single</span>
|
||||
<a id="__codelineno-0-23" name="__codelineno-0-23"></a><span class="sd"> given tag) and by time of creation.</span>
|
||||
<a id="__codelineno-0-24" name="__codelineno-0-24"></a>
|
||||
<a id="__codelineno-0-25" name="__codelineno-0-25"></a><span class="sd"> Examples:</span>
|
||||
<a id="__codelineno-0-26" name="__codelineno-0-26"></a><span class="sd"> >>> query_ground_truth()</span>
|
||||
<a id="__codelineno-0-27" name="__codelineno-0-27"></a><span class="sd"> [...]</span>
|
||||
<a id="__codelineno-0-28" name="__codelineno-0-28"></a>
|
||||
<a id="__codelineno-0-29" name="__codelineno-0-29"></a><span class="sd"> Args:</span>
|
||||
<a id="__codelineno-0-30" name="__codelineno-0-30"></a><span class="sd"> conjunctive_tags: Single tag or a list of tags which the returned traces have to</span>
|
||||
<a id="__codelineno-0-31" name="__codelineno-0-31"></a><span class="sd"> match. The relationship between the tags is conjunctive (AND).</span>
|
||||
<a id="__codelineno-0-32" name="__codelineno-0-32"></a><span class="sd"> since: Only return traces created after the given timestamp. `None` means no</span>
|
||||
<a id="__codelineno-0-33" name="__codelineno-0-33"></a><span class="sd"> filtering.</span>
|
||||
<a id="__codelineno-0-34" name="__codelineno-0-34"></a><span class="sd"> until: Only return traces created before the given timestamp. `None` means no</span>
|
||||
<a id="__codelineno-0-35" name="__codelineno-0-35"></a><span class="sd"> filtering.</span>
|
||||
<a id="__codelineno-0-36" name="__codelineno-0-36"></a><span class="sd"> return_max_count: Return at-most this many traces. (take, limit)</span>
|
||||
<a id="__codelineno-0-37" name="__codelineno-0-37"></a><span class="sd"> """</span>
|
||||
<a id="__codelineno-0-38" name="__codelineno-0-38"></a>
|
||||
<a id="__codelineno-0-39" name="__codelineno-0-39"></a> <span class="n">tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-40" name="__codelineno-0-40"></a> <span class="n">conjunctive_tags</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">conjunctive_tags</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-41" name="__codelineno-0-41"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-42" name="__codelineno-0-42"></a> <span class="n">db</span> <span class="o">=</span> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span>
|
||||
<a id="__codelineno-0-43" name="__codelineno-0-43"></a>
|
||||
<a id="__codelineno-0-44" name="__codelineno-0-44"></a> <span class="n">items</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-45" name="__codelineno-0-45"></a> <span class="n">conjunctive_tags</span><span class="o">=</span><span class="n">tags</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-46" name="__codelineno-0-46"></a> <span class="n">since</span><span class="o">=</span><span class="n">since</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-47" name="__codelineno-0-47"></a> <span class="n">until</span><span class="o">=</span><span class="n">until</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-48" name="__codelineno-0-48"></a> <span class="n">take</span><span class="o">=</span><span class="n">return_max_count</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-49" name="__codelineno-0-49"></a> <span class="n">has_feedback</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-50" name="__codelineno-0-50"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-51" name="__codelineno-0-51"></a> <span class="k">return</span> <span class="n">items</span>
|
||||
</code></pre></div></td></tr></table></div>
|
||||
</details>
|
||||
</div>
|
||||
|
|
@ -3439,13 +3750,69 @@ Returns:
|
|||
|
||||
|
||||
<h3 id="great_ai.delete_ground_truth" class="doc doc-heading">
|
||||
<code class="highlight language-python"><span class="n">delete_ground_truth</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="o">=</span><span class="p">[],</span> <span class="o">*</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span></code>
|
||||
<code class="highlight language-python"><span class="n">delete_ground_truth</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="o">=</span><span class="p">[],</span> <span class="o">*</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span></code>
|
||||
|
||||
<a href="#great_ai.delete_ground_truth" class="headerlink" title="Permanent link">#</a></h3>
|
||||
|
||||
|
||||
<div class="doc doc-contents first">
|
||||
|
||||
<p>Delete traces matching the given criteria.</p>
|
||||
<p>Takes the same arguments as <code>query_ground_truth</code> but instead of returning them,
|
||||
it simply deletes them.</p>
|
||||
<p>You can rely on the efficiency of the delete's implementation.</p>
|
||||
|
||||
<p><strong>Examples:</strong></p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="gp">>>> </span><span class="n">delete_ground_truth</span><span class="p">([</span><span class="s1">'train'</span><span class="p">,</span> <span class="s1">'test'</span><span class="p">,</span> <span class="s1">'validation'</span><span class="p">])</span>
|
||||
</code></pre></div>
|
||||
|
||||
<p><strong>Parameters:</strong></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
<th>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>conjunctive_tags</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Union">Union</span>[<span title="typing.List">List</span>[str], str]</code>
|
||||
</td>
|
||||
<td><p>Single tag or a list of tags which the deleted traces have to
|
||||
match. The relationship between the tags is conjunctive (AND).</p></td>
|
||||
<td>
|
||||
<code>[]</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>since</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Optional">Optional</span>[<span title="datetime.datetime">datetime</span>]</code>
|
||||
</td>
|
||||
<td><p>Only delete traces created after the given timestamp. <code>None</code> means no
|
||||
filtering.</p></td>
|
||||
<td>
|
||||
<code>None</code>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>until</code></td>
|
||||
<td>
|
||||
<code><span title="typing.Optional">Optional</span>[<span title="datetime.datetime">datetime</span>]</code>
|
||||
</td>
|
||||
<td><p>Only delete traces created before the given timestamp. <code>None</code> means no
|
||||
filtering.</p></td>
|
||||
<td>
|
||||
<code>None</code>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<details class="quote">
|
||||
<summary>Source code in <code>great_ai/tracing/delete_ground_truth.py</code></summary>
|
||||
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span></span><span class="normal"><a href="#__codelineno-0-7"> 7</a></span>
|
||||
|
|
@ -3463,22 +3830,60 @@ Returns:
|
|||
<span class="normal"><a href="#__codelineno-0-19">19</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-20">20</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-21">21</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-22">22</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-7" name="__codelineno-0-7"></a><span class="k">def</span> <span class="nf">delete_ground_truth</span><span class="p">(</span>
|
||||
<span class="normal"><a href="#__codelineno-0-22">22</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-23">23</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-24">24</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-25">25</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-26">26</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-27">27</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-28">28</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-29">29</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-30">30</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-31">31</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-32">32</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-33">33</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-34">34</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-35">35</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-36">36</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-37">37</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-38">38</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-39">39</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-40">40</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-41">41</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-7" name="__codelineno-0-7"></a><span class="k">def</span> <span class="nf">delete_ground_truth</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-8" name="__codelineno-0-8"></a> <span class="n">conjunctive_tags</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[],</span>
|
||||
<a id="__codelineno-0-9" name="__codelineno-0-9"></a> <span class="o">*</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-10" name="__codelineno-0-10"></a> <span class="n">until</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-11" name="__codelineno-0-11"></a> <span class="n">since</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-10" name="__codelineno-0-10"></a> <span class="n">since</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-11" name="__codelineno-0-11"></a> <span class="n">until</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12"></a><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a> <span class="n">tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a> <span class="n">conjunctive_tags</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">conjunctive_tags</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a> <span class="n">db</span> <span class="o">=</span> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a> <span class="sd">"""Delete traces matching the given criteria.</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a><span class="sd"> Takes the same arguments as `query_ground_truth` but instead of returning them,</span>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a><span class="sd"> it simply deletes them.</span>
|
||||
<a id="__codelineno-0-17" name="__codelineno-0-17"></a>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a> <span class="n">items</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a> <span class="n">conjunctive_tags</span><span class="o">=</span><span class="n">tags</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="n">until</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="n">since</span><span class="p">,</span> <span class="n">has_feedback</span><span class="o">=</span><span class="kc">True</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a> <span class="n">db</span><span class="o">.</span><span class="n">delete_batch</span><span class="p">([</span><span class="n">i</span><span class="o">.</span><span class="n">trace_id</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">items</span><span class="p">])</span>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a><span class="sd"> You can rely on the efficiency of the delete's implementation.</span>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a><span class="sd"> Examples:</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a><span class="sd"> >>> delete_ground_truth(['train', 'test', 'validation'])</span>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a>
|
||||
<a id="__codelineno-0-23" name="__codelineno-0-23"></a><span class="sd"> Args:</span>
|
||||
<a id="__codelineno-0-24" name="__codelineno-0-24"></a><span class="sd"> conjunctive_tags: Single tag or a list of tags which the deleted traces have to</span>
|
||||
<a id="__codelineno-0-25" name="__codelineno-0-25"></a><span class="sd"> match. The relationship between the tags is conjunctive (AND).</span>
|
||||
<a id="__codelineno-0-26" name="__codelineno-0-26"></a><span class="sd"> since: Only delete traces created after the given timestamp. `None` means no</span>
|
||||
<a id="__codelineno-0-27" name="__codelineno-0-27"></a><span class="sd"> filtering.</span>
|
||||
<a id="__codelineno-0-28" name="__codelineno-0-28"></a><span class="sd"> until: Only delete traces created before the given timestamp. `None` means no</span>
|
||||
<a id="__codelineno-0-29" name="__codelineno-0-29"></a><span class="sd"> filtering.</span>
|
||||
<a id="__codelineno-0-30" name="__codelineno-0-30"></a><span class="sd"> """</span>
|
||||
<a id="__codelineno-0-31" name="__codelineno-0-31"></a>
|
||||
<a id="__codelineno-0-32" name="__codelineno-0-32"></a> <span class="n">tags</span> <span class="o">=</span> <span class="p">(</span>
|
||||
<a id="__codelineno-0-33" name="__codelineno-0-33"></a> <span class="n">conjunctive_tags</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">conjunctive_tags</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">conjunctive_tags</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-34" name="__codelineno-0-34"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-35" name="__codelineno-0-35"></a> <span class="n">db</span> <span class="o">=</span> <span class="n">get_context</span><span class="p">()</span><span class="o">.</span><span class="n">tracing_database</span>
|
||||
<a id="__codelineno-0-36" name="__codelineno-0-36"></a>
|
||||
<a id="__codelineno-0-37" name="__codelineno-0-37"></a> <span class="n">items</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-38" name="__codelineno-0-38"></a> <span class="n">conjunctive_tags</span><span class="o">=</span><span class="n">tags</span><span class="p">,</span> <span class="n">until</span><span class="o">=</span><span class="n">until</span><span class="p">,</span> <span class="n">since</span><span class="o">=</span><span class="n">since</span><span class="p">,</span> <span class="n">has_feedback</span><span class="o">=</span><span class="kc">True</span>
|
||||
<a id="__codelineno-0-39" name="__codelineno-0-39"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-40" name="__codelineno-0-40"></a>
|
||||
<a id="__codelineno-0-41" name="__codelineno-0-41"></a> <span class="n">db</span><span class="o">.</span><span class="n">delete_batch</span><span class="p">([</span><span class="n">i</span><span class="o">.</span><span class="n">trace_id</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">items</span><span class="p">])</span>
|
||||
</code></pre></div></td></tr></table></div>
|
||||
</details>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -665,7 +692,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -679,7 +706,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -693,7 +720,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -797,6 +824,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/reference/large-file.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -772,7 +799,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -786,7 +813,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -800,7 +827,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -1011,6 +1038,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/reference/utilities.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -3161,6 +3189,7 @@ which is between 1 and <code>chunk_size</code>.</p>
|
|||
<p>Turn a stream of chunks of items into a stream of items (flatten operation).</p>
|
||||
<p>The inverse operation of <a class="autorefs autorefs-internal" href="#great_ai.utilities.chunk.chunk">chunk</a>.
|
||||
Useful for parallel processing.</p>
|
||||
<p>Similar to itertools.chain but ignores <code>None</code> chunks.</p>
|
||||
|
||||
<p><strong>Examples:</strong></p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">unchunk</span><span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="p">[</span><span class="mi">6</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="p">[</span><span class="mi">9</span><span class="p">]]))</span>
|
||||
|
|
@ -3230,26 +3259,30 @@ Useful for parallel processing.</p>
|
|||
<span class="normal"><a href="#__codelineno-0-22">22</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-23">23</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-24">24</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-25">25</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-6" name="__codelineno-0-6"></a><span class="k">def</span> <span class="nf">unchunk</span><span class="p">(</span><span class="n">chunks</span><span class="p">:</span> <span class="n">Iterable</span><span class="p">[</span><span class="n">Optional</span><span class="p">[</span><span class="n">Iterable</span><span class="p">[</span><span class="n">T</span><span class="p">]]])</span> <span class="o">-></span> <span class="n">Iterable</span><span class="p">[</span><span class="n">T</span><span class="p">]:</span>
|
||||
<span class="normal"><a href="#__codelineno-0-25">25</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-26">26</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-27">27</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-6" name="__codelineno-0-6"></a><span class="k">def</span> <span class="nf">unchunk</span><span class="p">(</span><span class="n">chunks</span><span class="p">:</span> <span class="n">Iterable</span><span class="p">[</span><span class="n">Optional</span><span class="p">[</span><span class="n">Iterable</span><span class="p">[</span><span class="n">T</span><span class="p">]]])</span> <span class="o">-></span> <span class="n">Iterable</span><span class="p">[</span><span class="n">T</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-7" name="__codelineno-0-7"></a> <span class="sd">"""Turn a stream of chunks of items into a stream of items (flatten operation).</span>
|
||||
<a id="__codelineno-0-8" name="__codelineno-0-8"></a>
|
||||
<a id="__codelineno-0-9" name="__codelineno-0-9"></a><span class="sd"> The inverse operation of [chunk][great_ai.utilities.chunk.chunk].</span>
|
||||
<a id="__codelineno-0-10" name="__codelineno-0-10"></a><span class="sd"> Useful for parallel processing.</span>
|
||||
<a id="__codelineno-0-11" name="__codelineno-0-11"></a>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12"></a><span class="sd"> Examples:</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a><span class="sd"> >>> list(unchunk([[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]))</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a><span class="sd"> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a><span class="sd"> Args:</span>
|
||||
<a id="__codelineno-0-17" name="__codelineno-0-17"></a><span class="sd"> chunks: Stream of chunks to unpack.</span>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a><span class="sd"> Yields:</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a><span class="sd"> The next item in the flattened iterable.</span>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a><span class="sd"> """</span>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a>
|
||||
<a id="__codelineno-0-23" name="__codelineno-0-23"></a> <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">chunks</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-24" name="__codelineno-0-24"></a> <span class="k">if</span> <span class="n">chunk</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-25" name="__codelineno-0-25"></a> <span class="k">yield from</span> <span class="n">chunk</span>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12"></a><span class="sd"> Similar to itertools.chain but ignores `None` chunks.</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a><span class="sd"> Examples:</span>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a><span class="sd"> >>> list(unchunk([[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]))</span>
|
||||
<a id="__codelineno-0-16" name="__codelineno-0-16"></a><span class="sd"> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span>
|
||||
<a id="__codelineno-0-17" name="__codelineno-0-17"></a>
|
||||
<a id="__codelineno-0-18" name="__codelineno-0-18"></a><span class="sd"> Args:</span>
|
||||
<a id="__codelineno-0-19" name="__codelineno-0-19"></a><span class="sd"> chunks: Stream of chunks to unpack.</span>
|
||||
<a id="__codelineno-0-20" name="__codelineno-0-20"></a>
|
||||
<a id="__codelineno-0-21" name="__codelineno-0-21"></a><span class="sd"> Yields:</span>
|
||||
<a id="__codelineno-0-22" name="__codelineno-0-22"></a><span class="sd"> The next item in the flattened iterable.</span>
|
||||
<a id="__codelineno-0-23" name="__codelineno-0-23"></a><span class="sd"> """</span>
|
||||
<a id="__codelineno-0-24" name="__codelineno-0-24"></a>
|
||||
<a id="__codelineno-0-25" name="__codelineno-0-25"></a> <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">chunks</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-26" name="__codelineno-0-26"></a> <span class="k">if</span> <span class="n">chunk</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-27" name="__codelineno-0-27"></a> <span class="k">yield from</span> <span class="n">chunk</span>
|
||||
</code></pre></div></td></tr></table></div>
|
||||
</details>
|
||||
</div>
|
||||
|
|
@ -3463,7 +3496,9 @@ with the <code>dict</code> interface.</p>
|
|||
<span class="normal"><a href="#__codelineno-0-155">155</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-156">156</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-157">157</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-158">158</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-12" name="__codelineno-0-12"></a><span class="k">class</span> <span class="nc">ConfigFile</span><span class="p">(</span><span class="n">Mapping</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]):</span>
|
||||
<span class="normal"><a href="#__codelineno-0-158">158</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-159">159</a></span>
|
||||
<span class="normal"><a href="#__codelineno-0-160">160</a></span></pre></div></td><td class="code"><div><pre><span></span><code><a id="__codelineno-0-12" name="__codelineno-0-12"></a><span class="k">class</span> <span class="nc">ConfigFile</span><span class="p">(</span><span class="n">Mapping</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]):</span>
|
||||
<a id="__codelineno-0-13" name="__codelineno-0-13"></a> <span class="sd">"""A small and safe `INI`-style configuration loader with `dict` and `ENV` support.</span>
|
||||
<a id="__codelineno-0-14" name="__codelineno-0-14"></a>
|
||||
<a id="__codelineno-0-15" name="__codelineno-0-15"></a><span class="sd"> The values can be accessed using both dot- and index-notation. It is compatible</span>
|
||||
|
|
@ -3549,67 +3584,69 @@ with the <code>dict</code> interface.</p>
|
|||
<a id="__codelineno-0-95" name="__codelineno-0-95"></a> <span class="n">value</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">v</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">values</span> <span class="k">if</span> <span class="n">v</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-96" name="__codelineno-0-96"></a> <span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-97" name="__codelineno-0-97"></a> <span class="k">raise</span> <span class="n">ParseError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-98" name="__codelineno-0-98"></a> <span class="sa">f</span><span class="s2">"Cannot parse config file (</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="o">.</span><span class="n">absolute</span><span class="p">()</span><span class="si">}</span><span class="s2">), error at key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">`"</span>
|
||||
<a id="__codelineno-0-99" name="__codelineno-0-99"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-100" name="__codelineno-0-100"></a>
|
||||
<a id="__codelineno-0-101" name="__codelineno-0-101"></a> <span class="n">already_exists</span> <span class="o">=</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span>
|
||||
<a id="__codelineno-0-102" name="__codelineno-0-102"></a> <span class="k">if</span> <span class="n">already_exists</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-103" name="__codelineno-0-103"></a> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span><span class="si">}</span><span class="s2">:"</span>
|
||||
<a id="__codelineno-0-104" name="__codelineno-0-104"></a> <span class="p">):</span>
|
||||
<a id="__codelineno-0-105" name="__codelineno-0-105"></a> <span class="k">raise</span> <span class="ne">KeyError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-106" name="__codelineno-0-106"></a> <span class="sa">f</span><span class="s2">"Key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` has been already defined and its value is `</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span><span class="si">}</span><span class="s2">`"</span>
|
||||
<a id="__codelineno-0-107" name="__codelineno-0-107"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-108" name="__codelineno-0-108"></a>
|
||||
<a id="__codelineno-0-109" name="__codelineno-0-109"></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span><span class="si">}</span><span class="s2">:"</span><span class="p">):</span>
|
||||
<a id="__codelineno-0-110" name="__codelineno-0-110"></a> <span class="n">_</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">":"</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-111" name="__codelineno-0-111"></a> <span class="k">if</span> <span class="n">value</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-112" name="__codelineno-0-112"></a> <span class="n">issue</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""The value of `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` contains the "</span><span class="si">{</span>
|
||||
<a id="__codelineno-0-113" name="__codelineno-0-113"></a> <span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span>
|
||||
<a id="__codelineno-0-114" name="__codelineno-0-114"></a> <span class="si">}</span><span class="s2">` prefix but `</span><span class="si">{</span><span class="n">value</span><span class="si">}</span><span class="s2">` is not defined as an environment variable"""</span>
|
||||
<a id="__codelineno-0-115" name="__codelineno-0-115"></a> <span class="k">if</span> <span class="n">already_exists</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-116" name="__codelineno-0-116"></a> <span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-117" name="__codelineno-0-117"></a> <span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">issue</span><span class="si">}</span><span class="s2">, using the default value defined above (`</span><span class="si">{</span>
|
||||
<a id="__codelineno-0-118" name="__codelineno-0-118"></a> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-119" name="__codelineno-0-119"></a> <span class="si">}</span><span class="s2">`)"""</span>
|
||||
<a id="__codelineno-0-120" name="__codelineno-0-120"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-121" name="__codelineno-0-121"></a> <span class="k">continue</span>
|
||||
<a id="__codelineno-0-122" name="__codelineno-0-122"></a> <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ignore_missing</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-123" name="__codelineno-0-123"></a> <span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="n">issue</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-124" name="__codelineno-0-124"></a> <span class="k">else</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-125" name="__codelineno-0-125"></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-126" name="__codelineno-0-126"></a> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">issue</span><span class="si">}</span><span class="s2"> and no default value has been provided"</span>
|
||||
<a id="__codelineno-0-127" name="__codelineno-0-127"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-128" name="__codelineno-0-128"></a> <span class="k">else</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-129" name="__codelineno-0-129"></a> <span class="n">value</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">value</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-130" name="__codelineno-0-130"></a>
|
||||
<a id="__codelineno-0-131" name="__codelineno-0-131"></a> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
|
||||
<a id="__codelineno-0-98" name="__codelineno-0-98"></a> <span class="sa">f</span><span class="s2">"""Cannot parse config file (</span><span class="si">{</span>
|
||||
<a id="__codelineno-0-99" name="__codelineno-0-99"></a> <span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="o">.</span><span class="n">absolute</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-100" name="__codelineno-0-100"></a> <span class="si">}</span><span class="s2">), error at key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">`"""</span>
|
||||
<a id="__codelineno-0-101" name="__codelineno-0-101"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-102" name="__codelineno-0-102"></a>
|
||||
<a id="__codelineno-0-103" name="__codelineno-0-103"></a> <span class="n">already_exists</span> <span class="o">=</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span>
|
||||
<a id="__codelineno-0-104" name="__codelineno-0-104"></a> <span class="k">if</span> <span class="n">already_exists</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">value</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-105" name="__codelineno-0-105"></a> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span><span class="si">}</span><span class="s2">:"</span>
|
||||
<a id="__codelineno-0-106" name="__codelineno-0-106"></a> <span class="p">):</span>
|
||||
<a id="__codelineno-0-107" name="__codelineno-0-107"></a> <span class="k">raise</span> <span class="ne">KeyError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-108" name="__codelineno-0-108"></a> <span class="sa">f</span><span class="s2">"Key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` has been already defined and its value is `</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span><span class="si">}</span><span class="s2">`"</span>
|
||||
<a id="__codelineno-0-109" name="__codelineno-0-109"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-110" name="__codelineno-0-110"></a>
|
||||
<a id="__codelineno-0-111" name="__codelineno-0-111"></a> <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span><span class="si">}</span><span class="s2">:"</span><span class="p">):</span>
|
||||
<a id="__codelineno-0-112" name="__codelineno-0-112"></a> <span class="n">_</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">":"</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-113" name="__codelineno-0-113"></a> <span class="k">if</span> <span class="n">value</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-114" name="__codelineno-0-114"></a> <span class="n">issue</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""The value of `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` contains the "</span><span class="si">{</span>
|
||||
<a id="__codelineno-0-115" name="__codelineno-0-115"></a> <span class="bp">self</span><span class="o">.</span><span class="n">ENVIRONMENT_VARIABLE_KEY_PREFIX</span>
|
||||
<a id="__codelineno-0-116" name="__codelineno-0-116"></a> <span class="si">}</span><span class="s2">` prefix but `</span><span class="si">{</span><span class="n">value</span><span class="si">}</span><span class="s2">` is not defined as an environment variable"""</span>
|
||||
<a id="__codelineno-0-117" name="__codelineno-0-117"></a> <span class="k">if</span> <span class="n">already_exists</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-118" name="__codelineno-0-118"></a> <span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-119" name="__codelineno-0-119"></a> <span class="sa">f</span><span class="s2">"""</span><span class="si">{</span><span class="n">issue</span><span class="si">}</span><span class="s2">, using the default value defined above (`</span><span class="si">{</span>
|
||||
<a id="__codelineno-0-120" name="__codelineno-0-120"></a> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-121" name="__codelineno-0-121"></a> <span class="si">}</span><span class="s2">`)"""</span>
|
||||
<a id="__codelineno-0-122" name="__codelineno-0-122"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-123" name="__codelineno-0-123"></a> <span class="k">continue</span>
|
||||
<a id="__codelineno-0-124" name="__codelineno-0-124"></a> <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ignore_missing</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-125" name="__codelineno-0-125"></a> <span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="n">issue</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-126" name="__codelineno-0-126"></a> <span class="k">else</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-127" name="__codelineno-0-127"></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-128" name="__codelineno-0-128"></a> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">issue</span><span class="si">}</span><span class="s2"> and no default value has been provided"</span>
|
||||
<a id="__codelineno-0-129" name="__codelineno-0-129"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-130" name="__codelineno-0-130"></a> <span class="k">else</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-131" name="__codelineno-0-131"></a> <span class="n">value</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">value</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-132" name="__codelineno-0-132"></a>
|
||||
<a id="__codelineno-0-133" name="__codelineno-0-133"></a> <span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-134" name="__codelineno-0-134"></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-135" name="__codelineno-0-135"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-136" name="__codelineno-0-136"></a> <span class="k">raise</span> <span class="ne">KeyError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-137" name="__codelineno-0-137"></a> <span class="sa">f</span><span class="s2">"Key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` is not found in configuration file (</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="o">.</span><span class="n">absolute</span><span class="p">()</span><span class="si">}</span><span class="s2">)"</span>
|
||||
<a id="__codelineno-0-138" name="__codelineno-0-138"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-139" name="__codelineno-0-139"></a>
|
||||
<a id="__codelineno-0-140" name="__codelineno-0-140"></a> <span class="fm">__getitem__</span> <span class="o">=</span> <span class="fm">__getattr__</span>
|
||||
<a id="__codelineno-0-133" name="__codelineno-0-133"></a> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
|
||||
<a id="__codelineno-0-134" name="__codelineno-0-134"></a>
|
||||
<a id="__codelineno-0-135" name="__codelineno-0-135"></a> <span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-136" name="__codelineno-0-136"></a> <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-137" name="__codelineno-0-137"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
|
||||
<a id="__codelineno-0-138" name="__codelineno-0-138"></a> <span class="k">raise</span> <span class="ne">KeyError</span><span class="p">(</span>
|
||||
<a id="__codelineno-0-139" name="__codelineno-0-139"></a> <span class="sa">f</span><span class="s2">"Key `</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">` is not found in configuration file (</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="o">.</span><span class="n">absolute</span><span class="p">()</span><span class="si">}</span><span class="s2">)"</span>
|
||||
<a id="__codelineno-0-140" name="__codelineno-0-140"></a> <span class="p">)</span>
|
||||
<a id="__codelineno-0-141" name="__codelineno-0-141"></a>
|
||||
<a id="__codelineno-0-142" name="__codelineno-0-142"></a> <span class="k">def</span> <span class="fm">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">Iterator</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-143" name="__codelineno-0-143"></a> <span class="k">return</span> <span class="nb">iter</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-144" name="__codelineno-0-144"></a>
|
||||
<a id="__codelineno-0-145" name="__codelineno-0-145"></a> <span class="k">def</span> <span class="fm">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-146" name="__codelineno-0-146"></a> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-147" name="__codelineno-0-147"></a>
|
||||
<a id="__codelineno-0-148" name="__codelineno-0-148"></a> <span class="k">def</span> <span class="nf">keys</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">KeysView</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-149" name="__codelineno-0-149"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-150" name="__codelineno-0-150"></a>
|
||||
<a id="__codelineno-0-151" name="__codelineno-0-151"></a> <span class="k">def</span> <span class="nf">values</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">ValuesView</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-152" name="__codelineno-0-152"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-153" name="__codelineno-0-153"></a>
|
||||
<a id="__codelineno-0-154" name="__codelineno-0-154"></a> <span class="k">def</span> <span class="nf">items</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">ItemsView</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-155" name="__codelineno-0-155"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-156" name="__codelineno-0-156"></a>
|
||||
<a id="__codelineno-0-157" name="__codelineno-0-157"></a> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-158" name="__codelineno-0-158"></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2">(path=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="si">}</span><span class="s2">) </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="si">}</span><span class="s2">"</span>
|
||||
<a id="__codelineno-0-142" name="__codelineno-0-142"></a> <span class="fm">__getitem__</span> <span class="o">=</span> <span class="fm">__getattr__</span>
|
||||
<a id="__codelineno-0-143" name="__codelineno-0-143"></a>
|
||||
<a id="__codelineno-0-144" name="__codelineno-0-144"></a> <span class="k">def</span> <span class="fm">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">Iterator</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-145" name="__codelineno-0-145"></a> <span class="k">return</span> <span class="nb">iter</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-146" name="__codelineno-0-146"></a>
|
||||
<a id="__codelineno-0-147" name="__codelineno-0-147"></a> <span class="k">def</span> <span class="fm">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-148" name="__codelineno-0-148"></a> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-149" name="__codelineno-0-149"></a>
|
||||
<a id="__codelineno-0-150" name="__codelineno-0-150"></a> <span class="k">def</span> <span class="nf">keys</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">KeysView</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-151" name="__codelineno-0-151"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-152" name="__codelineno-0-152"></a>
|
||||
<a id="__codelineno-0-153" name="__codelineno-0-153"></a> <span class="k">def</span> <span class="nf">values</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">ValuesView</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-154" name="__codelineno-0-154"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-155" name="__codelineno-0-155"></a>
|
||||
<a id="__codelineno-0-156" name="__codelineno-0-156"></a> <span class="k">def</span> <span class="nf">items</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">ItemsView</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
|
||||
<a id="__codelineno-0-157" name="__codelineno-0-157"></a> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
|
||||
<a id="__codelineno-0-158" name="__codelineno-0-158"></a>
|
||||
<a id="__codelineno-0-159" name="__codelineno-0-159"></a> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
|
||||
<a id="__codelineno-0-160" name="__codelineno-0-160"></a> <span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2">(path=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_path</span><span class="o">.</span><span class="n">as_posix</span><span class="p">()</span><span class="si">}</span><span class="s2">) </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_key_values</span><span class="si">}</span><span class="s2">"</span>
|
||||
</code></pre></div></td></tr></table></div>
|
||||
</details>
|
||||
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -351,6 +364,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -644,7 +671,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -658,7 +685,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -672,7 +699,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -755,6 +782,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/reference/views.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -999,13 +1027,13 @@
|
|||
|
||||
|
||||
|
||||
<a href="../../examples/simple/data/" class="md-footer__link md-footer__link--next" aria-label="Next: Train a domain classifier on the semantic scholar dataset" rel="next">
|
||||
<a href="../../examples/simple/data/" class="md-footer__link md-footer__link--next" aria-label="Next: Simple example: data engineering" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -100,4 +100,9 @@
|
|||
<lastmod>2022-07-12</lastmod>
|
||||
<changefreq>daily</changefreq>
|
||||
</url>
|
||||
<url>
|
||||
<loc>None</loc>
|
||||
<lastmod>2022-07-12</lastmod>
|
||||
<changefreq>daily</changefreq>
|
||||
</url>
|
||||
</urlset>
|
||||
BIN
sitemap.xml.gz
BIN
sitemap.xml.gz
Binary file not shown.
|
|
@ -6,9 +6,9 @@
|
|||
"source": [
|
||||
"# Harden and deploy your app\n",
|
||||
"\n",
|
||||
"Finally, it's time to deploy your model. But before you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.\n",
|
||||
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.\n",
|
||||
"\n",
|
||||
"With `GreatAI`, it has become a matter of 2 lines of code."
|
||||
"With `GreatAI`, it has become a matter of 4 lines of code."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -20,22 +20,22 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | GreatAI (v0.1.0): configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | 🔩 dashboard_table_size: 20\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:38:56 | WARNING | > Find out more at https://se-ml.github.io/practices\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | Fetching cached versions of my-domain-predictor\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | Latest version of my-domain-predictor is 5 (from versions: 0, 1, 2, 3, 4, 5)\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:56 | INFO | File my-domain-predictor-5 found in cache\u001b[0m\n"
|
||||
"\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
||||
"\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39mGreatAI (v0.1.4): configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n",
|
||||
"\u001b[38;5;39mFetching cached versions of my-domain-predictor\u001b[0m\n",
|
||||
"\u001b[38;5;39mLatest version of my-domain-predictor is 9 (from versions: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9)\u001b[0m\n",
|
||||
"\u001b[38;5;39mFile my-domain-predictor-9 found in cache\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -59,16 +59,16 @@
|
|||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Trace[str]({ 'created': '2022-07-09T20:38:56.394746',\n",
|
||||
"Trace[str]({'created': '2022-07-12T13:34:26.743292',\n",
|
||||
" 'exception': None,\n",
|
||||
" 'feedback': None,\n",
|
||||
" 'logged_values': { 'arg:sentence:length': 29,\n",
|
||||
" 'arg:sentence:value': 'Mountains are just big rocks.'},\n",
|
||||
" 'models': [{'key': 'my-domain-predictor', 'version': 5}],\n",
|
||||
" 'original_execution_time_ms': 4.999,\n",
|
||||
" 'models': [{'key': 'my-domain-predictor', 'version': 9}],\n",
|
||||
" 'original_execution_time_ms': 6.9699,\n",
|
||||
" 'output': 'geography',\n",
|
||||
" 'tags': ['predict_domain', 'online', 'development'],\n",
|
||||
" 'trace_id': 'aad1f83d-a81f-4b8b-898e-d02f8076616f'})"
|
||||
" 'trace_id': 'c80bdee3-602b-49dd-a84d-6eef80127e5a'})"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
|
|
@ -77,8 +77,16 @@
|
|||
}
|
||||
],
|
||||
"source": [
|
||||
"predict_domain(\"Mountains are just big rocks.\")\n",
|
||||
"# the original return value is under the 'output' key"
|
||||
"predict_domain(\"Mountains are just big rocks.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Notice how the original return value is under the `.output` key. Additionally, a plethora of metadata has been added which will be useful later on.\n",
|
||||
"\n",
|
||||
"Running your app in development-mode is as easy as executing `great-ai deploy.ipynb` from your terminal."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -90,33 +98,33 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[38;5;39m2022-07-09 22:38:58 | INFO | Converting notebook to Python script\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:58 | INFO | Found `predict_domain` to be the GreatAI app \u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:38:58 | INFO | Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | GreatAI (v0.1.0): configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | 🔩 dashboard_table_size: 20\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-09 22:39:00 | WARNING | > Find out more at https://se-ml.github.io/practices\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | Fetching cached versions of my-domain-predictor\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | Latest version of my-domain-predictor is 5 (from versions: 0, 1, 2, 3, 4, 5)\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | File my-domain-predictor-5 found in cache\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | Started server process [882179]\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | Waiting for application startup.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:00 | INFO | Application startup complete.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:28 | INFO | Converting notebook to Python script\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:29 | INFO | Found `predict_domain` to be the GreatAI app \u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:29 | INFO | Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | GreatAI (v0.1.4): configured ✅\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 large_file_implementation: LargeFileLocal\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 is_production: False\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 should_log_exception_stack: True\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 prediction_cache_size: 512\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | 🔩 dashboard_table_size: 50\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n",
|
||||
"\u001b[38;5;226m2022-07-12 15:34:31 | WARNING | > Find out more at https://se-ml.github.io/practices\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | Fetching cached versions of my-domain-predictor\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | Latest version of my-domain-predictor is 9 (from versions: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9)\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | File my-domain-predictor-9 found in cache\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | Started server process [199794]\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | Waiting for application startup.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:31 | INFO | Application startup complete.\u001b[0m\n",
|
||||
"^C\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:04 | INFO | Shutting down\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:04 | INFO | Waiting for application shutdown.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:04 | INFO | Application shutdown complete.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-09 22:39:04 | INFO | Finished server process [882179]\u001b[0m\n"
|
||||
"\u001b[38;5;39m2022-07-12 15:34:33 | INFO | Shutting down\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:33 | INFO | Waiting for application shutdown.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:33 | INFO | Application shutdown complete.\u001b[0m\n",
|
||||
"\u001b[38;5;39m2022-07-12 15:34:33 | INFO | Finished server process [199794]\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -129,16 +137,15 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Congrats, you've just created your first GreatAI service! 🎉\n",
|
||||
"\n",
|
||||
"Now that you've made sure your application is hardened enough for the intended use case, it is time to deploy it. The responsibilities of GreatAI end when it wraps your inference function and model into a production-ready service. You're given the freedom and responsibility to deploy this service. Fortunately, you (or your organisation) probably already has an established routine for deploying services.\n",
|
||||
"\n",
|
||||
"There are three main approaches to deploy a GreatAI service: For more info about them, check out [the deployment how-to](/how-to-guides/use-service)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### [Go back to the summary](/tutorial)"
|
||||
"There are three main approaches to deploy a GreatAI service: For more info about them, check out [the deployment how-to](/how-to-guides/use-service).\n",
|
||||
"\n",
|
||||
"For more thorough examples, see the [examples page](/examples/simple/data).\n",
|
||||
"\n",
|
||||
"### [Go back to the summary](/tutorial/#summary)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -394,6 +407,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -644,7 +671,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -658,7 +685,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -672,7 +699,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -752,6 +779,7 @@
|
|||
<article class="md-content__inner md-typeset">
|
||||
|
||||
|
||||
|
||||
<a href="deploy.ipynb" title="Download Notebook" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M5 20h14v-2H5m14-9h-4V3H9v6H5l7 7 7-7Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -1801,8 +1829,8 @@ Licensed under the Apache License, Version 2.0.
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">¶</a></h1><p>Finally, it's time to deploy your model. But before you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.</p>
|
||||
<p>With <code>GreatAI</code>, it has become a matter of 2 lines of code.</p>
|
||||
<h1 id="harden-and-deploy-your-app">Harden and deploy your app<a class="anchor-link" href="#harden-and-deploy-your-app">¶</a></h1><p>Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment <a href="https://se-ml.github.io/">best-practices</a>. In the past, this step was too often either the source of unexpected struggles, or worse, simply ignored.</p>
|
||||
<p>With <code>GreatAI</code>, it has become a matter of 4 lines of code.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1868,22 +1896,22 @@ def predict_domain(sentence, model):
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr">
|
||||
<pre><span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | GreatAI (v0.1.0): configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 tracing_database: ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 large_file_implementation: LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | 🔩 dashboard_table_size: 20</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:38:56 | WARNING | > Find out more at https://se-ml.github.io/practices</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | Fetching cached versions of my-domain-predictor</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | Latest version of my-domain-predictor is 5 (from versions: 0, 1, 2, 3, 4, 5)</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:56 | INFO | File my-domain-predictor-5 found in cache</span>
|
||||
<pre><span style="color: rgb(255,255,0)">Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(255,255,0)">Cannot find credentials files, defaulting to using ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(255,255,0)">The selected tracing database (ParallelTinyDbDriver) is not recommended for production</span>
|
||||
<span style="color: rgb(255,255,0)">Cannot find credentials files, defaulting to using LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">GreatAI (v0.1.4): configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 tracing_database: ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 large_file_implementation: LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)"> 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">> Find out more at https://se-ml.github.io/practices</span>
|
||||
<span style="color: rgb(0,175,255)">Fetching cached versions of my-domain-predictor</span>
|
||||
<span style="color: rgb(0,175,255)">Latest version of my-domain-predictor is 9 (from versions: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9)</span>
|
||||
<span style="color: rgb(0,175,255)">File my-domain-predictor-9 found in cache</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1915,10 +1943,8 @@ def predict_domain(sentence, model):
|
|||
</clipboard-copy>
|
||||
</div>
|
||||
<div class="highlight-ipynb hl-python"><pre><span></span><span class="n">predict_domain</span><span class="p">(</span><span class="s2">"Mountains are just big rocks."</span><span class="p">)</span>
|
||||
<span class="c1"># the original return value is under the 'output' key</span>
|
||||
</pre></div>
|
||||
<div id="cell-2" class="clipboard-copy-txt">predict_domain("Mountains are just big rocks.")
|
||||
# the original return value is under the 'output' key</div>
|
||||
<div id="cell-2" class="clipboard-copy-txt">predict_domain("Mountains are just big rocks.")</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1942,16 +1968,16 @@ def predict_domain(sentence, model):
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output jp-OutputArea-executeResult" data-mime-type="text/plain">
|
||||
<pre>Trace[str]({ 'created': '2022-07-09T20:38:56.394746',
|
||||
<pre>Trace[str]({'created': '2022-07-12T13:34:26.743292',
|
||||
'exception': None,
|
||||
'feedback': None,
|
||||
'logged_values': { 'arg:sentence:length': 29,
|
||||
'arg:sentence:value': 'Mountains are just big rocks.'},
|
||||
'models': [{'key': 'my-domain-predictor', 'version': 5}],
|
||||
'original_execution_time_ms': 4.999,
|
||||
'models': [{'key': 'my-domain-predictor', 'version': 9}],
|
||||
'original_execution_time_ms': 6.9699,
|
||||
'output': 'geography',
|
||||
'tags': ['predict_domain', 'online', 'development'],
|
||||
'trace_id': 'aad1f83d-a81f-4b8b-898e-d02f8076616f'})</pre>
|
||||
'trace_id': 'c80bdee3-602b-49dd-a84d-6eef80127e5a'})</pre>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
|
@ -1960,6 +1986,19 @@ def predict_domain(sentence, model):
|
|||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<p>Notice how the original return value is under the <code>.output</code> key. Additionally, a plethora of metadata has been added which will be useful later on.</p>
|
||||
<p>Running your app in development-mode is as easy as executing <code>great-ai deploy.ipynb</code> from your terminal.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
<div class="jp-Cell jp-CodeCell jp-Notebook-cell ">
|
||||
|
|
@ -2008,33 +2047,33 @@ def predict_domain(sentence, model):
|
|||
|
||||
|
||||
<div class="jp-RenderedText jp-OutputArea-output" data-mime-type="text/plain">
|
||||
<pre><span style="color: rgb(0,175,255)">2022-07-09 22:38:58 | INFO | Converting notebook to Python script</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:58 | INFO | Found `predict_domain` to be the GreatAI app </span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:38:58 | INFO | Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | GreatAI (v0.1.0): configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 tracing_database: ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 large_file_implementation: LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | 🔩 dashboard_table_size: 20</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-09 22:39:00 | WARNING | > Find out more at https://se-ml.github.io/practices</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | Fetching cached versions of my-domain-predictor</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | Latest version of my-domain-predictor is 5 (from versions: 0, 1, 2, 3, 4, 5)</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | File my-domain-predictor-5 found in cache</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | Started server process [882179]</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | Waiting for application startup.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:00 | INFO | Application startup complete.</span>
|
||||
<pre><span style="color: rgb(0,175,255)">2022-07-12 15:34:28 | INFO | Converting notebook to Python script</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:29 | INFO | Found `predict_domain` to be the GreatAI app </span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:29 | INFO | Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | Cannot find credentials files, defaulting to using ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | The selected tracing database (ParallelTinyDbDriver) is not recommended for production</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | Cannot find credentials files, defaulting to using LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | GreatAI (v0.1.4): configured ✅</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 tracing_database: ParallelTinyDbDriver</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 large_file_implementation: LargeFileLocal</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 is_production: False</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 should_log_exception_stack: True</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 prediction_cache_size: 512</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | 🔩 dashboard_table_size: 50</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | You still need to check whether you follow all best practices before trusting your deployment.</span>
|
||||
<span style="color: rgb(255,255,0)">2022-07-12 15:34:31 | WARNING | > Find out more at https://se-ml.github.io/practices</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | Fetching cached versions of my-domain-predictor</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | Latest version of my-domain-predictor is 9 (from versions: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9)</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | File my-domain-predictor-9 found in cache</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | Started server process [199794]</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | Waiting for application startup.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:31 | INFO | Application startup complete.</span>
|
||||
^C
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:04 | INFO | Shutting down</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:04 | INFO | Waiting for application shutdown.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:04 | INFO | Application shutdown complete.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-09 22:39:04 | INFO | Finished server process [882179]</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:33 | INFO | Shutting down</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:33 | INFO | Waiting for application shutdown.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:33 | INFO | Application shutdown complete.</span>
|
||||
<span style="color: rgb(0,175,255)">2022-07-12 15:34:33 | INFO | Finished server process [199794]</span>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2051,20 +2090,11 @@ def predict_domain(sentence, model):
|
|||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<p>Congrats, you've just created your first GreatAI service! 🎉</p>
|
||||
<p>Now that you've made sure your application is hardened enough for the intended use case, it is time to deploy it. The responsibilities of GreatAI end when it wraps your inference function and model into a production-ready service. You're given the freedom and responsibility to deploy this service. Fortunately, you (or your organisation) probably already has an established routine for deploying services.</p>
|
||||
<p>There are three main approaches to deploy a GreatAI service: For more info about them, check out <a href="/how-to-guides/use-service">the deployment how-to</a>.</p>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell">
|
||||
<div class="jp-Cell-inputWrapper">
|
||||
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
|
||||
</div>
|
||||
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
|
||||
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput " data-mime-type="text/markdown">
|
||||
<h3 id="go-back-to-the-summary"><a href="/tutorial">Go back to the summary</a><a class="anchor-link" href="#go-back-to-the-summary">¶</a></h3>
|
||||
<p>For more thorough examples, see the <a href="/examples/simple/data">examples page</a>.</p>
|
||||
<h3 id="go-back-to-the-summary"><a href="/tutorial/#summary">Go back to the summary</a><a class="anchor-link" href="#go-back-to-the-summary">¶</a></h3>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -2078,7 +2108,7 @@ def predict_domain(sentence, model):
|
|||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
@ -2115,13 +2145,13 @@ def predict_domain(sentence, model):
|
|||
|
||||
|
||||
|
||||
<a href="../../how-to-guides/create-service/" class="md-footer__link md-footer__link--next" aria-label="Next: How to create a GreatAI service" rel="next">
|
||||
<a href="../../how-to-guides/install/" class="md-footer__link md-footer__link--next" aria-label="Next: Installation guide" rel="next">
|
||||
<div class="md-footer__title">
|
||||
<div class="md-ellipsis">
|
||||
<span class="md-footer__direction">
|
||||
Next
|
||||
</span>
|
||||
How to create a GreatAI service
|
||||
Installation guide
|
||||
</div>
|
||||
</div>
|
||||
<div class="md-footer__button md-icon">
|
||||
|
|
|
|||
|
|
@ -48,6 +48,19 @@
|
|||
|
||||
|
||||
|
||||
<meta property="og:title" content="">
|
||||
<meta property="og:site_name" content="">
|
||||
<meta property="og:url" content="">
|
||||
<meta property="og:description" content="Transform your prototype AI code into production-ready software.">
|
||||
<meta property="og:type" content="">
|
||||
<meta property="og:image" content=https://great-ai.scoutinscience.com/media/og-image.png>
|
||||
|
||||
<style>
|
||||
.jupyter-wrapper a {
|
||||
color: var(--md-typeset-a-color) !important;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
|
||||
|
|
@ -334,15 +347,15 @@
|
|||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-training-notebook" class="md-nav__link">
|
||||
The training notebook
|
||||
<a href="#training-notebook" class="md-nav__link">
|
||||
Training notebook
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-deployment-notebook" class="md-nav__link">
|
||||
The deployment notebook
|
||||
<a href="#deployment-notebook" class="md-nav__link">
|
||||
Deployment notebook
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
|
@ -428,6 +441,20 @@
|
|||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/install/" class="md-nav__link">
|
||||
Installation guide
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../how-to-guides/create-service/" class="md-nav__link">
|
||||
How to create a GreatAI service
|
||||
|
|
@ -678,7 +705,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/data/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Simple example: data engineering
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -692,7 +719,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/train/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Optimise and train a model
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -706,7 +733,7 @@
|
|||
|
||||
<li class="md-nav__item">
|
||||
<a href="../examples/simple/deploy/" class="md-nav__link">
|
||||
Train a domain classifier on the semantic scholar dataset
|
||||
Hardening and deployment
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
|
@ -790,15 +817,15 @@
|
|||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-training-notebook" class="md-nav__link">
|
||||
The training notebook
|
||||
<a href="#training-notebook" class="md-nav__link">
|
||||
Training notebook
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-deployment-notebook" class="md-nav__link">
|
||||
The deployment notebook
|
||||
<a href="#deployment-notebook" class="md-nav__link">
|
||||
Deployment notebook
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
|
@ -823,6 +850,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
<a href="https://github.com/schmelczer/great-ai/edit/main/docs/tutorial/index.md" title="Edit this page" class="md-content__button md-icon">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25Z"/></svg>
|
||||
</a>
|
||||
|
|
@ -830,11 +858,11 @@
|
|||
|
||||
|
||||
<h1 id="train-and-deploy-a-sota-model">Train and deploy a SOTA model<a class="headerlink" href="#train-and-deploy-a-sota-model" title="Permanent link">#</a></h1>
|
||||
<p>Let's see GreatAI in action by going over the life-cycle of a simple service.</p>
|
||||
<p>Let's see <code>great-ai</code> in action by going over the life-cycle of a simple service.</p>
|
||||
<h2 id="objectives">Objectives<a class="headerlink" href="#objectives" title="Permanent link">#</a></h2>
|
||||
<ol>
|
||||
<li>You will see how the [great_ai.utilities][] can integrate into your Data Science workflow.</li>
|
||||
<li>You will use [great_ai.large_file][] to version and store your trained model.</li>
|
||||
<li>You will see how the <a href="/reference/utilities">great_ai.utilities</a> can integrate into your Data Science workflow.</li>
|
||||
<li>You will use <a href="/reference/large_file">great_ai.large_file</a> to version and store your trained model.</li>
|
||||
<li>You will use <a class="autorefs autorefs-internal" href="../reference/#great_ai.GreatAI">GreatAI</a> to prepare your model for a robust and responsible deployment.</li>
|
||||
</ol>
|
||||
<h2 id="overview">Overview<a class="headerlink" href="#overview" title="Permanent link">#</a></h2>
|
||||
|
|
@ -849,13 +877,13 @@
|
|||
<p><a class="md-button md-button--primary" href="deploy/"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M6 20a6 6 0 0 1-6-6c0-3.09 2.34-5.64 5.35-5.96A7.496 7.496 0 0 1 12 4a7.5 7.5 0 0 1 7.35 6A5.02 5.02 0 0 1 24 15a5 5 0 0 1-5 5H6M9.09 8.4 4.5 13l4.59 4.6 1.41-1.42L7.32 13l3.18-3.18L9.09 8.4m5.82 0L13.5 9.82 16.68 13l-3.18 3.18 1.41 1.42L19.5 13l-4.59-4.6Z"/></svg></span> Deploy it</a></p>
|
||||
</div>
|
||||
<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">#</a></h2>
|
||||
<h3 id="the-training-notebook">The <a href="train/">training notebook</a><a class="headerlink" href="#the-training-notebook" title="Permanent link">#</a></h3>
|
||||
<p>We load and preprocess the dataset while relying on [great_ai.utilities.clean][] for the heavy-lifting. Additionally, the preprocessing is parallelised using <a class="autorefs autorefs-internal" href="../reference/utilities/#great_ai.utilities.simple_parallel_map">great_ai.utilities.simple_parallel_map</a></p>
|
||||
<h3 id="training-notebook"><a href="train/">Training notebook</a><a class="headerlink" href="#training-notebook" title="Permanent link">#</a></h3>
|
||||
<p>We load and preprocess the dataset while relying on <a class="autorefs autorefs-internal" href="../reference/utilities/#great_ai.utilities.clean.clean">great_ai.utilities.clean</a> for doing the heavy-lifting. Additionally, the preprocessing is parallelised using <a class="autorefs autorefs-internal" href="../reference/utilities/#great_ai.utilities.simple_parallel_map">great_ai.utilities.simple_parallel_map</a></p>
|
||||
<p>After training and evaluating a model, it is exported using <a class="autorefs autorefs-internal" href="../reference/#great_ai.save_model">great_ai.save_model</a>.</p>
|
||||
<details class="tip">
|
||||
<summary>Remote storage</summary>
|
||||
<p>To store your model remotely, you need to set your credentials before calling <code>save_model</code>.</p>
|
||||
<p>For example, to use <a href="https://aws.amazon.com/s3/">AWS S3</a>:
|
||||
<p>For example, to use <a href="https://aws.amazon.com/s3" target="_blank">AWS S3</a>:
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileS3</span>
|
||||
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>
|
||||
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a><span class="n">LargeFileS3</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span>
|
||||
|
|
@ -869,8 +897,9 @@
|
|||
<a id="__codelineno-0-11" name="__codelineno-0-11" href="#__codelineno-0-11"></a>
|
||||
<a id="__codelineno-0-12" name="__codelineno-0-12" href="#__codelineno-0-12"></a><span class="n">save_model</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'my-domain-predictor'</span><span class="p">)</span>
|
||||
</code></pre></div></p>
|
||||
<p>For more info, checkout <a href="/how-to-guides/configure-service">the configuration how-to page</a>.</p>
|
||||
</details>
|
||||
<h3 id="the-deployment-notebook">The <a href="deploy/">deployment notebook</a><a class="headerlink" href="#the-deployment-notebook" title="Permanent link">#</a></h3>
|
||||
<h3 id="deployment-notebook"><a href="deploy/">Deployment notebook</a><a class="headerlink" href="#deployment-notebook" title="Permanent link">#</a></h3>
|
||||
<p>We create an inference function that can be hardened by wrapping it in a <a class="autorefs autorefs-internal" href="../reference/#great_ai.GreatAI">GreatAI</a> instance.</p>
|
||||
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">GreatAI</span><span class="p">,</span> <span class="n">use_model</span>
|
||||
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a><span class="kn">from</span> <span class="nn">great_ai.utilities</span> <span class="kn">import</span> <span class="n">clean</span>
|
||||
|
|
@ -883,16 +912,20 @@
|
|||
</code></pre></div>
|
||||
<ol>
|
||||
<li><a class="autorefs autorefs-internal" href="../reference/#great_ai.use_model">@use_model</a> loads and injects your model into the <code>predict_domain</code> function's <code>model</code> argument.
|
||||
You can freely reference it knowing that it is always given to the function.</li>
|
||||
You can freely reference it knowing that the function is always provided with it.</li>
|
||||
</ol>
|
||||
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened-service.</a></p>
|
||||
<p>Finally, we test the model's inference function through the GreatAI dashboard. <a href="/how-to-guides/use-service">The only thing left is to deploy the hardened-service properly.</a></p>
|
||||
<div style="display: flex; justify-content: center;">
|
||||
<p><a class="md-button md-button--primary" href="/how-to-guides/create-service"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 22a2 2 0 0 0 2-2V4a2 2 0 0 0-2-2h-6v7L9.5 7.5 7 9V2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12Z"/></svg></span> Learn about all the features</a></p>
|
||||
<p><a class="md-button md-button--secondary" href="/examples/simple/data"><span class="twemoji"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M7 2v2h1v14a4 4 0 0 0 4 4 4 4 0 0 0 4-4V4h1V2H7m4 14c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m2-4c-.6 0-1-.4-1-1s.4-1 1-1 1 .4 1 1-.4 1-1 1m1-5h-4V4h4v3Z"/></svg></span> Look at more examples</a></p>
|
||||
</div>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
# Train and deploy a SOTA model
|
||||
|
||||
Let's see GreatAI in action by going over the life-cycle of a simple service.
|
||||
Let's see `great-ai` in action by going over the life-cycle of a simple service.
|
||||
|
||||
## Objectives
|
||||
|
||||
1. You will see how the [great_ai.utilities][] can integrate into your Data Science workflow.
|
||||
2. You will use [great_ai.large_file][] to version and store your trained model.
|
||||
1. You will see how the [great_ai.utilities](/reference/utilities) can integrate into your Data Science workflow.
|
||||
2. You will use [great_ai.large_file](/reference/large_file) to version and store your trained model.
|
||||
3. You will use [GreatAI][great_ai.GreatAI] to prepare your model for a robust and responsible deployment.
|
||||
|
||||
## Overview
|
||||
|
|
@ -14,7 +14,6 @@ You are going to train a field of study (domain) classifier for scientific sente
|
|||
|
||||
We use the same synthetic dataset derived from the [Microsoft Academic Graph](https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/). The dataset is [available here](https://github.com/allenai/scibert/tree/master/data/text_classification/mag).
|
||||
|
||||
|
||||
!!! success
|
||||
You are ready to start the tutorial. Feel free to come back to the [summary](#summary) section once you're finished.
|
||||
|
||||
|
|
@ -24,19 +23,18 @@ We use the same synthetic dataset derived from the [Microsoft Academic Graph](ht
|
|||
[:material-cloud-tags: Deploy it](deploy.ipynb){ .md-button .md-button--primary }
|
||||
</div>
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
### The [training notebook](train.ipynb)
|
||||
### [Training notebook](train.ipynb)
|
||||
|
||||
We load and preprocess the dataset while relying on [great_ai.utilities.clean][] for the heavy-lifting. Additionally, the preprocessing is parallelised using [great_ai.utilities.simple_parallel_map][]
|
||||
We load and preprocess the dataset while relying on [great_ai.utilities.clean][great_ai.utilities.clean.clean] for doing the heavy-lifting. Additionally, the preprocessing is parallelised using [great_ai.utilities.simple_parallel_map][]
|
||||
|
||||
After training and evaluating a model, it is exported using [great_ai.save_model][].
|
||||
|
||||
??? tip "Remote storage"
|
||||
To store your model remotely, you need to set your credentials before calling `save_model`.
|
||||
|
||||
For example, to use [AWS S3](https://aws.amazon.com/s3/):
|
||||
For example, to use [AWS S3](https://aws.amazon.com/s3){ target=_blank }:
|
||||
```python
|
||||
from great_ai.large_file import LargeFileS3
|
||||
|
||||
|
|
@ -52,7 +50,9 @@ After training and evaluating a model, it is exported using [great_ai.save_model
|
|||
save_model(model, key='my-domain-predictor')
|
||||
```
|
||||
|
||||
### The [deployment notebook](deploy.ipynb)
|
||||
For more info, checkout [the configuration how-to page](/how-to-guides/configure-service).
|
||||
|
||||
### [Deployment notebook](deploy.ipynb)
|
||||
|
||||
We create an inference function that can be hardened by wrapping it in a [GreatAI][great_ai.GreatAI] instance.
|
||||
|
||||
|
|
@ -68,6 +68,12 @@ def predict_domain(sentence, model):
|
|||
```
|
||||
|
||||
1. [@use_model][great_ai.use_model] loads and injects your model into the `predict_domain` function's `model` argument.
|
||||
You can freely reference it knowing that it is always given to the function.
|
||||
You can freely reference it knowing that the function is always provided with it.
|
||||
|
||||
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened-service.](/how-to-guides/use-service)
|
||||
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened-service properly.](/how-to-guides/use-service)
|
||||
|
||||
<div style="display: flex; justify-content: center;" markdown>
|
||||
[:material-book: Learn about all the features](/how-to-guides/create-service){ .md-button .md-button--primary }
|
||||
|
||||
[:material-test-tube: Look at more examples](/examples/simple/data){ .md-button .md-button--secondary }
|
||||
</div>
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Loading…
Add table
Add a link
Reference in a new issue