Fix best practices hyphenation
|
|
@ -31,7 +31,7 @@ Start it by executing `great-ai demo.py`, find the dashboard at [http://localhos
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
That's it. Your GreatAI service is _nearly_ ready for production use. Many of the [SE4ML best-practices](https://se-ml.github.io) are configured and implemented automatically (of course, these can be customised as well).
|
That's it. Your GreatAI service is _nearly_ ready for production use. Many of the [SE4ML best practices](https://se-ml.github.io) are configured and implemented automatically (of course, these can be customised as well).
|
||||||
|
|
||||||
[Check out the full documentation here](https://great-ai.scoutinscience.com).
|
[Check out the full documentation here](https://great-ai.scoutinscience.com).
|
||||||
|
|
||||||
|
|
@ -51,7 +51,7 @@ GreatAI fits between the prototype and deployment phases of your AI development
|
||||||
|
|
||||||
There are other, existing solutions aiming to facilitate this phase. [Amazon SageMaker](https://aws.amazon.com/sagemaker) and [Seldon Core](https://www.seldon.io/solutions/open-source-projects/core) provide the most comprehensive suite of features. If you have the opportunity to use them, do that because they're great.
|
There are other, existing solutions aiming to facilitate this phase. [Amazon SageMaker](https://aws.amazon.com/sagemaker) and [Seldon Core](https://www.seldon.io/solutions/open-source-projects/core) provide the most comprehensive suite of features. If you have the opportunity to use them, do that because they're great.
|
||||||
|
|
||||||
However, [research indicates](https://great-ai.scoutinscience.com) that professionals rarely use them. This may be due to their inherent setup and operating complexity. **GreatAI is designed to be as simple to use as possible.** Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the [SE4ML best-practices](https://se-ml.github.io), and thus, can meaningfully improve your deployment without requiring prohibitively large effort.
|
However, [research indicates](https://great-ai.scoutinscience.com) that professionals rarely use them. This may be due to their inherent setup and operating complexity. **GreatAI is designed to be as simple to use as possible.** Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many of the [SE4ML best practices](https://se-ml.github.io), and thus, can meaningfully improve your deployment without requiring prohibitively large effort.
|
||||||
|
|
||||||
## [Learn more](https://great-ai.scoutinscience.com)
|
## [Learn more](https://great-ai.scoutinscience.com)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@
|
||||||
"source": [
|
"source": [
|
||||||
"# Simple example: data engineering\n",
|
"# Simple example: data engineering\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
"Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best practices. In short, we train a domain classifier on the [semantic scholar dataset](https://api.semanticscholar.org/corpus) by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
"> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.\n",
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@ In order to simplify your training data management, `great-ai` provide two compl
|
||||||
At the start of your experiments' first iteration, after you've gathered suitable samples for training, you can call [great_ai.add_ground_truth][]. This automatically stores a timestamp and also allows you to assign tags to the data. Using these attributes, [great_ai.query_ground_truth][] can be called to get a filtered view of the training data.
|
At the start of your experiments' first iteration, after you've gathered suitable samples for training, you can call [great_ai.add_ground_truth][]. This automatically stores a timestamp and also allows you to assign tags to the data. Using these attributes, [great_ai.query_ground_truth][] can be called to get a filtered view of the training data.
|
||||||
|
|
||||||
!!! important "Train-test-validation splits"
|
!!! important "Train-test-validation splits"
|
||||||
It is a best-practice to lock-away the test split of your data that is only used for the final quality assessment. This prevents you from accidentally training on it, or inadvertently tuning the model to have the highest accuracy metrics on the test split. This, of course, may lead to dubious results, hence, care must be taken to avoid it.
|
It is a best practice to lock-away the test split of your data that is only used for the final quality assessment. This prevents you from accidentally training on it, or inadvertently tuning the model to have the highest accuracy metrics on the test split. This, of course, may lead to dubious results, hence, care must be taken to avoid it.
|
||||||
|
|
||||||
With [great_ai.add_ground_truth][], there is an option to tag the samples with `train`, `test`, and `validation` randomly, following a predefined distribution. This happens as soon as they're written in the database. Later, these can be queried by providing the name of the appropriate tags.
|
With [great_ai.add_ground_truth][], there is an option to tag the samples with `train`, `test`, and `validation` randomly, following a predefined distribution. This happens as soon as they're written in the database. Later, these can be queried by providing the name of the appropriate tags.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -70,7 +70,7 @@ Oftentimes, especially when working with data-heavy applications, large files ca
|
||||||
|
|
||||||
## From the command-line
|
## From the command-line
|
||||||
|
|
||||||
The main reason for using the `large-file` or `python3 -m great_ai.large_file` commands is to upload or download models from the terminal. For example, when building a docker image, it is best-practice to cache the referred models.
|
The main reason for using the `large-file` or `python3 -m great_ai.large_file` commands is to upload or download models from the terminal. For example, when building a docker image, it is best practice to cache the referred models.
|
||||||
|
|
||||||
### Setup
|
### Setup
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -187,7 +187,7 @@
|
||||||
<div data-drawio-colors="color: #ffffff; " style="box-sizing: border-box; font-size: 0px; text-align: center;">
|
<div data-drawio-colors="color: #ffffff; " style="box-sizing: border-box; font-size: 0px; text-align: center;">
|
||||||
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(255, 255, 255); line-height: 1.2; pointer-events: none; white-space: normal; overflow-wrap: normal;">
|
<div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(255, 255, 255); line-height: 1.2; pointer-events: none; white-space: normal; overflow-wrap: normal;">
|
||||||
<font style="font-size: 14px">
|
<font style="font-size: 14px">
|
||||||
Artifact combining the model with deplyoment best-practices
|
Artifact combining the model with deplyoment best practices
|
||||||
</font>
|
</font>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
|
Before Width: | Height: | Size: 19 KiB After Width: | Height: | Size: 19 KiB |
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
|
|
@ -40,7 +40,7 @@ SageMaker offers the most comprehensive suite of tools and service; most importa
|
||||||
|
|
||||||
When it comes to open-source libraries, we can find the MLOps libraries of both TensorFlow and PyTorch: TensorFlow Extended (TFX) \cite{baylor2017tfx} and TorchX\footnote{\href{https://pytorch.org/torchx/latest/}{pytorch.org/torchx/latest}}. TFX comes with a more mature set of features with the caveat that initial time-investment is needed for their setup. The features of TorchX only concern the distributed deployment to a wide range of providers, including Kubernetes (K8s), AWS Batch, or Ray \cite{moritz2018ray}. There is no augmentation for the SE4ML best practices. Given the tight coupling between these libraries and their corresponding ML frameworks, they cannot generalise to models\footnote{The Open Neural Network Exchange (\href{https://onnx.ai/}{onnx.ai}) format could be an option for overcoming these incompatibilities, however, a more universal support is needed for seamless integration.} or algorithms of other frameworks and technologies.
|
When it comes to open-source libraries, we can find the MLOps libraries of both TensorFlow and PyTorch: TensorFlow Extended (TFX) \cite{baylor2017tfx} and TorchX\footnote{\href{https://pytorch.org/torchx/latest/}{pytorch.org/torchx/latest}}. TFX comes with a more mature set of features with the caveat that initial time-investment is needed for their setup. The features of TorchX only concern the distributed deployment to a wide range of providers, including Kubernetes (K8s), AWS Batch, or Ray \cite{moritz2018ray}. There is no augmentation for the SE4ML best practices. Given the tight coupling between these libraries and their corresponding ML frameworks, they cannot generalise to models\footnote{The Open Neural Network Exchange (\href{https://onnx.ai/}{onnx.ai}) format could be an option for overcoming these incompatibilities, however, a more universal support is needed for seamless integration.} or algorithms of other frameworks and technologies.
|
||||||
|
|
||||||
Open-source platforms also exist such as MLflow and Seldon Core. They both rely on Kubernetes to provide their features. MLflow puts more emphasis on the training phase (in deployment, it lacks a feedback loop which is essential for reaching many of the best-practices), while Seldon Core focuses on the deployment stage. The latter comes integrated with a powerful explanation engine, Alibi Explain \cite{klaise2021alibi}. It also boasts the most comprehensive suite of features including outlier detection, online model selection (with multi-armed bandit theory), and distributed tracing. In short, it seems to be the ideal candidate for the title of \textit{framework for robust end-to-end AI deployments}. Its only downside is the amount of complexity propagated to its clients: it is built on top of Kubernetes, and relies on Helm, Ambasador/Istio, Prometheus, and Jaeger for its features. Hence, the first step in using it is setting up a K8s cluster with all the required components, then when it comes to model deployment, a Kubernetes configuration file has to be created to make use of Seldon's Custom Resource Definition. These are smaller obstacles if the project is already built on top of K8s; however, even then, software engineers with strong cloud and DevOps background are actively required for using Seldon Core.
|
Open-source platforms also exist such as MLflow and Seldon Core. They both rely on Kubernetes to provide their features. MLflow puts more emphasis on the training phase (in deployment, it lacks a feedback loop which is essential for reaching many of the best practices), while Seldon Core focuses on the deployment stage. The latter comes integrated with a powerful explanation engine, Alibi Explain \cite{klaise2021alibi}. It also boasts the most comprehensive suite of features including outlier detection, online model selection (with multi-armed bandit theory), and distributed tracing. In short, it seems to be the ideal candidate for the title of \textit{framework for robust end-to-end AI deployments}. Its only downside is the amount of complexity propagated to its clients: it is built on top of Kubernetes, and relies on Helm, Ambasador/Istio, Prometheus, and Jaeger for its features. Hence, the first step in using it is setting up a K8s cluster with all the required components, then when it comes to model deployment, a Kubernetes configuration file has to be created to make use of Seldon's Custom Resource Definition. These are smaller obstacles if the project is already built on top of K8s; however, even then, software engineers with strong cloud and DevOps background are actively required for using Seldon Core.
|
||||||
|
|
||||||
Additionally, increasing attention is given to ML deployments in embedded systems both from a theoretical \cite{john2020ai} and practical \cite{prado2020bonseyes} point of view. Prado et al. \cite{prado2020bonseyes} survey the available deployment frameworks and end-to-end solutions including those for embedded devices. They note their inefficiencies that come from the lack of features and too much rigidity. They introduce their framework for embedded AI deployments which can be used out-of-the box but also lets the users easily replace and extend its pipeline with steps to fit their changing needs and advancements of the field. While Meenu et al. \cite{john2020ai} present and compare different architectural choices for large-scale deployments in edge-computing. They also note that: \textit{"...there is a need to consider and adapt well established SE practices which have been ignored or had a very narrow focus in ML literature"}.
|
Additionally, increasing attention is given to ML deployments in embedded systems both from a theoretical \cite{john2020ai} and practical \cite{prado2020bonseyes} point of view. Prado et al. \cite{prado2020bonseyes} survey the available deployment frameworks and end-to-end solutions including those for embedded devices. They note their inefficiencies that come from the lack of features and too much rigidity. They introduce their framework for embedded AI deployments which can be used out-of-the box but also lets the users easily replace and extend its pipeline with steps to fit their changing needs and advancements of the field. While Meenu et al. \cite{john2020ai} present and compare different architectural choices for large-scale deployments in edge-computing. They also note that: \textit{"...there is a need to consider and adapt well established SE practices which have been ignored or had a very narrow focus in ML literature"}.
|
||||||
|
|
||||||
|
|
@ -68,7 +68,7 @@ No DevOps dependencies\textsuperscript{4}& & &
|
||||||
\begin{tablenotes}
|
\begin{tablenotes}
|
||||||
\item[1] For privacy and accountability reasons. \cite{bosch2021engineering}
|
\item[1] For privacy and accountability reasons. \cite{bosch2021engineering}
|
||||||
\item[2] Minimising required glue code. \cite{sculley2015hidden}
|
\item[2] Minimising required glue code. \cite{sculley2015hidden}
|
||||||
\item[3] Implementing best-practices. \cite{serban2020adoption,serban2021practices}
|
\item[3] Implementing best practices. \cite{serban2020adoption,serban2021practices}
|
||||||
\item[4] Easy integration into existing processes. \cite{haakman2021ai,thiee2021systematic}
|
\item[4] Easy integration into existing processes. \cite{haakman2021ai,thiee2021systematic}
|
||||||
\item[*] Only partial support.
|
\item[*] Only partial support.
|
||||||
\end{tablenotes}
|
\end{tablenotes}
|
||||||
|
|
|
||||||
|
|
@ -41,7 +41,7 @@ SSC is much larger: it contains over 80 million abstracts. Having more data cert
|
||||||
MAG needs no further preprocessing if we aim to match SciBERT's setup \cite{beltagy2019scibert}. But since SSC contains a heap of metadata, the relevant parts have to be extracted and preprocessed. In this case, these are the concatenation of the abstract's text, paper's title and the journal's name along with the paper's domains (there can be multiple domains for a single paper, it is a multi-label classification task). Lastly, the non-English entries are discarded because we only expect to process papers in English.
|
MAG needs no further preprocessing if we aim to match SciBERT's setup \cite{beltagy2019scibert}. But since SSC contains a heap of metadata, the relevant parts have to be extracted and preprocessed. In this case, these are the concatenation of the abstract's text, paper's title and the journal's name along with the paper's domains (there can be multiple domains for a single paper, it is a multi-label classification task). Lastly, the non-English entries are discarded because we only expect to process papers in English.
|
||||||
|
|
||||||
\begin{displayquote}
|
\begin{displayquote}
|
||||||
\textbf{How should we preprocess the data?} These simple processing steps (filter, map, project) are almost always present in the data science life-cycle. For example, cleaning the input text from various HTML, OCR, PDF, or \LaTeX \hskip 0.12cm extraction artifacts is almost always necessary for text analysis. This is captured in the AI best-practices collection under the following category: \textit{Write Reusable Scripts for Data Cleaning and Merging}. Also, the best practice of \textit{Test all Feature Extraction Code} is somewhat applicable: the applied processing steps must not introduce unwanted artifacts.
|
\textbf{How should we preprocess the data?} These simple processing steps (filter, map, project) are almost always present in the data science life-cycle. For example, cleaning the input text from various HTML, OCR, PDF, or \LaTeX \hskip 0.12cm extraction artifacts is almost always necessary for text analysis. This is captured in the AI best practices collection under the following category: \textit{Write Reusable Scripts for Data Cleaning and Merging}. Also, the best practice of \textit{Test all Feature Extraction Code} is somewhat applicable: the applied processing steps must not introduce unwanted artifacts.
|
||||||
\end{displayquote}
|
\end{displayquote}
|
||||||
|
|
||||||
\subsection{Methods}
|
\subsection{Methods}
|
||||||
|
|
@ -50,7 +50,7 @@ Our aims are twofold: (1) to evaluate a sentence classification model on MAG and
|
||||||
|
|
||||||
It seems reasonable that only considering the distribution (frequencies) of individual terms may be sufficient. To test this hypothesis, a unigram language model (Multinomial Naïve Bayes) is constructed and its accuracy is compared with SciBERT's. The former definitely aligns with the advice to \textit{Use The Most Efficient Models}\footnote{\href{https://se-ml.github.io/best_practices/02-efficient-models/}{se-ml.github.io/best\_practices/02-efficient-models}}.
|
It seems reasonable that only considering the distribution (frequencies) of individual terms may be sufficient. To test this hypothesis, a unigram language model (Multinomial Naïve Bayes) is constructed and its accuracy is compared with SciBERT's. The former definitely aligns with the advice to \textit{Use The Most Efficient Models}\footnote{\href{https://se-ml.github.io/best_practices/02-efficient-models/}{se-ml.github.io/best\_practices/02-efficient-models}}.
|
||||||
|
|
||||||
Using the MNB implementation of scikit-learn \cite{pedregosa2011scikit}, it only took a couple of lines to create, hyperparameter-optimise, and test a text classifier. Including data loading and visualisations, it takes 71 lines of code (LOC) to be more precise. \footnote{The code is available at \href{https://great-ai.scoutinscience.com/tutorial/}{great-ai.scoutinscience.com/tutorial}.} This further proves relatively how simple it is to use standard packages. The code can be considered for satisfying the \textit{Automate Hyper-Parameter Optimisation} best-practice, since it also implements an automated hyperparameter sweep.
|
Using the MNB implementation of scikit-learn \cite{pedregosa2011scikit}, it only took a couple of lines to create, hyperparameter-optimise, and test a text classifier. Including data loading and visualisations, it takes 71 lines of code (LOC) to be more precise. \footnote{The code is available at \href{https://great-ai.scoutinscience.com/tutorial/}{great-ai.scoutinscience.com/tutorial}.} This further proves relatively how simple it is to use standard packages. The code can be considered for satisfying the \textit{Automate Hyper-Parameter Optimisation} best practice, since it also implements an automated hyperparameter sweep.
|
||||||
|
|
||||||
The sentences are tokenised into words and vectorised with TF-IDF (with logarithmic term frequency) \cite{buckley1985implementation}, the hyperparameters found via 3-fold cross-validation on the \textit{train} split lead to filtering out tokens which occur in fewer than 5 documents or more than 5\% of the documents.
|
The sentences are tokenised into words and vectorised with TF-IDF (with logarithmic term frequency) \cite{buckley1985implementation}, the hyperparameters found via 3-fold cross-validation on the \textit{train} split lead to filtering out tokens which occur in fewer than 5 documents or more than 5\% of the documents.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@
|
||||||
"source": [
|
"source": [
|
||||||
"# Harden and deploy your app\n",
|
"# Harden and deploy your app\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best-practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.\n",
|
"Finally, it's time to deploy your model. But before that, you have to make sure you follow AI deployment [best practices](https://se-ml.github.io/). In the past, this step was too often either the source of unexpected struggle, or worse, simply ignored.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"With `GreatAI`, it has become a matter of 4 lines of code."
|
"With `GreatAI`, it has become a matter of 4 lines of code."
|
||||||
]
|
]
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,7 @@ V = TypeVar("V")
|
||||||
|
|
||||||
|
|
||||||
class GreatAI(Generic[T, V]):
|
class GreatAI(Generic[T, V]):
|
||||||
"""Wrapper for a prediction function providing the implementation of SE4ML best-practices.
|
"""Wrapper for a prediction function providing the implementation of SE4ML best practices.
|
||||||
|
|
||||||
Provides caching (with argument freezing), a TracingContext during execution, the
|
Provides caching (with argument freezing), a TracingContext during execution, the
|
||||||
scaffolding of HTTP endpoints using FastAPI and a dashboard using Dash.
|
scaffolding of HTTP endpoints using FastAPI and a dashboard using Dash.
|
||||||
|
|
|
||||||