Start documentation

2022-07-08 20:29:22 +02:00 · 2022-07-08 20:29:22 +02:00 · 46ffe7a70e
commit 46ffe7a70e
parent c06862ab76
6 changed files with 241 additions and 16 deletions
--- a/README.md
+++ b/README.md
@ -46,12 +46,12 @@ Find the dashboard at [http://localhost:6060](http://localhost:6060/dashboard/).
 ```sh
 python3 -m venv --copies .env
 source .env/bin/activate
-pip install './[dev]'
-flit install --symlink
+python3 -m pip install flit
+python3 -m flit install --symlink --deps=all
 ```

 #### Documentation

 ```sh
-mkdocs serve
+mkdocs serve --dirtyreload
 ```
--- a/docs/index.md
+++ b/docs/index.md
@ -1,17 +1,88 @@
-# Welcome to MkDocs
+# Welcome to GreatAI

-For full documentation visit [mkdocs.org](https://www.mkdocs.org).
+Applying AI is becoming increasingly easier but many case studies have shown that these applications are often deployed poorly. GreatAI helps you transform your prototype AI code into production-ready software. 

-## Commands
+??? quote
+    There is a need to consider and adapt well established SE practices which have been ignored or had a very narrow focus in ML literature.
+    &mdash; [Meenu et al.](https://ieeexplore.ieee.org/abstract/document/9359253)

-* `mkdocs new [dir-name]` - Create a new project.
-* `mkdocs serve` - Start the live-reloading docs server.
-* `mkdocs build` - Build the documentation site.
-* `mkdocs -h` - Print help message and exit.

-## Project layout
+## Features

-    mkdocs.yml    # The configuration file.
-    docs/
-        index.md  # The documentation homepage.
-        ...       # Other markdown pages, images and other files.
+- [x] Save traces of each prediction including arguments and model version
+- [x] Store models and data on shared infrastructure (MongoDB GridFS, S3-compatible storage, local-disk)
+- [x] Scaffold REST API and OpenAPI schema for easy integration
+- [x] Seamless support for `async` inference methods
+- [x] Support calling remote GreatAI instances
+- [x] Save feedback and merge it into the ground-truth database
+- [x] Built-in parallelising for batch processing
+- [x] Well-tested utilities for common NLP tasks (cleaning, language-tagging, etc.)
+- [x] Dashboard for high-level overview and searching traces
+- [x] Fully-typed API for IntelliSense support
+- [x] Docker support for deployment
+- [x] Auto-reload fro development
+
+> Can be highly customised, including turning of in specific cases, and disabling the logging of sensitive data.
+
+## Hello world
+
+```
+pip install great-ai
+```
+
+```python title="hello-world.py" 
+from great_ai import GreatAI
+
+@GreatAI.create  #(1) 
+def hello_world(name: str) -> str:
+    return f"Hello {name}!"
+```
+
+1.  `@GreatAI.create` wraps your `hello_world` function with a `GreatAI` instance. The function will behave very similarly but:
+    1. its return value becomes a `Trace[str]`,
+    2. it gets a `process_batch` method for supporting parallel execution,
+    3. and it can be deployed using the `great-ai` command-line tool.
+
+
+??? note
+    In practice, `hello_world` could be an inference function of some AI/ML application.
+
+```bash title="terminal" 
+great-ai hello-world.py
+```
+> Navigate to [localhost:6060](http://127.0.0.1:6060/) in your browser.
+
+<div style="display: flex; justify-content: space-evenly;" markdown>
+![](media/hello-world-dashboard.png){ loading=lazy }
+
+![](media/hello-world-docs.png){ loading=lazy }
+</div>
+
+!!! success
+    Your GreatAI service is ready for production use. Many of the [SE4ML best-practices](https://se-ml.github.io/) are configured and implemented automatically. To have full control over your service and to understand what else you might need to do in your use case, continue reading this documentation.
+
+
+## Why is this GREAT?
+
+![scope of GreatAI](scope-simple.drawio.svg)
+
+GreatAI fits between the prototype and deployment phase of your (or your organisation's) AI development lifecycle. This is highlighted with blue in the diagram. Here, a number of best practices can be automatically implemented concerning the following 5 aspects:
+
+- **G**eneral: support for all kinds of Python libraries without restriction
+- **R**obust: robust error-handling and well-tested utilities out-of-the-box 
+- **E**nd-to-end: managing end-to-end feedback is a built-in, first-class concept
+- **A**utomated: focus only on what actually requires your attention
+- **T**rustworthy: deploy models that you and society can confidently trust
+
+## Why GreatAI?
+
+There are other existing solutions aiming to solve facilitate this phase. [Amazon SageMaker](https://aws.amazon.com/sagemaker/) and [Seldon Core](https://www.seldon.io/solutions/open-source-projects/core) provide the most comprehensive suite of features. If you have the opportunity use those, they're great.
+
+However, research indicates that professionals rarely use them. This may be due to their inherent setup and operating complexity. GreatAI is designed to be as simple to use as possible. Its clear, high-level API and sensible default configuration makes it extremely easy to start using. Despite its relative simplicity over Seldon Core, it still implements many [best-practices](https://se-ml.github.io/), and thus, can meaningfully improve your deployment without too much effort.
+
+
+<div style="display: flex; justify-content: space-evenly;" markdown>
+[:fontawesome-brands-python: Find it on PyPI](https://pypi.org/project/great-ai/){ .md-button .md-button--primary }
+
+[:fontawesome-brands-docker: Find it on DockerHub](https://hub.docker.com/repository/docker/schmelczera/great-ai){ .md-button .md-button--primary }
+</div>
--- a/docs/media/hello-world-dashboard.png
+++ b/docs/media/hello-world-dashboard.png
--- a/docs/media/hello-world-docs.png
+++ b/docs/media/hello-world-docs.png
--- a/docs/scope-simple.drawio.svg
+++ b/docs/scope-simple.drawio.svg
@ -0,0 +1,125 @@
+<svg host="65bd71144e" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1161px" height="96px" viewBox="-0.5 -0.5 1161 96" content="&lt;mxfile&gt;&lt;diagram id=&quot;ccudx8biEBM-x_P_p45R&quot; name=&quot;Page-1&quot;&gt;3VhLc9owEP41zKSHZPwAA8cESHpoppmhM02Owl5sTWXJkQWY/vqubAnbvJJS8phywNrVev2t9tuV7I4/Sos7SbLkXkTAOp4TFR1/3PE81/P6eNGadaUZ9I0iljQyRrViSn+DUTpGu6AR5C1DJQRTNGsrQ8E5hKqlI1KKVdtsLlj7qRmJYUcxDQnb1f6kkUpMFDYsrf8KNE7sk91gWM2kxBqbSPKERGLVUPmTjj+SQqhqlBYjYHrx7LpU990emN0Ak8DVa24wiHO1trFBhKEakQuOl5tEpQwlF4dzwZXJh9tFGQqqHs2cHj/h2LnqGWmsM+5YYW2EXYwGdi4WMjQwfJNWImMwVt1KpQE2bjNx3YFIQck1GkhgRNFlO1fEpDze2G1ufRAUgXiOoaffc66Gjd+gcmCoGjhtfxVm46JeaBw0MNWqcvn3p8JEvCRsYaB3vIBh7DczHMSqXLlKobPQylrwvBB24jIv83ONBu4gK+pJ62VMFNERhM8LmlNFBbd+EWLluv04VO9AmMmTQfX2gbowqCY8phxAUh5/OQhri7JYRJkeIqUIY8BELEmKhhm6wQSA3J57qCduVglVMM1ISbwV9qstwtMCbAdyj5F3CVJBcZSYlmBOi1EDI67qTrJhWdLoIsPeYSa3aHeEY4N/LXeOD3y0Ra2FRsFrsa74Uvqrku/ulrxt+G9d871B/2iVV5jOUeXdg1W+r/Jmp1f5eXrFpMiYkKRqErgoAUl1gVT/e3tBNcVnub6kZuufLSiLsKQ/UaOZhhR4CIebzP/Zez6w+bjBsR3u5N0MkKHrMgSMQLshPCIyepmcQsaE07wkN9FAMaJERPl7snEb08V0nV9HKcVqw/V0xrD8jmfacrwLQUh9hi7jT6FEMSVk+im3ze2N5AxUHn4klZ23oPIPSXg+FzItO6WDm414mcWZFNEi1By+lEAivUA5yCXFxLxnV73/hkT99NTzynnGRgJ3tRKK786IC56xa+jn5Q/1uZLiFzRmHCeYXN+eh8TBB5LYnkXe/TR4+kvj1jnwxSOlrdPWmTJ4myNkL2hlsv+6F8UdP33XOern5KMoivV3hcq8/jrjT/4A&lt;/diagram&gt;&lt;/mxfile&gt;">
+    <defs/>
+    <g>
+        <path d="M 250 47.5 L 303.63 47.5" fill="none" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="stroke"/>
+        <path d="M 308.88 47.5 L 301.88 51 L 303.63 47.5 L 301.88 44 Z" fill="rgb(0, 0, 0)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+        <path d="M 0 95 L 20 0 L 260 0 L 240 95 Z" fill="rgb(255, 255, 255)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+        <g transform="translate(-0.5 -0.5)">
+            <switch>
+                <foreignObject pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" style="overflow: visible; text-align: left;">
+                    <div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 48px; margin-left: 1px;">
+                        <div data-drawio-colors="color: rgb(0, 0, 0); " style="box-sizing: border-box; font-size: 0px; text-align: center;">
+                            <div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; pointer-events: all; white-space: normal; overflow-wrap: normal;">
+                                <b>
+                                    <font style="font-size: 18px">
+                                        Data acquisition
+                                    </font>
+                                </b>
+                                <br/>
+                                <font style="font-size: 15px">
+                                    (Data Engineering)
+                                </font>
+                            </div>
+                        </div>
+                    </div>
+                </foreignObject>
+                <text x="130" y="51" fill="rgb(0, 0, 0)" font-family="Helvetica" font-size="12px" text-anchor="middle">
+                    Data acquisition...
+                </text>
+            </switch>
+        </g>
+        <path d="M 550 47.5 L 603.63 47.5" fill="none" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="stroke"/>
+        <path d="M 608.88 47.5 L 601.88 51 L 603.63 47.5 L 601.88 44 Z" fill="rgb(0, 0, 0)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+        <path d="M 300 95 L 320 0 L 560 0 L 540 95 Z" fill="rgb(255, 255, 255)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+        <g transform="translate(-0.5 -0.5)">
+            <switch>
+                <foreignObject pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" style="overflow: visible; text-align: left;">
+                    <div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 48px; margin-left: 301px;">
+                        <div data-drawio-colors="color: rgb(0, 0, 0); " style="box-sizing: border-box; font-size: 0px; text-align: center;">
+                            <div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; pointer-events: all; white-space: normal; overflow-wrap: normal;">
+                                <font>
+                                    <b style="font-size: 18px">
+                                        <font style="font-size: 18px">
+                                            Exploration &amp;
+                                            <br/>
+                                            model building
+                                        </font>
+                                    </b>
+                                    <br/>
+                                    <font style="font-size: 15px">
+                                        (Data Science)
+                                    </font>
+                                </font>
+                            </div>
+                        </div>
+                    </div>
+                </foreignObject>
+                <text x="430" y="51" fill="rgb(0, 0, 0)" font-family="Helvetica" font-size="12px" text-anchor="middle">
+                    Exploration &amp;...
+                </text>
+            </switch>
+        </g>
+        <path d="M 900 95 L 920 0 L 1160 0 L 1140 95 Z" fill="rgb(255, 255, 255)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+        <g transform="translate(-0.5 -0.5)">
+            <switch>
+                <foreignObject pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" style="overflow: visible; text-align: left;">
+                    <div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 48px; margin-left: 901px;">
+                        <div data-drawio-colors="color: rgb(0, 0, 0); " style="box-sizing: border-box; font-size: 0px; text-align: center;">
+                            <div style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; pointer-events: all; white-space: normal; overflow-wrap: normal;">
+                                <b style="font-size: 18px">
+                                    Deploy with standard
+                                    <br/>
+                                    organisational methods
+                                </b>
+                                <br/>
+                                <font style="font-size: 15px">
+                                    (SysAdmin, DevOps,
+                                    <br/>
+                                    or deployment SaaS)
+                                </font>
+                            </div>
+                        </div>
+                    </div>
+                </foreignObject>
+                <text x="1030" y="52" fill="rgb(0, 0, 0)" font-family="Helvetica" font-size="14px" text-anchor="middle">
+                    Deploy with standard...
+                </text>
+            </switch>
+        </g>
+        <path d="M 600 95 L 620 0 L 860 0 L 840 95 Z" fill="#1ba1e2" stroke="#006eaf" stroke-miterlimit="10" pointer-events="all"/>
+        <g transform="translate(-0.5 -0.5)">
+            <switch>
+                <foreignObject pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" style="overflow: visible; text-align: left;">
+                    <div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 48px; margin-left: 601px;">
+                        <div data-drawio-colors="color: #ffffff; " style="box-sizing: border-box; font-size: 0px; text-align: center;">
+                            <div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(255, 255, 255); line-height: 1.2; pointer-events: all; white-space: normal; overflow-wrap: normal;">
+                                <b style="font-size: 18px">
+                                    Transforming into
+                                    <br/>
+                                    production-ready service
+                                </b>
+                                <br/>
+                                <font style="font-size: 15px">
+                                    (MLOps)
+                                </font>
+                            </div>
+                        </div>
+                    </div>
+                </foreignObject>
+                <text x="730" y="51" fill="#ffffff" font-family="Helvetica" font-size="12px" text-anchor="middle">
+                    Transforming into...
+                </text>
+            </switch>
+        </g>
+        <path d="M 850 47.5 L 903.63 47.5" fill="none" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="stroke"/>
+        <path d="M 908.88 47.5 L 901.88 51 L 903.63 47.5 L 901.88 44 Z" fill="rgb(0, 0, 0)" stroke="rgb(0, 0, 0)" stroke-miterlimit="10" pointer-events="all"/>
+    </g>
+    <switch>
+        <g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/>
+        <a transform="translate(0,-5)" xlink:href="https://www.diagrams.net/doc/faq/svg-export-text-problems" target="_blank">
+            <text text-anchor="middle" font-size="10px" x="50%" y="100%">
+                Viewer does not support full SVG 1.1
+            </text>
+        </a>
+    </switch>
+</svg>
--- a/mkdocs.yaml
+++ b/mkdocs.yaml
@ -1,11 +1,17 @@
 site_name: GreatAI documentation
+site_description: GreatAI helps you easily transform your prototype AI code into production-ready software.
+site_author: András Schmelczer

 repo_url: https://github.com/schmelczer/great-ai
+repo_name: schmelczer/great-ai
 edit_uri: edit/main/docs/

 theme:
  name: "material"
  custom_dir: docs/overrides
+  features:
+    - content.code.annotate
+    - content.tooltips
  palette:
    - scheme: default
      toggle:
@ -17,6 +23,10 @@ theme:
        icon: material/brightness-4
        name: Switch to light mode

+watch:
+- docs
+- mkdocs.yaml
+
 plugins:
  - git-revision-date-localized:
  - mkdocstrings
@ -28,8 +38,27 @@ plugins:
      # execute_ignore: "my-secret-files/*.ipynb"
      # theme: dark

+markdown_extensions:
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - pymdownx.superfences
+  - pymdownx.tasklist:
+      custom_checkbox: true
+  - toc:
+      permalink: "#"
+  - admonition
+  - pymdownx.details
+  - md_in_html
+  - pymdownx.emoji:
+      emoji_index: !!python/name:materialx.emoji.twemoji
+      emoji_generator: !!python/name:materialx.emoji.to_svg
+  - abbr
+  - attr_list
+
 nav:
-  - Home: index.md
+  - Overview: index.md
  - tutorials.md
  - How-To Guides: how-to-guides.md
  - reference.md