How to deploy a GreatAI service

After creating a GreatAI service by wrapping your prediction function, and optionally configuring it, it's time to do some prediction.

Let's take the following example:

from great_ai import GreatAI

@GreatAI.create
def greeter(your_name: str) -> str:
    return f'Hi {your_name}'

One-off prediction

Even though greeter is now an instance of [GreatAI][great_ai.GreatAI], you can continue using it as a regular function.

>>> greeter('Bob')
Trace[str]({'created': '2022-07-11T14:31:46.183764',
  'exception': None,
  'feedback': None,
  'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
  'models': [],
  'original_execution_time_ms': 0.0381,
  'output': 'Hi Bob',
  'tags': ['greeter', 'online', 'development'],
  'trace_id': '7c284fd7-7f0d-4464-b5f8-3ef126df34af'})

As you can see, the original return value is wrapped in a [Trace][great_ai.Trace] object (which is also persisted in your database of choice). You can access the original value under the output property.

Online prediction

Likely, the main way you would like to expose your model is through an HTTP API. [@GreatAI.create][great_ai.GreatAI.create] scaffolds many REST API endpoints for your model and creates a FastAPI{ target=_blank } app available under [GreatAI.app][great_ai.GreatAI]. This can be served using uvicorn{ target=_blank } or any other ASGI server{ target=_blank }.

Since most ML code lives in Jupyter{ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To this end, uvicorn is wrapped by the great-ai command-line utility which, among others, takes care of feeding a notebook into uvicorn. It also supports auto-reloading.

In development

great-ai greeter.py

!!! success Your model is accessible at localhost:6060{ target=_blank }.

Some configuration options are also supported.

great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10

??? note "More options" For more options (but no Notebook support), simply use uvicorn{ target=_blank } for starting your app (available at greeter.app).

In production

There are three main approaches for deploying a GreatAI service.

Manual deployment

The app is run in production-mode if the value of the ENVIRONMENT environment variable is set to production.

ENVIRONMENT=production great-ai greeter.py

Simply run ENVIRONMENT=production great-ai deploy.ipynb in the command-line of a production machine.

This is the crudest approach, however, it might be fitting for some contexts.

Containerised deployment

Run the notebook directly in a container or create a service for it using your favourite container orchestrator.

docker run -p 6060:6060 --volume `pwd`:/app --rm \
  schmelczera/great-ai deploy.ipynb

You can replace pwd with the path to your code's folder.

Use a Platform-as-a-Service

Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as AWS ECS{ target=_blank }, DO App platform{ target=_blank }, MLEM{ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.

To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as PyTorch{ target=_blank } or TensorFlow{ target=_blank }.

FROM schmelczera/great-ai:latest

# Remove this block if you don't have a requirements.txt
COPY requirements.txt ./   
RUN pip install --no-cache-dir --requirement requirements.txt

# If you store your models in S3 or GridFS, it may be a 
# good idea to cache them in the image so that you don't
# have to download it each time a container starts
RUN large-file --backend s3 --secrets s3.ini --cache my-domain-predictor

# Add you application code to the image
COPY . .

# The default ENTRYPOINT is great-ai, specify it's argument using CMD
CMD ["deploy.ipynb"]

Batch prediction

Processing larger amounts of data on a single machine is made easy by the [GreatAI][great_ai.GreatAI]'s [process_batch][great_ai.GreatAI.process_batch] method. This relies on multiprocessing ([parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map]) to take full advantage of all available CPU-cores.

>>> greeter.process_batch(['Alice', 'Bob'])
[Trace[str]({'created': '2022-07-11T14:36:37.119183',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 5, 'arg:your_name:value': 'Alice'},
   'models': [],
   'original_execution_time_ms':  0.1251,
   'output': 'Hi Alice',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': '90ffa15f-e839-41c4-8e7a-3211168bc138'}),
 Trace[str]({'created': '2022-07-11T14:36:37.166659',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
   'models': [],
   'original_execution_time_ms':  0.0571,
   'output': 'Hi Bob',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': 'f48e94c7-0815-48b3-a864-41349d3dae84'})]

5.5 KiB Raw Blame History