131 lines
5.4 KiB
Markdown
131 lines
5.4 KiB
Markdown
# How to use a GreatAI service
|
|
|
|
After [creating a GreatAI service](/how-to-guides/create-service) by wrapping your prediction function, and optionally [configuring it](/how-to-guides/configure-service), it's time to do some prediction.
|
|
|
|
Let's take the following example:
|
|
|
|
```python title="greeter.py"
|
|
from great_ai import GreatAI
|
|
|
|
@GreatAI.create
|
|
def greeter(your_name: str) -> str:
|
|
return f'Hi {your_name}'
|
|
```
|
|
|
|
## One-off prediction
|
|
|
|
Even though `greeter` is now an instance of [GreatAI][great_ai.GreatAI], you can continue using it as a regular function.
|
|
|
|
```python
|
|
>>> greeter('Bob')
|
|
Trace[str]({'created': '2022-07-11T14:31:46.183764',
|
|
'exception': None,
|
|
'feedback': None,
|
|
'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
|
|
'models': [],
|
|
'original_execution_time_ms': 0.0381,
|
|
'output': 'Hi Bob',
|
|
'tags': ['greeter', 'online', 'development'],
|
|
'trace_id': '7c284fd7-7f0d-4464-b5f8-3ef126df34af'})
|
|
```
|
|
|
|
As you can see, the original return value is wrapped in a [Trace][great_ai.Trace] object (which is also persisted in your database of choice). You can access the original value under the `output` property.
|
|
|
|
## Online prediction
|
|
|
|
Likely, the main way you would like to expose your model is through an HTTP API. [@GreatAI.create][great_ai.GreatAI.create] scaffolds many REST API endpoints for your model and creates a [FastAPI](https://fastapi.tiangolo.com/){ target=_blank } app available under [GreatAI.app][great_ai.GreatAI]. This can be served using [uvicorn](https://www.uvicorn.org/){ target=_blank } or any other [ASGI server](https://asgi.readthedocs.io/en/latest/){ target=_blank }.
|
|
|
|
Since most ML code lives in [Jupyter](https://jupyter.org/){ target=_blank } notebooks, therefore, deploying a notebook containing the inference function is supported. To this end, `uvicorn` is wrapped by the `great-ai` command-line utility which, among others, takes care of feeding a notebook into `uvicorn`. It also supports auto-reloading.
|
|
|
|
### In development
|
|
|
|
```sh
|
|
great-ai greeter.py
|
|
```
|
|
|
|
!!! success
|
|
Your model is accessible at [localhost:6060](http:/127.0.0.1:6060){ target=_blank }.
|
|
|
|
Some configuration options are also supported.
|
|
|
|
```sh
|
|
great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10
|
|
```
|
|
> For more options (but no Notebook support, use [uvicorn](https://www.uvicorn.org/){ target=_blank })
|
|
|
|
### In production
|
|
|
|
There are three main approaches for deploying a GreatAI service.
|
|
|
|
#### Manual deployment
|
|
|
|
The app is run in *production-mode* if the value of the `ENVIRONMENT` environment variable is set to `production`.
|
|
|
|
```sh
|
|
ENVIRONMENT=production great-ai greeter.py
|
|
```
|
|
|
|
Simply run `ENVIRONMENT=production great-ai deploy.ipynb` in the command-line of a production machine.
|
|
> This is the crudest approach, however, it might be fitting for some contexts.
|
|
|
|
#### Containerised deployment
|
|
|
|
Run the notebook directly in a container or create a service for it using your favourite container orchestrator.
|
|
|
|
```sh
|
|
docker run -p 6060:6060 --volume `pwd`:/app --rm \
|
|
schmelczera/great-ai deploy.ipynb
|
|
```
|
|
> You can replace ``pwd`` with the path to your code's folder.
|
|
|
|
#### Use a Platform-as-a-Service
|
|
|
|
Similarly to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as [DO App platform](https://www.digitalocean.com/products/app-platform){ target=_blank } or [MLEM](https://mlem.ai/){ target=_blank }) that take a Docker image as a source and handle the rest of the deployment.
|
|
|
|
To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as [PyTorch](https://pytorch.org/){ target=_blank } or [TensorFlow](https://www.tensorflow.org/){ target=_blank }.
|
|
|
|
```Dockerfile
|
|
FROM schmelczera/great-ai:latest
|
|
|
|
# Remove this block if you don't have a requirements.txt
|
|
COPY requirements.txt ./
|
|
RUN pip install --no-cache-dir --requirement requirements.txt
|
|
|
|
# If you store your models in S3 or GridFS, it may be a
|
|
# good idea to cache them in the image so that you don't
|
|
# have to download it each time a container starts
|
|
RUN large-file --backend s3 --secrets s3.ini --cache my-domain-predictor
|
|
|
|
# Add you application code to the image
|
|
COPY . .
|
|
|
|
# The default ENTRYPOINT is great-ai, specify it's argument using CMD
|
|
CMD ["deploy.ipynb"]
|
|
|
|
```
|
|
|
|
## Batch prediction
|
|
|
|
Processing larger amounts of data on a single machine is made easy by the [GreatAI][great_ai.GreatAI]'s [process_batch][great_ai.GreatAI.process_batch] method. This relies on multiprocessing ([parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map]) to take full advantage of all available CPU-cores.
|
|
|
|
```python
|
|
>>> greeter.process_batch(['Alice', 'Bob'])
|
|
[Trace[str]({'created': '2022-07-11T14:36:37.119183',
|
|
'exception': None,
|
|
'feedback': None,
|
|
'logged_values': {'arg:your_name:length': 5, 'arg:your_name:value': 'Alice'},
|
|
'models': [],
|
|
'original_execution_time_ms': 0.1251,
|
|
'output': 'Hi Alice',
|
|
'tags': ['greeter', 'online', 'development'],
|
|
'trace_id': '90ffa15f-e839-41c4-8e7a-3211168bc138'}),
|
|
Trace[str]({'created': '2022-07-11T14:36:37.166659',
|
|
'exception': None,
|
|
'feedback': None,
|
|
'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
|
|
'models': [],
|
|
'original_execution_time_ms': 0.0571,
|
|
'output': 'Hi Bob',
|
|
'tags': ['greeter', 'online', 'development'],
|
|
'trace_id': 'f48e94c7-0815-48b3-a864-41349d3dae84'})]
|
|
```
|