Improve docs
This commit is contained in:
parent
446f1c021a
commit
1dcf5bb3af
6 changed files with 117 additions and 4 deletions
|
|
@ -1,2 +1,90 @@
|
||||||
# Call remote GreatAI instances
|
# How to call remote GreatAI instances
|
||||||
|
|
||||||
|
Microservices architecture (or [SOA](https://en.wikipedia.org/wiki/Service-oriented_architecture)) work well with ML applications. This is because their interfaces are usually very narrow, while the functionality provided quite comprehensive. Hence, drawing the boundaries of responsibilities is more straightforward in the case of ML services than in the case of more traditional business applications. For this reason, it is common to have a tree of models (preferably wrapped in GreatAI instances) communicating with each other.
|
||||||
|
|
||||||
|
Although regular HTTP POST requests could be sent to each service's `/predict` endpoint, `great-ai` comes with two convenience functions: [call_remote_great_ai][great_ai.call_remote_great_ai] and [call_remote_great_ai_async][great_ai.call_remote_great_ai_async] to wrap this request. These provide you with some level of robustness and deserialisation.
|
||||||
|
|
||||||
|
!!! note "Inside notebooks"
|
||||||
|
The async variant, [call_remote_great_ai_async][great_ai.call_remote_great_ai_async], requires a running event loop while the synchronous variant disallows other running event-loops. Therefore, when running inside a Jupyter Notebook, always call [call_remote_great_ai_async][great_ai.call_remote_great_ai_async].
|
||||||
|
|
||||||
|
## Simple example
|
||||||
|
|
||||||
|
Let's create two processes: a server and a client.
|
||||||
|
|
||||||
|
### Server
|
||||||
|
|
||||||
|
```python title="server.py"
|
||||||
|
from great_ai import GreatAI
|
||||||
|
from asyncio import sleep
|
||||||
|
|
||||||
|
@GreatAI.create
|
||||||
|
async def slow_greeter(your_name):
|
||||||
|
await sleep(2)
|
||||||
|
return f'Hi {your_name}!'
|
||||||
|
```
|
||||||
|
> Run this in development mode by executing `great-ai server.py` or `python3 -m great_ai server.py` if you're on Windows and [`great-ai` is not in your `PATH`](/how-to-guides/install).
|
||||||
|
|
||||||
|
### Client
|
||||||
|
|
||||||
|
```python title="client.py"
|
||||||
|
from great_ai import call_remote_great_ai
|
||||||
|
|
||||||
|
names = ['Olivér', 'Balázs', 'András']
|
||||||
|
|
||||||
|
results = [
|
||||||
|
call_remote_great_ai(
|
||||||
|
'http://localhost:6060',
|
||||||
|
{
|
||||||
|
'your_name': name
|
||||||
|
}
|
||||||
|
).output #(1)
|
||||||
|
for name in names
|
||||||
|
]
|
||||||
|
|
||||||
|
print(results)
|
||||||
|
```
|
||||||
|
1. Only return the outputs so we don't clutter up the terminal.
|
||||||
|
|
||||||
|
> Run this script as a regular Python script by executing `python3 client.py`.
|
||||||
|
|
||||||
|
{ loading=lazy }
|
||||||
|
|
||||||
|
As you can see, everything worked as expected. There is one way to improve it though.
|
||||||
|
## An `async` example
|
||||||
|
|
||||||
|
Let's send multiple requests at the same time to speed up the overall execution time. To do this, we will use the [call_remote_great_ai_async][great_ai.call_remote_great_ai_async] function.
|
||||||
|
|
||||||
|
??? note "Why is this possible?"
|
||||||
|
Note, that in `server.py`, the inference function is declared `async`. This means that multiple "copies" of it can run at the same time in the same thread. Since, there is no CPU-bottleneck, the server has a quite large throughpout (requests responded to per second), but its latency will stay around 2 seconds due to the async `sleep` command.
|
||||||
|
|
||||||
|
If your great-ai server is not `async`, higher throughput can be achieved by running multiple instances of it, either manually, or by running it with multiple `uvicorn` workers like this: `ENVIRONMENT=production great-ai server.py --worker_count 4`
|
||||||
|
|
||||||
|
### Async client
|
||||||
|
|
||||||
|
```python title="async-client.py"
|
||||||
|
from great_ai import call_remote_great_ai_async
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
names = ['Olivér', 'Balázs', 'András']
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
futures = [
|
||||||
|
call_remote_great_ai_async(
|
||||||
|
'http://localhost:6060',
|
||||||
|
{
|
||||||
|
'your_name': name
|
||||||
|
}
|
||||||
|
) for name in names
|
||||||
|
]
|
||||||
|
|
||||||
|
results = await asyncio.gather(*futures)
|
||||||
|
print([r.output for r in results])
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
> Replace `client.py` with this async client. Note that even though async support is significantly more streamlined in recent Python versions, it still requires a bit more boilerplate than its synchronous counterpart.
|
||||||
|
|
||||||
|
{ loading=lazy }
|
||||||
|
|
||||||
|
This also works, and in some use cases might be considerably quicker.
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,16 @@
|
||||||
# How to use LargeFile-s
|
# How to use LargeFile-s
|
||||||
|
|
||||||
The functions [save_model][great_ai.use_model] and [@use_model][great_ai.use_model] wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use this.
|
The functions [save_model][great_ai.use_model] and [@use_model][great_ai.use_model] wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use LargeFile-s directly.
|
||||||
|
|
||||||
## Motivation
|
## Motivation
|
||||||
|
|
||||||
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support deleting](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
|
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support deleting](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
|
||||||
|
|
||||||
|
[DVC](https://dvc.org/) is a viable alternative, however, it requires users to learn to use one more CLI tool.
|
||||||
|
|
||||||
??? note "Using LargeFile-s directly (usually not needed)"
|
??? note "Using LargeFile-s directly (usually not needed)"
|
||||||
|
LargeFile doesn't require users to learn too much new. It is a nearly exact copy of the built-in `open()` function of Python with which users are certainly already familiar.
|
||||||
|
|
||||||
## Simple example
|
## Simple example
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|
@ -61,7 +64,7 @@ Oftentimes, especially when working with data-heavy applications, large files ca
|
||||||
LargeFileS3("folder-of-my-bert-model").push('path_to_local/folder_or_file')
|
LargeFileS3("folder-of-my-bert-model").push('path_to_local/folder_or_file')
|
||||||
```
|
```
|
||||||
|
|
||||||
> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
|
> This way both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
|
||||||
|
|
||||||
Lastly, all version of the remote object can be deleted by calling `LargeFileS3("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
|
Lastly, all version of the remote object can be deleted by calling `LargeFileS3("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
|
||||||
|
|
||||||
|
|
|
||||||
BIN
docs/media/remote-async.png
Normal file
BIN
docs/media/remote-async.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
BIN
docs/media/remote-sync.png
Normal file
BIN
docs/media/remote-sync.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
|
|
@ -3,3 +3,23 @@
|
||||||
::: great_ai.Trace
|
::: great_ai.Trace
|
||||||
options:
|
options:
|
||||||
show_root_heading: true
|
show_root_heading: true
|
||||||
|
|
||||||
|
::: great_ai.RouteConfig
|
||||||
|
options:
|
||||||
|
show_root_heading: true
|
||||||
|
|
||||||
|
::: great_ai.ClassificationOutput
|
||||||
|
options:
|
||||||
|
show_root_heading: true
|
||||||
|
|
||||||
|
::: great_ai.MultiLabelClassificationOutput
|
||||||
|
options:
|
||||||
|
show_root_heading: true
|
||||||
|
|
||||||
|
::: great_ai.RegressionOutput
|
||||||
|
options:
|
||||||
|
show_root_heading: true
|
||||||
|
|
||||||
|
::: great_ai.SequenceLabelingOutput
|
||||||
|
options:
|
||||||
|
show_root_heading: true
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,9 @@ async def call_remote_great_ai_async(
|
||||||
try:
|
try:
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
except Exception:
|
except Exception:
|
||||||
raise RemoteCallError("Unexpected status code")
|
raise RemoteCallError(
|
||||||
|
f"Unexpected status code, reason: {response.text}"
|
||||||
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise RemoteCallError from e
|
raise RemoteCallError from e
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue