andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 9999b7e with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-15 18:04:53 +00:00
parent 087e049514
commit 7d09422380
37 changed files with 116 additions and 108 deletions
Download patch file Download diff file Expand all files Collapse all files

12
how-to-guides/call-remote/index.html
View file

@ -438,7 +438,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile-s
How to use LargeFiles
</a>
</li>
@ -919,7 +919,7 @@
<p>Let's send multiple requests at the same time to speed up the overall execution time. To do this, we will use the <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai_async">call_remote_great_ai_async</a> function.</p>
<details class="note">
<summary>Why is this possible?</summary>
<p>Note, that in <code>server.py</code>, the inference function is declared <code>async</code>. This means that multiple "copies" of it can run at the same time in the same thread. Since, there is no CPU-bottleneck, the server has a quite large throughpout (requests responded to per second), but its latency will stay around 2 seconds due to the async <code>sleep</code> command.</p>
<p>Note, that in <code>server.py</code>, the inference function is declared <code>async</code>. This means that multiple "copies" of it can run at the same time in the same thread. Since, there is no CPU bottleneck, the server has a quite large throughput (requests responded to per second), but its latency will stay around 2 seconds due to the async <code>sleep</code> command.</p>
<p>If your great-ai server is not <code>async</code>, higher throughput can be achieved by running multiple instances of it, either manually, or by running it with multiple <code>uvicorn</code> workers like this: <code>ENVIRONMENT=production great-ai server.py --worker_count 4</code></p>
</details>
<h3 id="async-client">Async client<a class="headerlink" href="#async-client" title="Permanent link">#</a></h3>
@ -947,14 +947,14 @@
<p>Replace <code>client.py</code> with this async client. Note that even though async support is significantly more streamlined in recent Python versions, it still requires a bit more boilerplate than its synchronous counterpart.</p>
</blockquote>
<p><img alt="screenshot of result" loading="lazy" src="/media/remote-async.png" /></p>
<p>This also works, and in some use cases might be considerably quicker.</p>
<p>This also works, and in some use-cases might be considerably quicker.</p>
<hr>
<div class="md-source-file">
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>
@ -975,7 +975,7 @@
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../large_file/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to use LargeFile-s" rel="prev">
<a href="../large_file/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to use LargeFiles" rel="prev">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</div>
@ -984,7 +984,7 @@
<span class="md-footer__direction">
Previous
</span>
How to use LargeFile-s
How to use LargeFiles
</div>
</div>
</a>