Deployed 03765e1 with MkDocs version: 1.3.0
This commit is contained in:
parent
8cda980b81
commit
f5267b5a06
31 changed files with 725 additions and 91 deletions
|
|
@ -17,7 +17,7 @@
|
|||
|
||||
|
||||
|
||||
<title>Call remote GreatAI instances - GreatAI documentation</title>
|
||||
<title>How to call remote GreatAI instances - GreatAI documentation</title>
|
||||
|
||||
|
||||
|
||||
|
|
@ -83,7 +83,7 @@
|
|||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#call-remote-greatai-instances" class="md-skip">
|
||||
<a href="#how-to-call-remote-greatai-instances" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
|
|
@ -116,7 +116,7 @@
|
|||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Call remote GreatAI instances
|
||||
How to call remote GreatAI instances
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
|
@ -460,10 +460,81 @@
|
|||
|
||||
|
||||
|
||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||
How to call remote GreatAI instances
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<a href="./" class="md-nav__link md-nav__link--active">
|
||||
Call remote GreatAI instances
|
||||
How to call remote GreatAI instances
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#simple-example" class="md-nav__link">
|
||||
Simple example
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Simple example">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#server" class="md-nav__link">
|
||||
Server
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#client" class="md-nav__link">
|
||||
Client
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#an-async-example" class="md-nav__link">
|
||||
An async example
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="An async example">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#async-client" class="md-nav__link">
|
||||
Async client
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
|
@ -723,6 +794,61 @@
|
|||
|
||||
|
||||
|
||||
<label class="md-nav__title" for="__toc">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
Table of contents
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#simple-example" class="md-nav__link">
|
||||
Simple example
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Simple example">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#server" class="md-nav__link">
|
||||
Server
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#client" class="md-nav__link">
|
||||
Client
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#an-async-example" class="md-nav__link">
|
||||
An async example
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="An async example">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#async-client" class="md-nav__link">
|
||||
Async client
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -743,14 +869,90 @@
|
|||
|
||||
|
||||
|
||||
<h1 id="call-remote-greatai-instances">Call remote GreatAI instances<a class="headerlink" href="#call-remote-greatai-instances" title="Permanent link">#</a></h1>
|
||||
<h1 id="how-to-call-remote-greatai-instances">How to call remote GreatAI instances<a class="headerlink" href="#how-to-call-remote-greatai-instances" title="Permanent link">#</a></h1>
|
||||
<p>Microservices architecture (or <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">SOA</a>) work well with ML applications. This is because their interfaces are usually very narrow, while the functionality provided quite comprehensive. Hence, drawing the boundaries of responsibilities is more straightforward in the case of ML services than in the case of more traditional business applications. For this reason, it is common to have a tree of models (preferably wrapped in GreatAI instances) communicating with each other.</p>
|
||||
<p>Although regular HTTP POST requests could be sent to each service's <code>/predict</code> endpoint, <code>great-ai</code> comes with two convenience functions: <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai">call_remote_great_ai</a> and <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai_async">call_remote_great_ai_async</a> to wrap this request. These provide you with some level of robustness and deserialisation.</p>
|
||||
<div class="admonition note">
|
||||
<p class="admonition-title">Inside notebooks</p>
|
||||
<p>The async variant, <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai_async">call_remote_great_ai_async</a>, requires a running event loop while the synchronous variant disallows other running event-loops. Therefore, when running inside a Jupyter Notebook, always call <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai_async">call_remote_great_ai_async</a>.</p>
|
||||
</div>
|
||||
<h2 id="simple-example">Simple example<a class="headerlink" href="#simple-example" title="Permanent link">#</a></h2>
|
||||
<p>Let's create two processes: a server and a client.</p>
|
||||
<h3 id="server">Server<a class="headerlink" href="#server" title="Permanent link">#</a></h3>
|
||||
<div class="highlight"><span class="filename">server.py</span><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">GreatAI</span>
|
||||
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a><span class="kn">from</span> <span class="nn">asyncio</span> <span class="kn">import</span> <span class="n">sleep</span>
|
||||
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a>
|
||||
<a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a><span class="nd">@GreatAI</span><span class="o">.</span><span class="n">create</span>
|
||||
<a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a><span class="k">async</span> <span class="k">def</span> <span class="nf">slow_greeter</span><span class="p">(</span><span class="n">your_name</span><span class="p">):</span>
|
||||
<a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a> <span class="k">await</span> <span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
|
||||
<a id="__codelineno-0-7" name="__codelineno-0-7" href="#__codelineno-0-7"></a> <span class="k">return</span> <span class="sa">f</span><span class="s1">'Hi </span><span class="si">{</span><span class="n">your_name</span><span class="si">}</span><span class="s1">!'</span>
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>Run this in development mode by executing <code>great-ai server.py</code> or <code>python3 -m great_ai server.py</code> if you're on Windows and <a href="/how-to-guides/install"><code>great-ai</code> is not in your <code>PATH</code></a>.</p>
|
||||
</blockquote>
|
||||
<h3 id="client">Client<a class="headerlink" href="#client" title="Permanent link">#</a></h3>
|
||||
<p><div class="highlight"><span class="filename">client.py</span><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">call_remote_great_ai</span>
|
||||
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a>
|
||||
<a id="__codelineno-1-3" name="__codelineno-1-3" href="#__codelineno-1-3"></a><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'Olivér'</span><span class="p">,</span> <span class="s1">'Balázs'</span><span class="p">,</span> <span class="s1">'András'</span><span class="p">]</span>
|
||||
<a id="__codelineno-1-4" name="__codelineno-1-4" href="#__codelineno-1-4"></a>
|
||||
<a id="__codelineno-1-5" name="__codelineno-1-5" href="#__codelineno-1-5"></a><span class="n">results</span> <span class="o">=</span> <span class="p">[</span>
|
||||
<a id="__codelineno-1-6" name="__codelineno-1-6" href="#__codelineno-1-6"></a> <span class="n">call_remote_great_ai</span><span class="p">(</span>
|
||||
<a id="__codelineno-1-7" name="__codelineno-1-7" href="#__codelineno-1-7"></a> <span class="s1">'http://localhost:6060'</span><span class="p">,</span>
|
||||
<a id="__codelineno-1-8" name="__codelineno-1-8" href="#__codelineno-1-8"></a> <span class="p">{</span>
|
||||
<a id="__codelineno-1-9" name="__codelineno-1-9" href="#__codelineno-1-9"></a> <span class="s1">'your_name'</span><span class="p">:</span> <span class="n">name</span>
|
||||
<a id="__codelineno-1-10" name="__codelineno-1-10" href="#__codelineno-1-10"></a> <span class="p">}</span>
|
||||
<a id="__codelineno-1-11" name="__codelineno-1-11" href="#__codelineno-1-11"></a> <span class="p">)</span><span class="o">.</span><span class="n">output</span> <span class="c1">#(1)</span>
|
||||
<a id="__codelineno-1-12" name="__codelineno-1-12" href="#__codelineno-1-12"></a> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span>
|
||||
<a id="__codelineno-1-13" name="__codelineno-1-13" href="#__codelineno-1-13"></a><span class="p">]</span>
|
||||
<a id="__codelineno-1-14" name="__codelineno-1-14" href="#__codelineno-1-14"></a>
|
||||
<a id="__codelineno-1-15" name="__codelineno-1-15" href="#__codelineno-1-15"></a><span class="nb">print</span><span class="p">(</span><span class="n">results</span><span class="p">)</span>
|
||||
</code></pre></div>
|
||||
1. Only return the outputs so we don't clutter up the terminal.</p>
|
||||
<blockquote>
|
||||
<p>Run this script as a regular Python script by executing <code>python3 client.py</code>.</p>
|
||||
</blockquote>
|
||||
<p><img alt="screenshot of result" loading="lazy" src="/media/remote-sync.png" /></p>
|
||||
<p>As you can see, everything worked as expected. There is one way to improve it though.</p>
|
||||
<h2 id="an-async-example">An <code>async</code> example<a class="headerlink" href="#an-async-example" title="Permanent link">#</a></h2>
|
||||
<p>Let's send multiple requests at the same time to speed up the overall execution time. To do this, we will use the <a class="autorefs autorefs-internal" href="../../reference/#great_ai.call_remote_great_ai_async">call_remote_great_ai_async</a> function.</p>
|
||||
<details class="note">
|
||||
<summary>Why is this possible?</summary>
|
||||
<p>Note, that in <code>server.py</code>, the inference function is declared <code>async</code>. This means that multiple "copies" of it can run at the same time in the same thread. Since, there is no CPU-bottleneck, the server has a quite large throughpout (requests responded to per second), but its latency will stay around 2 seconds due to the async <code>sleep</code> command.</p>
|
||||
<p>If your great-ai server is not <code>async</code>, higher throughput can be achieved by running multiple instances of it, either manually, or by running it with multiple <code>uvicorn</code> workers like this: <code>ENVIRONMENT=production great-ai server.py --worker_count 4</code></p>
|
||||
</details>
|
||||
<h3 id="async-client">Async client<a class="headerlink" href="#async-client" title="Permanent link">#</a></h3>
|
||||
<div class="highlight"><span class="filename">async-client.py</span><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">call_remote_great_ai_async</span>
|
||||
<a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a><span class="kn">import</span> <span class="nn">asyncio</span>
|
||||
<a id="__codelineno-2-3" name="__codelineno-2-3" href="#__codelineno-2-3"></a>
|
||||
<a id="__codelineno-2-4" name="__codelineno-2-4" href="#__codelineno-2-4"></a><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'Olivér'</span><span class="p">,</span> <span class="s1">'Balázs'</span><span class="p">,</span> <span class="s1">'András'</span><span class="p">]</span>
|
||||
<a id="__codelineno-2-5" name="__codelineno-2-5" href="#__codelineno-2-5"></a>
|
||||
<a id="__codelineno-2-6" name="__codelineno-2-6" href="#__codelineno-2-6"></a><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
|
||||
<a id="__codelineno-2-7" name="__codelineno-2-7" href="#__codelineno-2-7"></a> <span class="n">futures</span> <span class="o">=</span> <span class="p">[</span>
|
||||
<a id="__codelineno-2-8" name="__codelineno-2-8" href="#__codelineno-2-8"></a> <span class="n">call_remote_great_ai_async</span><span class="p">(</span>
|
||||
<a id="__codelineno-2-9" name="__codelineno-2-9" href="#__codelineno-2-9"></a> <span class="s1">'http://localhost:6060'</span><span class="p">,</span>
|
||||
<a id="__codelineno-2-10" name="__codelineno-2-10" href="#__codelineno-2-10"></a> <span class="p">{</span>
|
||||
<a id="__codelineno-2-11" name="__codelineno-2-11" href="#__codelineno-2-11"></a> <span class="s1">'your_name'</span><span class="p">:</span> <span class="n">name</span>
|
||||
<a id="__codelineno-2-12" name="__codelineno-2-12" href="#__codelineno-2-12"></a> <span class="p">}</span>
|
||||
<a id="__codelineno-2-13" name="__codelineno-2-13" href="#__codelineno-2-13"></a> <span class="p">)</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span>
|
||||
<a id="__codelineno-2-14" name="__codelineno-2-14" href="#__codelineno-2-14"></a> <span class="p">]</span>
|
||||
<a id="__codelineno-2-15" name="__codelineno-2-15" href="#__codelineno-2-15"></a>
|
||||
<a id="__codelineno-2-16" name="__codelineno-2-16" href="#__codelineno-2-16"></a> <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">futures</span><span class="p">)</span>
|
||||
<a id="__codelineno-2-17" name="__codelineno-2-17" href="#__codelineno-2-17"></a> <span class="nb">print</span><span class="p">([</span><span class="n">r</span><span class="o">.</span><span class="n">output</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span><span class="p">])</span>
|
||||
<a id="__codelineno-2-18" name="__codelineno-2-18" href="#__codelineno-2-18"></a>
|
||||
<a id="__codelineno-2-19" name="__codelineno-2-19" href="#__codelineno-2-19"></a><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
|
||||
</code></pre></div>
|
||||
<blockquote>
|
||||
<p>Replace <code>client.py</code> with this async client. Note that even though async support is significantly more streamlined in recent Python versions, it still requires a bit more boilerplate than its synchronous counterpart.</p>
|
||||
</blockquote>
|
||||
<p><img alt="screenshot of result" loading="lazy" src="/media/remote-async.png" /></p>
|
||||
<p>This also works, and in some use cases might be considerably quicker.</p>
|
||||
|
||||
<hr>
|
||||
<div class="md-source-file">
|
||||
<small>
|
||||
|
||||
Last update:
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
|
||||
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
|
||||
|
||||
|
||||
</small>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue