andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 0585a4e with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-12 19:53:07 +00:00
parent 7305d2ead3
commit 8cda980b81
33 changed files with 782 additions and 464 deletions
Download patch file Download diff file Expand all files Collapse all files

64
how-to-guides/call-remote/index.html
View file

@ -337,12 +337,14 @@
<li class="md-nav__item md-nav__item--nested">
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_3" type="checkbox" id="__nav_3" >
<input class="md-nav__toggle md-toggle" data-md-toggle="__nav_3" type="checkbox" id="__nav_3" checked>
@ -408,7 +410,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -422,7 +424,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -436,7 +438,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -447,11 +449,21 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" data-md-toggle="toc" type="checkbox" id="__toc">
<a href="./" class="md-nav__link md-nav__link--active">
Call remote GreatAI instances
</a>
</li>
@ -755,6 +767,42 @@
<footer class="md-footer">
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../large_file/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to use LargeFile-s" rel="prev">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</div>
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Previous
</span>
How to use LargeFile-s
</div>
</div>
</a>
<a href="../scraping/" class="md-footer__link md-footer__link--next" aria-label="Next: Scraping" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
Scraping
</div>
</div>
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4Z"/></svg>
</div>
</a>
</nav>
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">

14
how-to-guides/configure-service/index.html
View file

@ -498,7 +498,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -512,7 +512,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -526,7 +526,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -539,8 +539,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>
@ -1009,13 +1009,13 @@
<a href="../use-service/" class="md-footer__link md-footer__link--next" aria-label="Next: How to use a GreatAI service" rel="next">
<a href="../use-service/" class="md-footer__link md-footer__link--next" aria-label="Next: How to deploy a GreatAI service" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
How to use a GreatAI service
How to deploy a GreatAI service
</div>
</div>
<div class="md-footer__button md-icon">

10
how-to-guides/create-service/index.html
View file

@ -492,7 +492,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -506,7 +506,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -520,7 +520,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -533,8 +533,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>

67
how-to-guides/handle-training-data/handle-training-data.md
View file

@ -1,6 +1,69 @@
# How to handle training data
# How to manage training data
In order to simplify your training data management, `great-ai` provide two complementing approaches for inputting new data-points.
## Upload data
## Use feedback
At the start of your experiments' first iteration, after you've gathered suitable samples for training, you can call [great_ai.add_ground_truth][]. This automatically stores a timestamp and also allows you to assign tags to the data. Using these attributes, [great_ai.query_ground_truth][] can be called to get a filtered view of the training data.
!!! important "Train-test-validation splits"
It is a best-practice to lock-away a test split of your data that is only used for the final quality assessment. This prevents you from accidentally training on it, or inadvertently tuning the model to have the highest accuracy metrics on the test split. This, of course, may lead to dubious results, hence, care must be taken to avoid it.
With [great_ai.add_ground_truth][], there is an option to tag the samples with `train`, `test`, and `validation` randomly, following a predefined distribution. This happens as soon as they're written in the database. Later on, these can be queried by providing the name of the appropriate tags.
The nice thing about this is that the 'input-expected output' pairs are stored as traces. Thus, they behave exactly like regular prediction traces.
```python
from great_ai import add_ground_truth
add_ground_truth(
[1, 2],
['odd', 'even'],
tags='my_tag',
train_split_ratio=1, #(1)
test_split_ratio=1
)
```
1. Note that the ratios don't have to add up to 1. They are just weights. There is also a `validation_split_ratio` which is 0 by default.
```python
>>> from great_ai import query_ground_truth
>>> query_ground_truth('my_tag')
[Trace[str]({'created': '2022-07-12T18:36:12.825706',
'exception': None,
'feedback': 'odd', #(1)
'logged_values': {'input': 1}, #(2)
'models': [],
'original_execution_time_ms': 0.0,
'output': 'odd',
'tags': ['ground_truth', 'test', 'my_tag'], #(3)
'trace_id': '4fcf2ce6-a172-469d-94b2-874577655814'}),
Trace[str]({'created': '2022-07-12T18:36:12.825706',
'exception': None,
'feedback': 'even',
'logged_values': {'input': 2},
'models': [],
'original_execution_time_ms': 0.0,
'output': 'even',
'tags': ['ground_truth', 'train', 'my_tag'],
'trace_id': 'abee0671-beb9-4284-8c3b-c65e5836ce38'})]
```
1. Expected output. This can be also accessed through the `.output` property.
2. The input value is stored here.
3. Notice how `ground_truth` always included as a tag when using [great_ai.add_ground_truth][].
## Get feedback
After the initial data gathering, end-to-end feedback can be also integrated into the dataset.
The scaffolded REST API contains endpoints for managing traces and their feedbacks.
![screenshot of swagger](/media/feedback.png){ loading=lazy }
When [great_ai.query_ground_truth][] is executed, it implicitly filters for traces that have feedback. Therefore, both the `ground_truth` and the `online` traces that have received feedback are returned. No matter the origin of the data, it can be accessed using the same API.
## Remove clutter
Traces can be deleted either through the REST API or by calling [great_ai.delete_ground_truth][]. The latter provides the same interface as [great_ai.query_ground_truth][] except it deletes the matched points.

107
how-to-guides/handle-training-data/index.html
View file

@ -17,7 +17,7 @@
<title>How to handle training data - GreatAI documentation</title>
<title>How to manage training data - GreatAI documentation</title>
@ -83,7 +83,7 @@
<div data-md-component="skip">
<a href="#how-to-handle-training-data" class="md-skip">
<a href="#how-to-manage-training-data" class="md-skip">
Skip to content
</a>
@ -116,7 +116,7 @@
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
How to handle training data
How to manage training data
</span>
</div>
@ -410,7 +410,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -433,12 +433,12 @@
<label class="md-nav__link md-nav__link--active" for="__toc">
How to handle training data
How to manage training data
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
How to handle training data
How to manage training data
</a>
@ -464,8 +464,15 @@
</li>
<li class="md-nav__item">
<a href="#use-feedback" class="md-nav__link">
Use feedback
<a href="#get-feedback" class="md-nav__link">
Get feedback
</a>
</li>
<li class="md-nav__item">
<a href="#remove-clutter" class="md-nav__link">
Remove clutter
</a>
</li>
@ -486,7 +493,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -499,8 +506,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>
@ -775,8 +782,15 @@
</li>
<li class="md-nav__item">
<a href="#use-feedback" class="md-nav__link">
Use feedback
<a href="#get-feedback" class="md-nav__link">
Get feedback
</a>
</li>
<li class="md-nav__item">
<a href="#remove-clutter" class="md-nav__link">
Remove clutter
</a>
</li>
@ -803,16 +817,69 @@
<h1 id="how-to-handle-training-data">How to handle training data<a class="headerlink" href="#how-to-handle-training-data" title="Permanent link">#</a></h1>
<h1 id="how-to-manage-training-data">How to manage training data<a class="headerlink" href="#how-to-manage-training-data" title="Permanent link">#</a></h1>
<p>In order to simplify your training data management, <code>great-ai</code> provide two complementing approaches for inputting new data-points.</p>
<h2 id="upload-data">Upload data<a class="headerlink" href="#upload-data" title="Permanent link">#</a></h2>
<h2 id="use-feedback">Use feedback<a class="headerlink" href="#use-feedback" title="Permanent link">#</a></h2>
<p>At the start of your experiments' first iteration, after you've gathered suitable samples for training, you can call <a class="autorefs autorefs-internal" href="../../reference/#great_ai.add_ground_truth">great_ai.add_ground_truth</a>. This automatically stores a timestamp and also allows you to assign tags to the data. Using these attributes, <a class="autorefs autorefs-internal" href="../../reference/#great_ai.query_ground_truth">great_ai.query_ground_truth</a> can be called to get a filtered view of the training data.</p>
<div class="admonition important">
<p class="admonition-title">Train-test-validation splits</p>
<p>It is a best-practice to lock-away a test split of your data that is only used for the final quality assessment. This prevents you from accidentally training on it, or inadvertently tuning the model to have the highest accuracy metrics on the test split. This, of course, may lead to dubious results, hence, care must be taken to avoid it.</p>
<p>With <a class="autorefs autorefs-internal" href="../../reference/#great_ai.add_ground_truth">great_ai.add_ground_truth</a>, there is an option to tag the samples with <code>train</code>, <code>test</code>, and <code>validation</code> randomly, following a predefined distribution. This happens as soon as they're written in the database. Later on, these can be queried by providing the name of the appropriate tags.</p>
</div>
<p>The nice thing about this is that the 'input-expected output' pairs are stored as traces. Thus, they behave exactly like regular prediction traces.</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">add_ground_truth</span>
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a><span class="n">add_ground_truth</span><span class="p">(</span>
<a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span>
<a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a> <span class="p">[</span><span class="s1">&#39;odd&#39;</span><span class="p">,</span> <span class="s1">&#39;even&#39;</span><span class="p">],</span>
<a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a> <span class="n">tags</span><span class="o">=</span><span class="s1">&#39;my_tag&#39;</span><span class="p">,</span>
<a id="__codelineno-0-7" name="__codelineno-0-7" href="#__codelineno-0-7"></a> <span class="n">train_split_ratio</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="c1">#(1)</span>
<a id="__codelineno-0-8" name="__codelineno-0-8" href="#__codelineno-0-8"></a> <span class="n">test_split_ratio</span><span class="o">=</span><span class="mi">1</span>
<a id="__codelineno-0-9" name="__codelineno-0-9" href="#__codelineno-0-9"></a><span class="p">)</span>
</code></pre></div>
<ol>
<li>Note that the ratios don't have to add up to 1. They are just weights. There is also a <code>validation_split_ratio</code> which is 0 by default.</li>
</ol>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">query_ground_truth</span>
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a><span class="o">&gt;&gt;&gt;</span> <span class="n">query_ground_truth</span><span class="p">(</span><span class="s1">&#39;my_tag&#39;</span><span class="p">)</span>
<a id="__codelineno-1-3" name="__codelineno-1-3" href="#__codelineno-1-3"></a><span class="p">[</span><span class="n">Trace</span><span class="p">[</span><span class="nb">str</span><span class="p">]({</span><span class="s1">&#39;created&#39;</span><span class="p">:</span> <span class="s1">&#39;2022-07-12T18:36:12.825706&#39;</span><span class="p">,</span>
<a id="__codelineno-1-4" name="__codelineno-1-4" href="#__codelineno-1-4"></a> <span class="s1">&#39;exception&#39;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<a id="__codelineno-1-5" name="__codelineno-1-5" href="#__codelineno-1-5"></a> <span class="s1">&#39;feedback&#39;</span><span class="p">:</span> <span class="s1">&#39;odd&#39;</span><span class="p">,</span> <span class="c1">#(1)</span>
<a id="__codelineno-1-6" name="__codelineno-1-6" href="#__codelineno-1-6"></a> <span class="s1">&#39;logged_values&#39;</span><span class="p">:</span> <span class="p">{</span><span class="s1">&#39;input&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">},</span> <span class="c1">#(2)</span>
<a id="__codelineno-1-7" name="__codelineno-1-7" href="#__codelineno-1-7"></a> <span class="s1">&#39;models&#39;</span><span class="p">:</span> <span class="p">[],</span>
<a id="__codelineno-1-8" name="__codelineno-1-8" href="#__codelineno-1-8"></a> <span class="s1">&#39;original_execution_time_ms&#39;</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">,</span>
<a id="__codelineno-1-9" name="__codelineno-1-9" href="#__codelineno-1-9"></a> <span class="s1">&#39;output&#39;</span><span class="p">:</span> <span class="s1">&#39;odd&#39;</span><span class="p">,</span>
<a id="__codelineno-1-10" name="__codelineno-1-10" href="#__codelineno-1-10"></a> <span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;ground_truth&#39;</span><span class="p">,</span> <span class="s1">&#39;test&#39;</span><span class="p">,</span> <span class="s1">&#39;my_tag&#39;</span><span class="p">],</span> <span class="c1">#(3) </span>
<a id="__codelineno-1-11" name="__codelineno-1-11" href="#__codelineno-1-11"></a> <span class="s1">&#39;trace_id&#39;</span><span class="p">:</span> <span class="s1">&#39;4fcf2ce6-a172-469d-94b2-874577655814&#39;</span><span class="p">}),</span>
<a id="__codelineno-1-12" name="__codelineno-1-12" href="#__codelineno-1-12"></a> <span class="n">Trace</span><span class="p">[</span><span class="nb">str</span><span class="p">]({</span><span class="s1">&#39;created&#39;</span><span class="p">:</span> <span class="s1">&#39;2022-07-12T18:36:12.825706&#39;</span><span class="p">,</span>
<a id="__codelineno-1-13" name="__codelineno-1-13" href="#__codelineno-1-13"></a> <span class="s1">&#39;exception&#39;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<a id="__codelineno-1-14" name="__codelineno-1-14" href="#__codelineno-1-14"></a> <span class="s1">&#39;feedback&#39;</span><span class="p">:</span> <span class="s1">&#39;even&#39;</span><span class="p">,</span>
<a id="__codelineno-1-15" name="__codelineno-1-15" href="#__codelineno-1-15"></a> <span class="s1">&#39;logged_values&#39;</span><span class="p">:</span> <span class="p">{</span><span class="s1">&#39;input&#39;</span><span class="p">:</span> <span class="mi">2</span><span class="p">},</span>
<a id="__codelineno-1-16" name="__codelineno-1-16" href="#__codelineno-1-16"></a> <span class="s1">&#39;models&#39;</span><span class="p">:</span> <span class="p">[],</span>
<a id="__codelineno-1-17" name="__codelineno-1-17" href="#__codelineno-1-17"></a> <span class="s1">&#39;original_execution_time_ms&#39;</span><span class="p">:</span> <span class="mf">0.0</span><span class="p">,</span>
<a id="__codelineno-1-18" name="__codelineno-1-18" href="#__codelineno-1-18"></a> <span class="s1">&#39;output&#39;</span><span class="p">:</span> <span class="s1">&#39;even&#39;</span><span class="p">,</span>
<a id="__codelineno-1-19" name="__codelineno-1-19" href="#__codelineno-1-19"></a> <span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;ground_truth&#39;</span><span class="p">,</span> <span class="s1">&#39;train&#39;</span><span class="p">,</span> <span class="s1">&#39;my_tag&#39;</span><span class="p">],</span>
<a id="__codelineno-1-20" name="__codelineno-1-20" href="#__codelineno-1-20"></a> <span class="s1">&#39;trace_id&#39;</span><span class="p">:</span> <span class="s1">&#39;abee0671-beb9-4284-8c3b-c65e5836ce38&#39;</span><span class="p">})]</span>
</code></pre></div>
<ol>
<li>Expected output. This can be also accessed through the <code>.output</code> property.</li>
<li>The input value is stored here.</li>
<li>Notice how <code>ground_truth</code> always included as a tag when using <a class="autorefs autorefs-internal" href="../../reference/#great_ai.add_ground_truth">great_ai.add_ground_truth</a>. </li>
</ol>
<h2 id="get-feedback">Get feedback<a class="headerlink" href="#get-feedback" title="Permanent link">#</a></h2>
<p>After the initial data gathering, end-to-end feedback can be also integrated into the dataset. </p>
<p>The scaffolded REST API contains endpoints for managing traces and their feedbacks.</p>
<p><img alt="screenshot of swagger" loading="lazy" src="/media/feedback.png" /></p>
<p>When <a class="autorefs autorefs-internal" href="../../reference/#great_ai.query_ground_truth">great_ai.query_ground_truth</a> is executed, it implicitly filters for traces that have feedback. Therefore, both the <code>ground_truth</code> and the <code>online</code> traces that have received feedback are returned. No matter the origin of the data, it can be accessed using the same API.</p>
<h2 id="remove-clutter">Remove clutter<a class="headerlink" href="#remove-clutter" title="Permanent link">#</a></h2>
<p>Traces can be deleted either through the REST API or by calling <a class="autorefs autorefs-internal" href="../../reference/#great_ai.delete_ground_truth">great_ai.delete_ground_truth</a>. The latter provides the same interface as <a class="autorefs autorefs-internal" href="../../reference/#great_ai.query_ground_truth">great_ai.query_ground_truth</a> except it deletes the matched points.</p>
<hr>
<div class="md-source-file">
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
</small>
@ -833,7 +900,7 @@
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../use-service/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to use a GreatAI service" rel="prev">
<a href="../use-service/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to deploy a GreatAI service" rel="prev">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</div>
@ -842,20 +909,20 @@
<span class="md-footer__direction">
Previous
</span>
How to use a GreatAI service
How to deploy a GreatAI service
</div>
</div>
</a>
<a href="../large_file/" class="md-footer__link md-footer__link--next" aria-label="Next: How to use LargeFile" rel="next">
<a href="../large_file/" class="md-footer__link md-footer__link--next" aria-label="Next: How to use LargeFile-s" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
How to use LargeFile
How to use LargeFile-s
</div>
</div>
<div class="md-footer__button md-icon">

10
how-to-guides/install/index.html
View file

@ -451,7 +451,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -465,7 +465,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -479,7 +479,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -492,8 +492,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>

210
how-to-guides/large_file/index.html
View file

@ -17,7 +17,7 @@
<title>How to use LargeFile - GreatAI documentation</title>
<title>How to use LargeFile-s - GreatAI documentation</title>
@ -83,7 +83,7 @@
<div data-md-component="skip">
<a href="#how-to-use-largefile" class="md-skip">
<a href="#how-to-use-largefile-s" class="md-skip">
Skip to content
</a>
@ -116,7 +116,7 @@
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
How to use LargeFile
How to use LargeFile-s
</span>
</div>
@ -410,7 +410,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -424,7 +424,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -447,12 +447,12 @@
<label class="md-nav__link md-nav__link--active" for="__toc">
How to use LargeFile
How to use LargeFile-s
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
How to use LargeFile
How to use LargeFile-s
</a>
@ -478,19 +478,12 @@
</li>
<li class="md-nav__item">
<a href="#solution" class="md-nav__link">
Solution
</a>
<nav class="md-nav" aria-label="Solution">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#simple-example" class="md-nav__link">
Simple example
</a>
</li>
<nav class="md-nav" aria-label="Simple example">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#more-details" class="md-nav__link">
@ -512,12 +505,17 @@
</li>
<li class="md-nav__item">
<a href="#command-line-example" class="md-nav__link">
Command-line example
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#from-the-command-line" class="md-nav__link">
From the command-line
</a>
<nav class="md-nav" aria-label="Command-line example">
<nav class="md-nav" aria-label="From the command-line">
<ul class="md-nav__list">
<li class="md-nav__item">
@ -525,13 +523,6 @@
Setup
</a>
</li>
<li class="md-nav__item">
<a href="#print-the-expected-options" class="md-nav__link">
Print the expected options
</a>
</li>
<li class="md-nav__item">
@ -558,11 +549,6 @@
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul>
@ -580,8 +566,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>
@ -856,19 +842,12 @@
</li>
<li class="md-nav__item">
<a href="#solution" class="md-nav__link">
Solution
</a>
<nav class="md-nav" aria-label="Solution">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#simple-example" class="md-nav__link">
Simple example
</a>
</li>
<nav class="md-nav" aria-label="Simple example">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#more-details" class="md-nav__link">
@ -890,12 +869,17 @@
</li>
<li class="md-nav__item">
<a href="#command-line-example" class="md-nav__link">
Command-line example
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#from-the-command-line" class="md-nav__link">
From the command-line
</a>
<nav class="md-nav" aria-label="Command-line example">
<nav class="md-nav" aria-label="From the command-line">
<ul class="md-nav__list">
<li class="md-nav__item">
@ -903,13 +887,6 @@
Setup
</a>
</li>
<li class="md-nav__item">
<a href="#print-the-expected-options" class="md-nav__link">
Print the expected options
</a>
</li>
<li class="md-nav__item">
@ -936,11 +913,6 @@
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul>
@ -965,88 +937,88 @@
<h1 id="how-to-use-largefile">How to use LargeFile<a class="headerlink" href="#how-to-use-largefile" title="Permanent link">#</a></h1>
<p>Storing, versioning, and downloading files from S3 made as easy as using <code>open()</code> in Python. Caching included.</p>
<h1 id="how-to-use-largefile-s">How to use LargeFile-s<a class="headerlink" href="#how-to-use-largefile-s" title="Permanent link">#</a></h1>
<p>The functions <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">save_model</a> and <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">@use_model</a> wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use this.</p>
<h2 id="motivation">Motivation<a class="headerlink" href="#motivation" title="Permanent link">#</a></h2>
<p>Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation <a href="https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository">doesn't support the deletion</a> of large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.</p>
<h2 id="solution">Solution<a class="headerlink" href="#solution" title="Permanent link">#</a></h2>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>pip install open-large
<p>Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation <a href="https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository">doesn't support deleting</a> large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.</p>
<details class="note">
<summary>Using LargeFile-s directly (usually not needed)</summary>
<h2 id="simple-example">Simple example<a class="headerlink" href="#simple-example" title="Permanent link">#</a></h2>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileS3</span>
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>
<a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a><span class="n">LargeFileS3</span><span class="o">.</span><span class="n">configure_credentials</span><span class="p">({</span>
<a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a> <span class="s2">&quot;aws_region_name&quot;</span><span class="p">:</span> <span class="s2">&quot;your_region_like_eu-west-2&quot;</span><span class="p">,</span>
<a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a> <span class="s2">&quot;aws_access_key_id&quot;</span><span class="p">:</span> <span class="s2">&quot;YOUR_ACCESS_KEY_ID&quot;</span><span class="p">,</span>
<a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a> <span class="s2">&quot;aws_secret_access_key&quot;</span><span class="p">:</span> <span class="s2">&quot;YOUR_VERY_SECRET_ACCESS_KEY&quot;</span><span class="p">,</span>
<a id="__codelineno-0-7" name="__codelineno-0-7" href="#__codelineno-0-7"></a> <span class="s2">&quot;large_files_bucket_name&quot;</span><span class="p">:</span> <span class="s2">&quot;create_a_bucket_and_put_its_name_here&quot;</span><span class="p">,</span>
<a id="__codelineno-0-8" name="__codelineno-0-8" href="#__codelineno-0-8"></a><span class="p">})</span>
<a id="__codelineno-0-9" name="__codelineno-0-9" href="#__codelineno-0-9"></a>
<a id="__codelineno-0-10" name="__codelineno-0-10" href="#__codelineno-0-10"></a><span class="c1"># Creates a new version and deletes the older version </span>
<a id="__codelineno-0-11" name="__codelineno-0-11" href="#__codelineno-0-11"></a><span class="c1"># leaving the 3 most recently used intact</span>
<a id="__codelineno-0-12" name="__codelineno-0-12" href="#__codelineno-0-12"></a><span class="k">with</span> <span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;test.txt&quot;</span><span class="p">,</span> <span class="s2">&quot;w&quot;</span><span class="p">,</span> <span class="n">keep_last_n</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<a id="__codelineno-0-13" name="__codelineno-0-13" href="#__codelineno-0-13"></a> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100000</span><span class="p">):</span>
<a id="__codelineno-0-14" name="__codelineno-0-14" href="#__codelineno-0-14"></a> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">&#39;test</span><span class="se">\n</span><span class="s1">&#39;</span><span class="p">)</span>
<a id="__codelineno-0-15" name="__codelineno-0-15" href="#__codelineno-0-15"></a>
<a id="__codelineno-0-16" name="__codelineno-0-16" href="#__codelineno-0-16"></a><span class="c1"># By default the latest version is returned</span>
<a id="__codelineno-0-17" name="__codelineno-0-17" href="#__codelineno-0-17"></a><span class="c1"># but an optional `version` keyword argument can be provided as well</span>
<a id="__codelineno-0-18" name="__codelineno-0-18" href="#__codelineno-0-18"></a><span class="k">with</span> <span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;test.txt&quot;</span><span class="p">,</span> <span class="s2">&quot;r&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="c1">#(1)</span>
<a id="__codelineno-0-19" name="__codelineno-0-19" href="#__codelineno-0-19"></a> <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readlines</span><span class="p">()[</span><span class="mi">0</span><span class="p">])</span>
</code></pre></div>
<h3 id="simple-example">Simple example<a class="headerlink" href="#simple-example" title="Permanent link">#</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileS3</span>
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a>
<a id="__codelineno-1-3" name="__codelineno-1-3" href="#__codelineno-1-3"></a><span class="n">LargeFileS3</span><span class="o">.</span><span class="n">configure_credentials</span><span class="p">({</span>
<a id="__codelineno-1-4" name="__codelineno-1-4" href="#__codelineno-1-4"></a> <span class="s2">&quot;aws_region_name&quot;</span><span class="p">:</span> <span class="s2">&quot;your_region_like_eu-west-2&quot;</span><span class="p">,</span>
<a id="__codelineno-1-5" name="__codelineno-1-5" href="#__codelineno-1-5"></a> <span class="s2">&quot;aws_access_key_id&quot;</span><span class="p">:</span> <span class="s2">&quot;YOUR_ACCESS_KEY_ID&quot;</span><span class="p">,</span>
<a id="__codelineno-1-6" name="__codelineno-1-6" href="#__codelineno-1-6"></a> <span class="s2">&quot;aws_secret_access_key&quot;</span><span class="p">:</span> <span class="s2">&quot;YOUR_VERY_SECRET_ACCESS_KEY&quot;</span><span class="p">,</span>
<a id="__codelineno-1-7" name="__codelineno-1-7" href="#__codelineno-1-7"></a> <span class="s2">&quot;large_files_bucket_name&quot;</span><span class="p">:</span> <span class="s2">&quot;create_a_bucket_and_put_its_name_here&quot;</span><span class="p">,</span>
<a id="__codelineno-1-8" name="__codelineno-1-8" href="#__codelineno-1-8"></a><span class="p">})</span>
<a id="__codelineno-1-9" name="__codelineno-1-9" href="#__codelineno-1-9"></a>
<a id="__codelineno-1-10" name="__codelineno-1-10" href="#__codelineno-1-10"></a><span class="c1"># Creates a new version and deletes the older version leaving the 3 most recently used intact</span>
<a id="__codelineno-1-11" name="__codelineno-1-11" href="#__codelineno-1-11"></a><span class="k">with</span> <span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;test.txt&quot;</span><span class="p">,</span> <span class="s2">&quot;w&quot;</span><span class="p">,</span> <span class="n">keep_last_n</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<a id="__codelineno-1-12" name="__codelineno-1-12" href="#__codelineno-1-12"></a> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100000</span><span class="p">):</span>
<a id="__codelineno-1-13" name="__codelineno-1-13" href="#__codelineno-1-13"></a> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">&#39;test</span><span class="se">\n</span><span class="s1">&#39;</span><span class="p">)</span>
<a id="__codelineno-1-14" name="__codelineno-1-14" href="#__codelineno-1-14"></a>
<a id="__codelineno-1-15" name="__codelineno-1-15" href="#__codelineno-1-15"></a><span class="c1"># By default the latest version is returned</span>
<a id="__codelineno-1-16" name="__codelineno-1-16" href="#__codelineno-1-16"></a><span class="c1"># but an optional `version` keyword argument can be provided as well</span>
<a id="__codelineno-1-17" name="__codelineno-1-17" href="#__codelineno-1-17"></a><span class="k">with</span> <span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;test.txt&quot;</span><span class="p">,</span> <span class="s2">&quot;r&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<a id="__codelineno-1-18" name="__codelineno-1-18" href="#__codelineno-1-18"></a> <span class="nb">print</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">readlines</span><span class="p">()[</span><span class="mi">0</span><span class="p">])</span>
</code></pre></div>
<blockquote>
<p>Automatically creates a file, writes to it, uploads it to S3, and then queries the most recent version of it.
In this case, the latest version is already in the local cache, no download is required.</p>
</blockquote>
<ol>
<li>In this case, the latest version is already in the local cache, no download is required.</li>
</ol>
<h3 id="more-details">More details<a class="headerlink" href="#more-details" title="Permanent link">#</a></h3>
<p><code>LargeFile</code> behaves like an opened file (in the background it is a temp file after all). Binary reading and writing is supported along with the <a href="https://docs.python.org/3/library/functions.html#open">different keywords</a> <code>open()</code> accepts.</p>
<p><code>LargeFile</code> behaves like an opened file (in the background it is a temp file after all). Binary reads and writes are supported along with the <a href="https://docs.python.org/3/library/functions.html#open">different keywords <code>open()</code> accepts</a>.</p>
<p>The local cache can be configured with these properties:</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="n">LargeFile</span><span class="o">.</span><span class="n">cache_path</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s1">&#39;.cache&#39;</span><span class="p">)</span>
<a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a><span class="n">LargeFile</span><span class="o">.</span><span class="n">max_cache_size</span> <span class="o">=</span> <span class="s2">&quot;30 GB&quot;</span>
<div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a><span class="n">LargeFileS3</span><span class="o">.</span><span class="n">cache_path</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s1">&#39;.cache&#39;</span><span class="p">)</span>
<a id="__codelineno-1-2" name="__codelineno-1-2" href="#__codelineno-1-2"></a><span class="n">LargeFileS3</span><span class="o">.</span><span class="n">max_cache_size</span> <span class="o">=</span> <span class="s2">&quot;30 GB&quot;</span>
</code></pre></div>
<h4 id="i-only-need-a-path">I only need a path<a class="headerlink" href="#i-only-need-a-path" title="Permanent link">#</a></h4>
<p>In case you only need a path to the "remote" file, this pattern can be applied:</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="n">path_to_model</span> <span class="o">=</span> <span class="n">LargeFile</span><span class="p">(</span><span class="s2">&quot;folder-of-my-bert-model&quot;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="mi">31</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a><span class="n">path_to_model</span> <span class="o">=</span> <span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;folder-of-my-bert-model&quot;</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="mi">31</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
</code></pre></div>
<blockquote>
<p>This will first download the file/folder into your local cache folder. Then, it returns a <code>Path</code> object to the local version. Which can be turned into a string with <code>str(path_to_model)</code>.</p>
</blockquote>
<p>The same approach works for uploads:</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="n">LargeFile</span><span class="p">(</span><span class="s2">&quot;folder-of-my-bert-model&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">&#39;path_to_local/folder_or_file&#39;</span><span class="p">)</span>
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;folder-of-my-bert-model&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">&#39;path_to_local/folder_or_file&#39;</span><span class="p">)</span>
</code></pre></div>
<blockquote>
<p>This way, both regular files and folders can be handled. The uploaded file is called <strong>folder-of-my-bert-model</strong>, the local name is ignored.</p>
</blockquote>
<p>Lastly, all version of the remote object can be deleted by calling <code>LargeFile("my-file").delete()</code>. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.</p>
<h3 id="command-line-example">Command-line example<a class="headerlink" href="#command-line-example" title="Permanent link">#</a></h3>
<p>The package can be used as a module from the command-line to give you more flexibility.</p>
<h4 id="setup">Setup<a class="headerlink" href="#setup" title="Permanent link">#</a></h4>
<p>Lastly, all version of the remote object can be deleted by calling <code>LargeFileS3("my-file").delete()</code>. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.</p>
</details>
<h2 id="from-the-command-line">From the command-line<a class="headerlink" href="#from-the-command-line" title="Permanent link">#</a></h2>
<p>The main reason for using the <code>large-file</code> or <code>python3 -m great_ai.large_file</code> commands is to upload or download models from the terminal. For example, when building a docker image, it is best-practice to cache the referred models.</p>
<h3 id="setup">Setup<a class="headerlink" href="#setup" title="Permanent link">#</a></h3>
<p>Create an .ini file (or use <em>~/.aws/credentials</em>). It may look like this:</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a><span class="na">aws_region_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">your_region_like_eu-west-2</span><span class="w"></span>
<a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a><span class="na">aws_access_key_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">YOUR_ACCESS_KEY_ID</span><span class="w"></span>
<a id="__codelineno-5-3" name="__codelineno-5-3" href="#__codelineno-5-3"></a><span class="na">aws_secret_access_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">YOUR_VERY_SECRET_ACCESS_KEY</span><span class="w"></span>
<a id="__codelineno-5-4" name="__codelineno-5-4" href="#__codelineno-5-4"></a><span class="na">large_files_bucket_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">my_large_files</span><span class="w"></span>
<a id="__codelineno-5-5" name="__codelineno-5-5" href="#__codelineno-5-5"></a><span class="na">endpoint_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">this is optional, for backblaze, use this: https://s3.us-west-002.backblazeb2.com</span><span class="w"></span>
<div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="na">aws_region_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">your_region_like_eu-west-2</span><span class="w"></span>
<a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a><span class="na">aws_access_key_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">YOUR_ACCESS_KEY_ID</span><span class="w"></span>
<a id="__codelineno-4-3" name="__codelineno-4-3" href="#__codelineno-4-3"></a><span class="na">aws_secret_access_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">YOUR_VERY_SECRET_ACCESS_KEY</span><span class="w"></span>
<a id="__codelineno-4-4" name="__codelineno-4-4" href="#__codelineno-4-4"></a><span class="na">large_files_bucket_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">my_large_files</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>Just like in <a href="example_secrets.ini">example secrets</a>.</p>
</blockquote>
<h4 id="print-the-expected-options">Print the expected options<a class="headerlink" href="#print-the-expected-options" title="Permanent link">#</a></h4>
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a>python3 -m large_file --help
</code></pre></div>
<h4 id="upload-some-files">Upload some files<a class="headerlink" href="#upload-some-files" title="Permanent link">#</a></h4>
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a>python3 -m large_file --backend s3 --secrets secrets.ini --push my_first_file.json folder/my_second_file my_folder
<h3 id="upload-some-files">Upload some files<a class="headerlink" href="#upload-some-files" title="Permanent link">#</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a>large-file --backend s3 --secrets secrets.ini <span class="se">\</span>
<a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a> --push my_first_file.json folder/my_second_file my_folder
</code></pre></div>
<blockquote>
<p>Only the filename is used as the S3 name, the rest of the path is ignored.</p>
</blockquote>
<h4 id="download-some-files-to-the-local-cache">Download some files to the local cache<a class="headerlink" href="#download-some-files-to-the-local-cache" title="Permanent link">#</a></h4>
<div class="admonition important">
<p class="admonition-title">Using MongoDB</p>
<p>The possible values for <code>--backend</code> are <code>s3</code>, <code>mongo</code>, and <code>local</code>. The latter doesn't need credentials, it only versions and stores your files in a local folder. MongoDB on the other hand requires a <code>mongo_connection_string</code> and a <code>mongo_database</code> to be specified. For storing large files, it uses the GridFS specification.</p>
</div>
<h3 id="download-some-files-to-the-local-cache">Download some files to the local cache<a class="headerlink" href="#download-some-files-to-the-local-cache" title="Permanent link">#</a></h3>
<p>This can be useful when building a Docker image for example. This way, the files can already reside inside the container and need not be downloaded later.</p>
<div class="highlight"><pre><span></span><code><a id="__codelineno-8-1" name="__codelineno-8-1" href="#__codelineno-8-1"></a>python3 -m large_file --backend s3 -secrets ~/.aws/credentials --cache my_first_file.json:3 my_second_file my_folder:0
<div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a>large-file --backend s3 --secrets ~/.aws/credentials <span class="se">\</span>
<a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a> --cache my_first_file.json:3 my_second_file my_folder:0
</code></pre></div>
<blockquote>
<p>Versions may be specified by using <code>:</code>-s.</p>
</blockquote>
<h4 id="delete-remote-files">Delete remote files<a class="headerlink" href="#delete-remote-files" title="Permanent link">#</a></h4>
<div class="highlight"><pre><span></span><code><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a>python3 -m large_file --backend s3 --secrets ~/.aws/credentials --delete my_first_file.json
<h3 id="delete-remote-files">Delete remote files<a class="headerlink" href="#delete-remote-files" title="Permanent link">#</a></h3>
<div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a>large-file --backend s3 --secrets ~/.aws/credentials <span class="se">\</span>
<a id="__codelineno-7-2" name="__codelineno-7-2" href="#__codelineno-7-2"></a> --delete my_first_file.json
</code></pre></div>
<hr>
@ -1054,7 +1026,7 @@ In this case, the latest version is already in the local cache, no download is r
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 11, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
</small>
@ -1075,7 +1047,7 @@ In this case, the latest version is already in the local cache, no download is r
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../handle-training-data/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to handle training data" rel="prev">
<a href="../handle-training-data/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to manage training data" rel="prev">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</div>
@ -1084,20 +1056,20 @@ In this case, the latest version is already in the local cache, no download is r
<span class="md-footer__direction">
Previous
</span>
How to handle training data
How to manage training data
</div>
</div>
</a>
<a href="../scraping/" class="md-footer__link md-footer__link--next" aria-label="Next: Scraping" rel="next">
<a href="../call-remote/" class="md-footer__link md-footer__link--next" aria-label="Next: Call remote GreatAI instances" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
Scraping
Call remote GreatAI instances
</div>
</div>
<div class="md-footer__button md-icon">

125
how-to-guides/large_file/large_file.md
View file

@ -1,118 +1,111 @@
# How to use LargeFile
# How to use LargeFile-s
Storing, versioning, and downloading files from S3 made as easy as using `open()` in Python. Caching included.
The functions [save_model][great_ai.use_model] and [@use_model][great_ai.use_model] wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use this.
## Motivation
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support the deletion](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) of large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support deleting](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
## Solution
```
pip install open-large
```
??? note "Using LargeFile-s directly (usually not needed)"
## Simple example
### Simple example
```python
from great_ai.large_file import LargeFileS3
```python
from great_ai.large_file import LargeFileS3
LargeFileS3.configure_credentials({
"aws_region_name": "your_region_like_eu-west-2",
"aws_access_key_id": "YOUR_ACCESS_KEY_ID",
"aws_secret_access_key": "YOUR_VERY_SECRET_ACCESS_KEY",
"large_files_bucket_name": "create_a_bucket_and_put_its_name_here",
})
LargeFileS3.configure_credentials({
"aws_region_name": "your_region_like_eu-west-2",
"aws_access_key_id": "YOUR_ACCESS_KEY_ID",
"aws_secret_access_key": "YOUR_VERY_SECRET_ACCESS_KEY",
"large_files_bucket_name": "create_a_bucket_and_put_its_name_here",
})
# Creates a new version and deletes the older version
# leaving the 3 most recently used intact
with LargeFileS3("test.txt", "w", keep_last_n=3) as f:
for i in range(100000):
f.write('test\n')
# Creates a new version and deletes the older version leaving the 3 most recently used intact
with LargeFileS3("test.txt", "w", keep_last_n=3) as f:
for i in range(100000):
f.write('test\n')
# By default the latest version is returned
# but an optional `version` keyword argument can be provided as well
with LargeFileS3("test.txt", "r") as f: #(1)
print(f.readlines()[0])
```
# By default the latest version is returned
# but an optional `version` keyword argument can be provided as well
with LargeFileS3("test.txt", "r") as f:
print(f.readlines()[0])
```
1. In this case, the latest version is already in the local cache, no download is required.
> Automatically creates a file, writes to it, uploads it to S3, and then queries the most recent version of it.
> In this case, the latest version is already in the local cache, no download is required.
### More details
### More details
`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reads and writes are supported along with the [different keywords `open()` accepts](https://docs.python.org/3/library/functions.html#open).
`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reading and writing is supported along with the [different keywords](https://docs.python.org/3/library/functions.html#open) `open()` accepts.
The local cache can be configured with these properties:
The local cache can be configured with these properties:
```python
LargeFileS3.cache_path = Path('.cache')
LargeFileS3.max_cache_size = "30 GB"
```
```python
LargeFile.cache_path = Path('.cache')
LargeFile.max_cache_size = "30 GB"
```
#### I only need a path
#### I only need a path
In case you only need a path to the "remote" file, this pattern can be applied:
In case you only need a path to the "remote" file, this pattern can be applied:
```python
path_to_model = LargeFileS3("folder-of-my-bert-model", version=31).get()
```
```python
path_to_model = LargeFile("folder-of-my-bert-model", version=31).get()
```
> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
The same approach works for uploads:
The same approach works for uploads:
```python
LargeFileS3("folder-of-my-bert-model").push('path_to_local/folder_or_file')
```
```python
LargeFile("folder-of-my-bert-model").push('path_to_local/folder_or_file')
```
> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
Lastly, all version of the remote object can be deleted by calling `LargeFileS3("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
Lastly, all version of the remote object can be deleted by calling `LargeFile("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
## From the command-line
### Command-line example
The main reason for using the `large-file` or `python3 -m great_ai.large_file` commands is to upload or download models from the terminal. For example, when building a docker image, it is best-practice to cache the referred models.
The package can be used as a module from the command-line to give you more flexibility.
### Setup
#### Setup
Create an .ini file (or use _~/.aws/credentials_). It may look like this:
Create an .ini file (or use *~/.aws/credentials*). It may look like this:
```ini
aws_region_name = your_region_like_eu-west-2
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_VERY_SECRET_ACCESS_KEY
large_files_bucket_name = my_large_files
endpoint_url = this is optional, for backblaze, use this: https://s3.us-west-002.backblazeb2.com
```
> Just like in [example secrets](example_secrets.ini).
#### Print the expected options
### Upload some files
```sh
python3 -m large_file --help
```
#### Upload some files
```sh
python3 -m large_file --backend s3 --secrets secrets.ini --push my_first_file.json folder/my_second_file my_folder
large-file --backend s3 --secrets secrets.ini \
--push my_first_file.json folder/my_second_file my_folder
```
> Only the filename is used as the S3 name, the rest of the path is ignored.
#### Download some files to the local cache
!!! important "Using MongoDB"
The possible values for `--backend` are `s3`, `mongo`, and `local`. The latter doesn't need credentials, it only versions and stores your files in a local folder. MongoDB on the other hand requires a `mongo_connection_string` and a `mongo_database` to be specified. For storing large files, it uses the GridFS specification.
### Download some files to the local cache
This can be useful when building a Docker image for example. This way, the files can already reside inside the container and need not be downloaded later.
```sh
python3 -m large_file --backend s3 -secrets ~/.aws/credentials --cache my_first_file.json:3 my_second_file my_folder:0
large-file --backend s3 --secrets ~/.aws/credentials \
--cache my_first_file.json:3 my_second_file my_folder:0
```
> Versions may be specified by using `:`-s.
#### Delete remote files
### Delete remote files
```sh
python3 -m large_file --backend s3 --secrets ~/.aws/credentials --delete my_first_file.json
large-file --backend s3 --secrets ~/.aws/credentials \
--delete my_first_file.json
```

14
how-to-guides/scraping/index.html
View file

@ -405,7 +405,7 @@
<li class="md-nav__item">
<a href="../use-service/" class="md-nav__link">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
</li>
@ -419,7 +419,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -433,7 +433,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -446,8 +446,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>
@ -1955,7 +1955,7 @@ greeter("Bob")</div>
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../large_file/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How to use LargeFile" rel="prev">
<a href="../call-remote/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Call remote GreatAI instances" rel="prev">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</div>
@ -1964,7 +1964,7 @@ greeter("Bob")</div>
<span class="md-footer__direction">
Previous
</span>
How to use LargeFile
Call remote GreatAI instances
</div>
</div>
</a>

24
how-to-guides/use-service/index.html
View file

@ -17,7 +17,7 @@
<title>How to use a GreatAI service - GreatAI documentation</title>
<title>How to deploy a GreatAI service - GreatAI documentation</title>
@ -83,7 +83,7 @@
<div data-md-component="skip">
<a href="#how-to-use-a-greatai-service" class="md-skip">
<a href="#how-to-deploy-a-greatai-service" class="md-skip">
Skip to content
</a>
@ -116,7 +116,7 @@
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
How to use a GreatAI service
How to deploy a GreatAI service
</span>
</div>
@ -419,12 +419,12 @@
<label class="md-nav__link md-nav__link--active" for="__toc">
How to use a GreatAI service
How to deploy a GreatAI service
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
How to use a GreatAI service
How to deploy a GreatAI service
</a>
@ -526,7 +526,7 @@
<li class="md-nav__item">
<a href="../handle-training-data/" class="md-nav__link">
How to handle training data
How to manage training data
</a>
</li>
@ -540,7 +540,7 @@
<li class="md-nav__item">
<a href="../large_file/" class="md-nav__link">
How to use LargeFile
How to use LargeFile-s
</a>
</li>
@ -553,8 +553,8 @@
<li class="md-nav__item">
<a href="../call_remote.md" class="md-nav__link">
None
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
</a>
</li>
@ -911,7 +911,7 @@
<h1 id="how-to-use-a-greatai-service">How to use a GreatAI service<a class="headerlink" href="#how-to-use-a-greatai-service" title="Permanent link">#</a></h1>
<h1 id="how-to-deploy-a-greatai-service">How to deploy a GreatAI service<a class="headerlink" href="#how-to-deploy-a-greatai-service" title="Permanent link">#</a></h1>
<p>After <a href="/how-to-guides/create-service">creating a GreatAI service</a> by wrapping your prediction function, and optionally <a href="/how-to-guides/configure-service">configuring it</a>, it's time to do some prediction.</p>
<p>Let's take the following example:</p>
<div class="highlight"><span class="filename">greeter.py</span><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai</span> <span class="kn">import</span> <span class="n">GreatAI</span>
@ -1054,13 +1054,13 @@
<a href="../handle-training-data/" class="md-footer__link md-footer__link--next" aria-label="Next: How to handle training data" rel="next">
<a href="../handle-training-data/" class="md-footer__link md-footer__link--next" aria-label="Next: How to manage training data" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
How to handle training data
How to manage training data
</div>
</div>
<div class="md-footer__button md-icon">

2
how-to-guides/use-service/use-service.md
View file

@ -1,4 +1,4 @@
# How to use a GreatAI service
# How to deploy a GreatAI service
After [creating a GreatAI service](/how-to-guides/create-service) by wrapping your prediction function, and optionally [configuring it](/how-to-guides/configure-service), it's time to do some prediction.