andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 03765e1 with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-13 10:53:35 +00:00
parent 8cda980b81
commit f5267b5a06
31 changed files with 725 additions and 91 deletions
Download patch file Download diff file Expand all files Collapse all files

14
how-to-guides/large_file/index.html
View file

@ -567,7 +567,7 @@
<li class="md-nav__item">
<a href="../call-remote/" class="md-nav__link">
Call remote GreatAI instances
How to call remote GreatAI instances
</a>
</li>
@ -938,11 +938,13 @@
<h1 id="how-to-use-largefile-s">How to use LargeFile-s<a class="headerlink" href="#how-to-use-largefile-s" title="Permanent link">#</a></h1>
<p>The functions <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">save_model</a> and <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">@use_model</a> wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use this.</p>
<p>The functions <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">save_model</a> and <a class="autorefs autorefs-internal" href="../../reference/#great_ai.use_model">@use_model</a> wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use LargeFile-s directly.</p>
<h2 id="motivation">Motivation<a class="headerlink" href="#motivation" title="Permanent link">#</a></h2>
<p>Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation <a href="https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository">doesn't support deleting</a> large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.</p>
<p><a href="https://dvc.org/">DVC</a> is a viable alternative, however, it requires users to learn to use one more CLI tool.</p>
<details class="note">
<summary>Using LargeFile-s directly (usually not needed)</summary>
<p>LargeFile doesn't require users to learn too much new. It is a nearly exact copy of the built-in <code>open()</code> function of Python with which users are certainly already familiar.</p>
<h2 id="simple-example">Simple example<a class="headerlink" href="#simple-example" title="Permanent link">#</a></h2>
<div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a><span class="kn">from</span> <span class="nn">great_ai.large_file</span> <span class="kn">import</span> <span class="n">LargeFileS3</span>
<a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>
@ -984,7 +986,7 @@
<div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="n">LargeFileS3</span><span class="p">(</span><span class="s2">&quot;folder-of-my-bert-model&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">&#39;path_to_local/folder_or_file&#39;</span><span class="p">)</span>
</code></pre></div>
<blockquote>
<p>This way, both regular files and folders can be handled. The uploaded file is called <strong>folder-of-my-bert-model</strong>, the local name is ignored.</p>
<p>This way both regular files and folders can be handled. The uploaded file is called <strong>folder-of-my-bert-model</strong>, the local name is ignored.</p>
</blockquote>
<p>Lastly, all version of the remote object can be deleted by calling <code>LargeFileS3("my-file").delete()</code>. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.</p>
</details>
@ -1026,7 +1028,7 @@
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 12, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
</small>
@ -1063,13 +1065,13 @@
<a href="../call-remote/" class="md-footer__link md-footer__link--next" aria-label="Next: Call remote GreatAI instances" rel="next">
<a href="../call-remote/" class="md-footer__link md-footer__link--next" aria-label="Next: How to call remote GreatAI instances" rel="next">
<div class="md-footer__title">
<div class="md-ellipsis">
<span class="md-footer__direction">
Next
</span>
Call remote GreatAI instances
How to call remote GreatAI instances
</div>
</div>
<div class="md-footer__button md-icon">