andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 9999b7e with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-15 18:04:53 +00:00
parent 087e049514
commit 7d09422380
37 changed files with 116 additions and 108 deletions
Download patch file Download diff file Expand all files Collapse all files

5
reference/utilities/index.html
View file

@ -436,7 +436,7 @@
<li class="md-nav__item">
<a href="../../how-to-guides/large_file/" class="md-nav__link">
How to use LargeFile-s
How to use LargeFiles
</a>
</li>
@ -3471,7 +3471,6 @@ thread and <code>ignore_exceptions=False</code>.</p></td>
</div><h2 id="composable-parallel-processing">Composable parallel processing<a class="headerlink" href="#composable-parallel-processing" title="Permanent link">#</a></h2>
<p>Because both <a class="autorefs autorefs-internal" href="#great_ai.utilities.threaded_parallel_map">threaded_parallel_map</a> and <a class="autorefs autorefs-internal" href="#great_ai.utilities.parallel_map.parallel_map.parallel_map">parallel_map</a> have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.</p>
<p>For more info, check-out <a href="/how-to-guides/scraping">the scraping guide</a>.</p>
<div class="doc doc-object doc-module">
@ -4424,7 +4423,7 @@ the system and <code>ignore_missing=False</code>.</p></td>
<small>
Last update:
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 13, 2022</span>
<span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 15, 2022</span>
</small>

2
reference/utilities/utilities.md
View file

@ -31,8 +31,6 @@ Multiprocessing and multithreading-based parallelism with support for `async` fu
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
For more info, check-out [the scraping guide](/how-to-guides/scraping).
::: great_ai.utilities.chunk
::: great_ai.utilities.unchunk