andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Deployed 8d1ad9a with MkDocs version: 1.3.0

Browse source
This commit is contained in:
2022-07-11 17:21:45 +00:00
parent 1a6a68ffcc
commit 25a66fb86b
46 changed files with 10768 additions and 1924 deletions
Download patch file Download diff file Expand all files Collapse all files

843
reference/utilities/index.html
View file

File diff suppressed because it is too large Load diff

27
reference/utilities/utilities.md
View file

@ -10,22 +10,26 @@ Well-tested tools that can be used in production with confidence. The toolbox of
::: great_ai.utilities.clean
::: great_ai.utilities.get_sentences
::: great_ai.utilities.language.predict_language
::: great_ai.utilities.language.is_english
::: great_ai.utilities.language.english_name_of_language
::: great_ai.utilities.evaluate_ranking.evaluate_ranking
::: great_ai.utilities.predict_language
::: great_ai.utilities.is_english
::: great_ai.utilities.english_name_of_language
::: great_ai.utilities.evaluate_ranking
## Parallel processing
Multiprocessing and multithreading-based parallelism with support for `async` functions. Its main purpose is to implement [great_ai.GreatAI.process_batch][], however, the parallel processing functions are also convenient for covering other types of mapping needs with a friendlier API than [joblib](https://joblib.readthedocs.io/en/latest/parallel.html){ target=_blank } or [multiprocess](https://pypi.org/project/multiprocess/){ target=_blank }.
::: great_ai.utilities.parallel_map.simple_parallel_map
::: great_ai.utilities.simple_parallel_map
options:
show_root_heading: true
::: great_ai.utilities.parallel_map.parallel_map
::: great_ai.utilities.parallel_map.threaded_parallel_map
::: great_ai.utilities.threaded_parallel_map
options:
show_root_heading: true
## Composable parallel processing
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
For more info, check-out [the scraping guide](/how-to-guides/scraping).
@ -34,5 +38,10 @@ For more info, check-out [the scraping guide](/how-to-guides/scraping).
## Operations
::: great_ai.utilities.config_file.config_file
::: great_ai.utilities.logger.get_logger
::: great_ai.utilities.ConfigFile
options:
show_root_heading: true
::: great_ai.utilities.get_logger
options:
show_root_heading: true