47 lines
1.9 KiB
Markdown
47 lines
1.9 KiB
Markdown
# Utilities
|
|
|
|
```python
|
|
from great_ai.utilities import *
|
|
```
|
|
|
|
## NLP tools
|
|
|
|
Well-tested tools that can be used in production with confidence. The toolbox of feature-extraction functions is expected to grow to cover other domains as well.
|
|
|
|
::: great_ai.utilities.clean
|
|
::: great_ai.utilities.get_sentences
|
|
::: great_ai.utilities.language.predict_language
|
|
::: great_ai.utilities.language.english_name_of_language
|
|
::: great_ai.utilities.language.is_english
|
|
::: great_ai.utilities.evaluate_ranking.evaluate_ranking
|
|
|
|
## Parallel processing
|
|
|
|
Multiprocessing and multithreading-based parallelism with support for `async` functions. Its main purpose is to implement [great_ai.GreatAI.process_batch][], however, the parallel processing functions are also convenient for covering other types of mapping needs with a friendlier API than [joblib](https://joblib.readthedocs.io/en/latest/parallel.html){ target=_blank } or [multiprocess](https://pypi.org/project/multiprocess/){ target=_blank }.
|
|
|
|
::: great_ai.utilities.simple_parallel_map
|
|
options:
|
|
show_root_heading: true
|
|
::: great_ai.utilities.parallel_map.parallel_map
|
|
::: great_ai.utilities.threaded_parallel_map
|
|
options:
|
|
show_root_heading: true
|
|
|
|
## Composable parallel processing
|
|
|
|
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
|
|
|
|
For more info, check-out [the scraping guide](/how-to-guides/scraping).
|
|
|
|
::: great_ai.utilities.chunk
|
|
::: great_ai.utilities.unchunk
|
|
|
|
## Operations
|
|
|
|
::: great_ai.utilities.ConfigFile
|
|
options:
|
|
show_root_heading: true
|
|
|
|
::: great_ai.utilities.get_logger
|
|
options:
|
|
show_root_heading: true
|