Add more documentation
This commit is contained in:
parent
7165174f4f
commit
8d1ad9a242
21 changed files with 406 additions and 92 deletions
|
|
@ -1,4 +1,4 @@
|
|||
# Reference
|
||||
# GreatAI reference
|
||||
|
||||
```python
|
||||
from great_ai import *
|
||||
|
|
@ -54,3 +54,17 @@ from great_ai import *
|
|||
::: great_ai.delete_ground_truth
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
## Tracing databases
|
||||
|
||||
::: great_ai.TracingDatabaseDriver
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.MongoDbDriver
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.ParallelTinyDbDriver
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
|
|
|||
|
|
@ -3,3 +3,19 @@
|
|||
```python
|
||||
from great_ai.large_file import *
|
||||
```
|
||||
|
||||
::: great_ai.large_file.LargeFileLocal
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.large_file.LargeFileS3
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.large_file.LargeFileMongo
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.large_file.LargeFileBase
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
|
|
|||
|
|
@ -10,22 +10,26 @@ Well-tested tools that can be used in production with confidence. The toolbox of
|
|||
|
||||
::: great_ai.utilities.clean
|
||||
::: great_ai.utilities.get_sentences
|
||||
::: great_ai.utilities.language.predict_language
|
||||
::: great_ai.utilities.language.is_english
|
||||
::: great_ai.utilities.language.english_name_of_language
|
||||
::: great_ai.utilities.evaluate_ranking.evaluate_ranking
|
||||
::: great_ai.utilities.predict_language
|
||||
::: great_ai.utilities.is_english
|
||||
::: great_ai.utilities.english_name_of_language
|
||||
::: great_ai.utilities.evaluate_ranking
|
||||
|
||||
## Parallel processing
|
||||
|
||||
Multiprocessing and multithreading-based parallelism with support for `async` functions. Its main purpose is to implement [great_ai.GreatAI.process_batch][], however, the parallel processing functions are also convenient for covering other types of mapping needs with a friendlier API than [joblib](https://joblib.readthedocs.io/en/latest/parallel.html){ target=_blank } or [multiprocess](https://pypi.org/project/multiprocess/){ target=_blank }.
|
||||
|
||||
::: great_ai.utilities.parallel_map.simple_parallel_map
|
||||
::: great_ai.utilities.simple_parallel_map
|
||||
options:
|
||||
show_root_heading: true
|
||||
::: great_ai.utilities.parallel_map.parallel_map
|
||||
::: great_ai.utilities.parallel_map.threaded_parallel_map
|
||||
::: great_ai.utilities.threaded_parallel_map
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
## Composable parallel processing
|
||||
|
||||
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
|
||||
Because both [threaded_parallel_map][great_ai.utilities.parallel_map.threaded_parallel_map.threaded_parallel_map] and [parallel_map][great_ai.utilities.parallel_map.parallel_map] have a streaming interface, it is easy to compose them and end up with, for example, a process for each CPU core with its own thread-pool or event-loop. Longer pipelines are also easy to imagine. The chunking methods help in these compositions.
|
||||
|
||||
For more info, check-out [the scraping guide](/how-to-guides/scraping).
|
||||
|
||||
|
|
@ -34,5 +38,10 @@ For more info, check-out [the scraping guide](/how-to-guides/scraping).
|
|||
|
||||
## Operations
|
||||
|
||||
::: great_ai.utilities.config_file.config_file
|
||||
::: great_ai.utilities.logger.get_logger
|
||||
::: great_ai.utilities.ConfigFile
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
||||
::: great_ai.utilities.get_logger
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
# View models
|
||||
|
||||
::: great_ai.views.trace.Trace
|
||||
::: great_ai.Trace
|
||||
options:
|
||||
show_root_heading: true
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue