andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

Improve docs

Browse source
This commit is contained in:
Andras Schmelczer 2022-07-12 21:50:42 +02:00
parent 4e07161a16
commit 0585a4e4f9
No known key found for this signature in database
GPG key ID: 0EA1BC97D0AB076E
10 changed files with 143 additions and 81 deletions
Download patch file Download diff file Expand all files Collapse all files

19
docs/examples/simple/data.ipynb
View file

@ -51,7 +51,8 @@
"\n",
"manifest = (\n",
" urllib.request.urlopen(\n",
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt\"\n",
" \"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/\"\n",
" \"open-corpus/2022-02-01/manifest.txt\"\n",
" )\n",
" .read()\n",
" .decode()\n",
@ -61,7 +62,9 @@
"shuffle(lines)\n",
"chunks = lines[:MAX_CHUNK_COUNT]\n",
"\n",
"f\"Processing {len(chunks)} out of the {len(manifest.split())} available chunks\""
"f\"\"\"Processing {len(chunks)} out of the {\n",
" len(manifest.split())\n",
"} available chunks\"\"\""
]
},
{
@ -104,7 +107,8 @@
"def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:\n",
" # Extract\n",
" response = urllib.request.urlopen(\n",
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}\"\n",
" f\"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/\"\n",
" \"open-corpus/2022-02-01/{chunk_key}\"\n",
" ) # a gzipped JSON Lines file\n",
"\n",
" decompressed = gzip.decompress(response.read())\n",
@ -115,13 +119,14 @@
" return [\n",
" (\n",
" clean(\n",
" f'{c[\"title\"]} {c[\"paperAbstract\"]} {c[\"journalName\"]} {c[\"venue\"]}',\n",
" f'{c[\"title\"]} {c[\"paperAbstract\"]} '\n",
" f'{c[\"journalName\"]} {c[\"venue\"]}',\n",
" convert_to_ascii=True,\n",
" ), # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts\n",
" ), # The text is cleaned to remove common artifacts\n",
" c[\"fieldsOfStudy\"],\n",
" ) # Create pairs of `(text, [...domains])`\n",
" for c in chunk\n",
" if c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"]))\n",
" if (c[\"fieldsOfStudy\"] and is_english(predict_language(c[\"paperAbstract\"])))\n",
" ]\n",
"\n",
"\n",
@ -145,7 +150,7 @@
"source": [
"### Load\n",
"\n",
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.\n",
"Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each data-point with a split label according to the ratios we set. Additional tags can be also given.\n",
"\n",
"#### Production-ready backend\n",
"\n",

1
docs/examples/simple/deploy.ipynb
View file

@ -202,7 +202,6 @@
"\n",
" # Configure matplotlib to have nice, high-resolution charts\n",
" %matplotlib inline\n",
"\n",
" plt.rcParams[\"figure.figsize\"] = (30, 15)\n",
" plt.rcParams[\"figure.facecolor\"] = \"white\"\n",
" plt.rcParams[\"font.size\"] = 12\n",

6
docs/explanation.md
View file

@ -1,7 +1,9 @@
# Explanation
A lot more details and discussion about the problem context and approaches of GreatAI can be found in my thesis.
A lot more details and discussion about the problem context and approaches of GreatAI along with its evaluation can be found in my thesis.
![front page](/media/thesis-frontpage.png)
<div style="display: flex; justify-content: space-evenly;" markdown>
[::fontawesome-solid-graduation-cap: Download](thesis/main.pdf){ .md-button .md-button--primary }
[::fontawesome-solid-graduation-cap: Download (available soon)](thesis/main.pdf){ .md-button .md-button--primary }
</div>

67
docs/how-to-guides/handle-training-data.md
View file

@ -1,6 +1,69 @@
# How to handle training data
# How to manage training data
In order to simplify your training data management, `great-ai` provide two complementing approaches for inputting new data-points.
## Upload data
## Use feedback
At the start of your experiments' first iteration, after you've gathered suitable samples for training, you can call [great_ai.add_ground_truth][]. This automatically stores a timestamp and also allows you to assign tags to the data. Using these attributes, [great_ai.query_ground_truth][] can be called to get a filtered view of the training data.
!!! important "Train-test-validation splits"
It is a best-practice to lock-away a test split of your data that is only used for the final quality assessment. This prevents you from accidentally training on it, or inadvertently tuning the model to have the highest accuracy metrics on the test split. This, of course, may lead to dubious results, hence, care must be taken to avoid it.
With [great_ai.add_ground_truth][], there is an option to tag the samples with `train`, `test`, and `validation` randomly, following a predefined distribution. This happens as soon as they're written in the database. Later on, these can be queried by providing the name of the appropriate tags.
The nice thing about this is that the 'input-expected output' pairs are stored as traces. Thus, they behave exactly like regular prediction traces.
```python
from great_ai import add_ground_truth
add_ground_truth(
[1, 2],
['odd', 'even'],
tags='my_tag',
train_split_ratio=1, #(1)
test_split_ratio=1
)
```
1. Note that the ratios don't have to add up to 1. They are just weights. There is also a `validation_split_ratio` which is 0 by default.
```python
>>> from great_ai import query_ground_truth
>>> query_ground_truth('my_tag')
[Trace[str]({'created': '2022-07-12T18:36:12.825706',
'exception': None,
'feedback': 'odd', #(1)
'logged_values': {'input': 1}, #(2)
'models': [],
'original_execution_time_ms': 0.0,
'output': 'odd',
'tags': ['ground_truth', 'test', 'my_tag'], #(3)
'trace_id': '4fcf2ce6-a172-469d-94b2-874577655814'}),
Trace[str]({'created': '2022-07-12T18:36:12.825706',
'exception': None,
'feedback': 'even',
'logged_values': {'input': 2},
'models': [],
'original_execution_time_ms': 0.0,
'output': 'even',
'tags': ['ground_truth', 'train', 'my_tag'],
'trace_id': 'abee0671-beb9-4284-8c3b-c65e5836ce38'})]
```
1. Expected output. This can be also accessed through the `.output` property.
2. The input value is stored here.
3. Notice how `ground_truth` always included as a tag when using [great_ai.add_ground_truth][].
## Get feedback
After the initial data gathering, end-to-end feedback can be also integrated into the dataset.
The scaffolded REST API contains endpoints for managing traces and their feedbacks.
![screenshot of swagger](/media/feedback.png){ loading=lazy }
When [great_ai.query_ground_truth][] is executed, it implicitly filters for traces that have feedback. Therefore, both the `ground_truth` and the `online` traces that have received feedback are returned. No matter the origin of the data, it can be accessed using the same API.
## Remove clutter
Traces can be deleted either through the REST API or by calling [great_ai.delete_ground_truth][]. The latter provides the same interface as [great_ai.query_ground_truth][] except it deletes the matched points.

111
docs/how-to-guides/large_file.md
View file

@ -1,118 +1,111 @@
# How to use LargeFile
# How to use LargeFile-s
Storing, versioning, and downloading files from S3 made as easy as using `open()` in Python. Caching included.
The functions [save_model][great_ai.use_model] and [@use_model][great_ai.use_model] wrap LargeFile instances. Hence, besides configuring LargeFile, users have few reasons to use this.
## Motivation
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support the deletion](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) of large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
Oftentimes, especially when working with data-heavy applications, large files can proliferate in a repository. Version controlling them is an obvious next step, however, GitHub's git LFS implementation [doesn't support deleting](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository) large files, making it easy for them to eat-up the LFS quota and explode the size of your repos.
## Solution
```
pip install open-large
```
??? note "Using LargeFile-s directly (usually not needed)"
## Simple example
### Simple example
```python
from great_ai.large_file import LargeFileS3
```python
from great_ai.large_file import LargeFileS3
LargeFileS3.configure_credentials({
LargeFileS3.configure_credentials({
"aws_region_name": "your_region_like_eu-west-2",
"aws_access_key_id": "YOUR_ACCESS_KEY_ID",
"aws_secret_access_key": "YOUR_VERY_SECRET_ACCESS_KEY",
"large_files_bucket_name": "create_a_bucket_and_put_its_name_here",
})
})
# Creates a new version and deletes the older version leaving the 3 most recently used intact
with LargeFileS3("test.txt", "w", keep_last_n=3) as f:
# Creates a new version and deletes the older version
# leaving the 3 most recently used intact
with LargeFileS3("test.txt", "w", keep_last_n=3) as f:
for i in range(100000):
f.write('test\n')
# By default the latest version is returned
# but an optional `version` keyword argument can be provided as well
with LargeFileS3("test.txt", "r") as f:
# By default the latest version is returned
# but an optional `version` keyword argument can be provided as well
with LargeFileS3("test.txt", "r") as f: #(1)
print(f.readlines()[0])
```
```
> Automatically creates a file, writes to it, uploads it to S3, and then queries the most recent version of it.
> In this case, the latest version is already in the local cache, no download is required.
1. In this case, the latest version is already in the local cache, no download is required.
### More details
### More details
`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reading and writing is supported along with the [different keywords](https://docs.python.org/3/library/functions.html#open) `open()` accepts.
`LargeFile` behaves like an opened file (in the background it is a temp file after all). Binary reads and writes are supported along with the [different keywords `open()` accepts](https://docs.python.org/3/library/functions.html#open).
The local cache can be configured with these properties:
The local cache can be configured with these properties:
```python
LargeFile.cache_path = Path('.cache')
LargeFile.max_cache_size = "30 GB"
```
```python
LargeFileS3.cache_path = Path('.cache')
LargeFileS3.max_cache_size = "30 GB"
```
#### I only need a path
#### I only need a path
In case you only need a path to the "remote" file, this pattern can be applied:
In case you only need a path to the "remote" file, this pattern can be applied:
```python
path_to_model = LargeFile("folder-of-my-bert-model", version=31).get()
```
```python
path_to_model = LargeFileS3("folder-of-my-bert-model", version=31).get()
```
> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
> This will first download the file/folder into your local cache folder. Then, it returns a `Path` object to the local version. Which can be turned into a string with `str(path_to_model)`.
The same approach works for uploads:
The same approach works for uploads:
```python
LargeFile("folder-of-my-bert-model").push('path_to_local/folder_or_file')
```
```python
LargeFileS3("folder-of-my-bert-model").push('path_to_local/folder_or_file')
```
> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
> This way, both regular files and folders can be handled. The uploaded file is called **folder-of-my-bert-model**, the local name is ignored.
Lastly, all version of the remote object can be deleted by calling `LargeFile("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
Lastly, all version of the remote object can be deleted by calling `LargeFileS3("my-file").delete()`. It will still reside in your local cache afterwards, its deletion will happen next time your local cache has to be pruned.
### Command-line example
## From the command-line
The package can be used as a module from the command-line to give you more flexibility.
The main reason for using the `large-file` or `python3 -m great_ai.large_file` commands is to upload or download models from the terminal. For example, when building a docker image, it is best-practice to cache the referred models.
#### Setup
### Setup
Create an .ini file (or use _~/.aws/credentials_). It may look like this:
Create an .ini file (or use *~/.aws/credentials*). It may look like this:
```ini
aws_region_name = your_region_like_eu-west-2
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_VERY_SECRET_ACCESS_KEY
large_files_bucket_name = my_large_files
endpoint_url = this is optional, for backblaze, use this: https://s3.us-west-002.backblazeb2.com
```
> Just like in [example secrets](example_secrets.ini).
#### Print the expected options
### Upload some files
```sh
python3 -m large_file --help
```
#### Upload some files
```sh
python3 -m large_file --backend s3 --secrets secrets.ini --push my_first_file.json folder/my_second_file my_folder
large-file --backend s3 --secrets secrets.ini \
--push my_first_file.json folder/my_second_file my_folder
```
> Only the filename is used as the S3 name, the rest of the path is ignored.
#### Download some files to the local cache
!!! important "Using MongoDB"
The possible values for `--backend` are `s3`, `mongo`, and `local`. The latter doesn't need credentials, it only versions and stores your files in a local folder. MongoDB on the other hand requires a `mongo_connection_string` and a `mongo_database` to be specified. For storing large files, it uses the GridFS specification.
### Download some files to the local cache
This can be useful when building a Docker image for example. This way, the files can already reside inside the container and need not be downloaded later.
```sh
python3 -m large_file --backend s3 -secrets ~/.aws/credentials --cache my_first_file.json:3 my_second_file my_folder:0
large-file --backend s3 --secrets ~/.aws/credentials \
--cache my_first_file.json:3 my_second_file my_folder:0
```
> Versions may be specified by using `:`-s.
#### Delete remote files
### Delete remote files
```sh
python3 -m large_file --backend s3 --secrets ~/.aws/credentials --delete my_first_file.json
large-file --backend s3 --secrets ~/.aws/credentials \
--delete my_first_file.json
```

2
docs/how-to-guides/use-service.md
View file

@ -1,4 +1,4 @@
# How to use a GreatAI service
# How to deploy a GreatAI service
After [creating a GreatAI service](/how-to-guides/create-service) by wrapping your prediction function, and optionally [configuring it](/how-to-guides/configure-service), it's time to do some prediction.

BIN
docs/media/feedback.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 46 KiB

BIN
docs/media/thesis-frontpage.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 138 KiB

2
docs/tutorial/index.md
View file

@ -72,7 +72,7 @@ def predict_domain(sentence, model):
Finally, we test the model's inference function through the GreatAI dashboard. [The only thing left is to deploy the hardened-service properly.](/how-to-guides/use-service)
<div style="display: flex; justify-content: center;" markdown>
<div style="display: flex; justify-content: space-evenly;" markdown>
[:material-book: Learn about all the features](/how-to-guides/create-service){ .md-button .md-button--primary }
[:material-test-tube: Look at more examples](/examples/simple/data){ .md-button .md-button--secondary }

2
mkdocs.yaml
View file

@ -99,7 +99,7 @@ nav:
- how-to-guides/use-service.md
- how-to-guides/handle-training-data.md
- how-to-guides/large_file.md
- how-to-guides/call_remote.md
- how-to-guides/call-remote.md
- how-to-guides/scraping.ipynb
- Reference:
- reference/index.md