andras/great-ai
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases 4 Packages Wiki Activity Actions

More experimentation

Browse source 
Signed-off-by: András Schmelczer <andras@schmelczer.dev>
This commit is contained in:
Andras Schmelczer 2022-04-03 21:46:35 +02:00
parent 1c501db81a
commit 83d870d2ea
43 changed files with 749 additions and 496 deletions
Download patch file Download diff file Expand all files Collapse all files

12
example/README.md Normal file
View file

@ -0,0 +1,12 @@
# Train Domain classifier from the [semantic scholar dataset](https://api.semanticscholar.org/corpus)
## Upload the dataset (or a part of it) to shared infrastructure
```sh
mkdir ss-data && cd ss-data
wget https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt
wget -B https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/ -i manifest.txt
cd -
python3 -m good_ai.open_s3 --secrets s3.ini --push ss-data
rm -rf ss-data
```