History

András Schmelczer 7192ba064a Imrpove examples		2022-05-26 13:33:27 +02:00
..
Dockerfile	Split examples	2022-05-25 22:54:21 +02:00
Dockerfile.development	Split examples	2022-05-25 22:54:21 +02:00
main_batch.py	Imrpove examples	2022-05-26 13:33:27 +02:00
main_service.py	Split examples	2022-05-25 22:54:21 +02:00
predict_domain.py	Imrpove examples	2022-05-26 13:33:27 +02:00
preprocess.py	Split examples	2022-05-25 22:54:21 +02:00
README.md	Split examples	2022-05-25 22:54:21 +02:00

README.md

Train Domain classifier on the semantic scholar dataset

Upload the dataset (or a part of it) to shared infrastructure

mkdir ss-data && cd ss-data
wget https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt
wget -B https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/ -i manifest.txt
cd -
python3 -m great_ai.open_s3 --secrets s3.ini --push ss-data
rm -rf ss-data