Train a domain classifier on the semantic scholar dataset ¶

Part 1: obtain and clean data

position of this step in the lifecycle

The blue boxes show the steps implemented in this notebook.

Extract¶

This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.

In this case, we download the semantic scholar dataset from a public S3 bucket.

In [1]:

            
                Copied!
                
MAX_CHUNK_COUNT = 1
MAX_CHUNK_COUNT = 1

In [2]:

            
                Copied!
                
                    
                    
                
                

        
import urllib.request
from random import shuffle

manifest = (
    urllib.request.urlopen(
        "https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt"
    )
    .read()
    .decode()
)  # a list of available chunks separated by '\n' characters

lines = manifest.split()
shuffle(lines)
chunks = lines[:MAX_CHUNK_COUNT]

f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"
import urllib.request
from random import shuffle

manifest = (
    urllib.request.urlopen(
        "https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/manifest.txt"
    )
    .read()
    .decode()
)  # a list of available chunks separated by '\n' characters

lines = manifest.split()
shuffle(lines)
chunks = lines[:MAX_CHUNK_COUNT]

f"Processing {len(chunks)} out of the {len(manifest.split())} available chunks"

Out[2]:

'Processing 1 out of the 6002 available chunks'

Transform¶

Filter out non-English abstracts using great_ai.utilities.predict_language
Project it to only keep the necessary components (text and labels), clean the textual content using great_ai.utilities.clean
We will speed up processing using great_ai.utilities.parallel_map.

In [3]:

            
                Copied!
                
                    
                    
                
                

        
from typing import List, Tuple
import json
import gzip
from great_ai import parallel_map, clean, is_english, predict_language


def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
    # Extract
    response = urllib.request.urlopen(
        f"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}"
    )  # a gzipped JSON Lines file

    decompressed = gzip.decompress(response.read())
    decoded = decompressed.decode()
    chunk = [json.loads(line) for line in decoded.split("\n") if line]

    # Transform
    return [
        (
            clean(
                f'{c["title"]} {c["paperAbstract"]} {c["journalName"]} {c["venue"]}',
                convert_to_ascii=True,
            ),  # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts
            c["fieldsOfStudy"],
        )  # Create pairs of `(text, [...domains])`
        for c in chunk
        if c["fieldsOfStudy"] and is_english(predict_language(c["paperAbstract"]))
    ]


preprocessed_chunks = parallel_map(preprocess_chunk, chunks)
from typing import List, Tuple
import json
import gzip
from great_ai import parallel_map, clean, is_english, predict_language


def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
    # Extract
    response = urllib.request.urlopen(
        f"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2022-02-01/{chunk_key}"
    )  # a gzipped JSON Lines file

    decompressed = gzip.decompress(response.read())
    decoded = decompressed.decode()
    chunk = [json.loads(line) for line in decoded.split("\n") if line]

    # Transform
    return [
        (
            clean(
                f'{c["title"]} {c["paperAbstract"]} {c["journalName"]} {c["venue"]}',
                convert_to_ascii=True,
            ),  # The text is cleaned to remove PDF extraction, web scraping, and other common artifacts
            c["fieldsOfStudy"],
        )  # Create pairs of `(text, [...domains])`
        for c in chunk
        if c["fieldsOfStudy"] and is_english(predict_language(c["paperAbstract"]))
    ]


preprocessed_chunks = parallel_map(preprocess_chunk, chunks)

Spacy model en_core_web_sm not found locally, downloading...
Collecting en-core-web-sm==3.3.0
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.3.0/en_core_web_sm-3.3.0-py3-none-any.whl (12.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.8/12.8 MB 3.6 MB/s eta 0:00:00
Requirement already satisfied: spacy<3.4.0,>=3.3.0.dev0 in ./.env/lib/python3.10/site-packages (from en-core-web-sm==3.3.0) (3.3.1)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.64.0)
Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.9.1)
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.4.3)
Requirement already satisfied: typer<0.5.0,>=0.3.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.4.1)
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (59.6.0)
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.7)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.2)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.7.8)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.0.7)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.28.0)
Requirement already satisfied: numpy>=1.15.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.23.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.6)
Requirement already satisfied: pathy>=0.3.5 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (0.6.1)
Requirement already satisfied: jinja2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.1.2)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3.0)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.8.2)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.6)
Requirement already satisfied: thinc<8.1.0,>=8.0.14 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.0.17)
Requirement already satisfied: packaging>=20.0 in ./.env/lib/python3.10/site-packages (from spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.env/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.0.9)
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in ./.env/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (5.2.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.env/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (4.2.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2022.6.15)
Requirement already satisfied: idna<4,>=2.5 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (3.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.env/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (1.26.9)
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.env/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (8.1.3)
Requirement already satisfied: MarkupSafe>=2.0 in ./.env/lib/python3.10/site-packages (from jinja2->spacy<3.4.0,>=3.3.0.dev0->en-core-web-sm==3.3.0) (2.1.1)
Installing collected packages: en-core-web-sm
Successfully installed en-core-web-sm-3.3.0

2022-06-25 14:21:57,983 |  WARNING | Limiting concurrency to 1 because there are only 1 chunks
2022-06-25 14:21:57,984 |     INFO | Starting parallel map (concurrency: 1, chunk size: 1)
2022-06-25 14:21:57,984 |  WARNING | Running in series, there is no reason for parallelism
100%|██████████| 1/1 [03:26<00:00, 206.86s/it]

In [4]:

            
                Copied!
                
from itertools import chain

preprocessed_data = list(chain(*preprocessed_chunks))
X, y = zip(
    *preprocessed_data
)  # X is the input, y is the expected (ground truth) output
from itertools import chain

preprocessed_data = list(chain(*preprocessed_chunks))
X, y = zip(
    *preprocessed_data
)  # X is the input, y is the expected (ground truth) output

Load¶

Upload the dataset (or a part of it) to a central repository using great_ai.add_ground_truth. This step automatically tags each datapoint with a split label according to the ratios we set. Additional tags can be also given.

Production-ready backend¶

The MongoDB driver is automatically configured if mongo.ini exists with the following scheme:

mongo_connection_string=mongodb://localhost:27017/
mongo_database=my_great_ai_db

You can install MongoDB from here or use it as a service

In [5]:

            
                Copied!
                
from great_ai import add_ground_truth

add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)
from great_ai import add_ground_truth

add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)

2022-06-25 14:25:24,989 |  WARNING | Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️
2022-06-25 14:25:24,990 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising MongodbDriver
2022-06-25 14:25:24,991 |     INFO | Found credentials file (/data/projects/great_ai_example/mongo.ini), initialising LargeFileMongo
2022-06-25 14:25:24,992 |     INFO | Settings: configured ✅
2022-06-25 14:25:24,993 |     INFO | 🔩 tracing_database: MongodbDriver
2022-06-25 14:25:24,994 |     INFO | 🔩 large_file_implementation: LargeFileMongo
2022-06-25 14:25:24,994 |     INFO | 🔩 is_production: False
2022-06-25 14:25:24,995 |     INFO | 🔩 should_log_exception_stack: True
2022-06-25 14:25:24,996 |     INFO | 🔩 prediction_cache_size: 512
2022-06-25 14:25:24,997 |     INFO | 🔩 dashboard_table_size: 50
2022-06-25 14:25:24,998 |  WARNING | You still need to check whether you follow all best practices before trusting your deployment.
2022-06-25 14:25:24,998 |  WARNING | > Find out more at https://se-ml.github.io/practices/

Next: Part 2 ¶

Last update: July 11, 2022

Train a domain classifier on the semantic scholar dataset¶