Skip to content

# Simple example: data engineering¶

Here, we solve a problem similar to the tutorial's but with an explainable Naive Bayes classifier and more best-practices. In short, we train a domain classifier on the semantic scholar dataset by taking full advantage of `great-ai`. Subsequently, we create a production-ready deployment.

> The blue boxes show the steps of a typical AI-development lifecycle implemented in this notebook.

Since the true scope of `great-ai` is the phase between proof-of-concept code and production-ready service, it is predominantly used in the deployment notebook. Feel free to skip there, or continue reading if you'd like to see the full picture.

### Extract¶

This can be achieved by downloading a public dataset (such as in this case), or by having a Data Engineer setup and give us access to the organisation's data.

In this example, we download the semantic scholar dataset from a public S3 bucket.

In [1]:
Copied!

    MAX_CHUNK_COUNT = 4

MAX_CHUNK_COUNT = 4
In [2]:
Copied!

    import urllib.request
    from random import shuffle

    manifest = (
        urllib.request.urlopen(
            "https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/"
            "open-corpus/2022-02-01/manifest.txt"
        )
        .read()
        .decode()
    )  # a list of available chunks separated by '\n' characters

    lines = manifest.split()
    shuffle(lines)
    chunks = lines[:MAX_CHUNK_COUNT]

    f"""Processing {len(chunks)} out of the {
        len(manifest.split())
    } available chunks"""

import urllib.request from random import shuffle manifest = ( urllib.request.urlopen( "https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/" "open-corpus/2022-02-01/manifest.txt" ) .read() .decode() ) # a list of available chunks separated by '\n' characters lines = manifest.split() shuffle(lines) chunks = lines[:MAX_CHUNK_COUNT] f"""Processing {len(chunks)} out of the { len(manifest.split()) } available chunks"""
Out[2]:

    'Processing 4 out of the 6002 available chunks'

### Transform¶

- Filter out non-English abstracts using `great_ai.utilities.predict_language`
- Project it to only keep the necessary components (text and labels), clean the textual content using `great_ai.utilities.clean`
- We will speed up processing using `great_ai.utilities.simple_parallel_map`.

In [3]:
Copied!

    from typing import List, Tuple
    import json
    import gzip
    from great_ai.utilities import (
        simple_parallel_map,
        clean,
        is_english,
        predict_language,
        unchunk,
    )

    def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]:
        # Extract
        response = urllib.request.urlopen(
            f"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/"
            "open-corpus/2022-02-01/{chunk_key}"
        )  # a gzipped JSON Lines file

        decompressed = gzip.decompress(response.read())
        decoded = decompressed.decode()
        chunk = [json.loads(line) for line in decoded.split("\n") if line]

        # Transform
        return [
            (
                clean(
                    f'{c["title"]} {c["paperAbstract"]} '
                    f'{c["journalName"]} {c["venue"]}',
                    convert_to_ascii=True,
                ),  # The text is cleaned to remove common artifacts
                c["fieldsOfStudy"],
            )  # Create pairs of `(text, [...domains])`
            for c in chunk
            if (c["fieldsOfStudy"] and is_english(predict_language(c["paperAbstract"])))
        ]

    preprocessed_data = unchunk(
        simple_parallel_map(preprocess_chunk, chunks, concurrency=4)
    )

from typing import List, Tuple import json import gzip from great_ai.utilities import ( simple_parallel_map, clean, is_english, predict_language, unchunk, ) def preprocess_chunk(chunk_key: str) -> List[Tuple[str, List[str]]]: # Extract response = urllib.request.urlopen( f"https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/" "open-corpus/2022-02-01/{chunk_key}" ) # a gzipped JSON Lines file decompressed = gzip.decompress(response.read()) decoded = decompressed.decode() chunk = [json.loads(line) for line in decoded.split("\n") if line] # Transform return [ ( clean( f'{c["title"]} {c["paperAbstract"]} ' f'{c["journalName"]} {c["venue"]}', convert_to_ascii=True, ), # The text is cleaned to remove common artifacts c["fieldsOfStudy"], ) # Create pairs of `(text, [...domains])` for c in chunk if (c["fieldsOfStudy"] and is_english(predict_language(c["paperAbstract"]))) ] preprocessed_data = unchunk( simple_parallel_map(preprocess_chunk, chunks, concurrency=4) )

    100%|██████████| 4/4 [04:42<00:00, 70.62s/it]

In [4]:
Copied!

    X, y = zip(*preprocessed_data)  # X is the input, y is the expected output

X, y = zip(*preprocessed_data) # X is the input, y is the expected output

### Load¶

Upload the dataset (or a part of it) to a central repository using `great_ai.add_ground_truth`. This step automatically tags each data-point with a split label according to the ratios we set. Additional tags can be also given.

#### Production-ready backend¶

The MongoDB driver is automatically configured if `mongo.ini` exists with the following scheme:

    mongo_connection_string=mongodb://localhost:27017/
    mongo_database=my_great_ai_db

> You can install MongoDB from here or use it as a service

Otherwise, TinyDB is used which is just a local JSON file.

In [8]:
Copied!

    from great_ai import add_ground_truth

    add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)

from great_ai import add_ground_truth add_ground_truth(X, y, train_split_ratio=0.8, test_split_ratio=0.2)

### Next: Part 2¶

Last update: July 12, 2022