A meme generator demo can be viewed at meme.ashita.nl

Disclaimers:

• This article takes heavy inspiration from Dylan Wenzlau’s meme generation project
• The quality of the generated text will most likely be poor, thus to generate a “good” meme you will have to cherry pick between the model outputs

In this article we are going to create a text generation model for generating Drake memes. This post will be divided in four sections:

• Data collection
• Preprocessing and data augmentation
• Model training
• Meme generation!

Data collection

The first issue we have to tackle is data collection: memes come in a large variety of formats and they are continuously evolving, furthermore the text has to be extracted from the image.

Luckily, thanks to imgflip.com, we do not have to deal with these issues: imgflip offers collections of memes using premade templates, and most memes also have their text attached.

Our job for data collection is then reduced to a simple scraping script that, given the meme template URL:

• Iterates over the memes collection, one page at a time
• Using BeautifulSoup extract all the memes HTML image elements
• For each image, reads the alt HTML property, which will contain a string in the format

TEMPLATE_NAME | TOP_TEXT; BOTTOM_TEXT | TAGS | WATERMARK

We are only interested in the TOP_TEXT and BOTTOM_TEXT fields (there might be more text fields separated by a semicolon, depending on the chosen template).

Preprocessing

If you used the scraper you will now have a dataset in the format:

{
    "name": "Drake Hotline Bling",
    "memes": [
        {
            "url": "...",
            "text": "meme top; meme bottom"       
        }
    ]
}

We can now move on to pre-processing, which consists of the following steps:

• Text is lowercased, newlines are replaced by spaces
• Memes shorter than 15 characters and longer than 80 were discarded
• Non-English memes were removed using fastText
• Memes containing non-ASCII characters were removed (to keep the vocabulary size small)

You can download the preprocessing script here and you can run it with

python preprocess.py --filename source.json --out destination.json

Data augmentation

Since after preprocessing I was left with ~5000 memes I decided to try to augment the dataset by generating additional samples. For this purpose I used the nlpaug library, in particular I used ContextualWordEmbsAug with roberta-base. For each meme whose length was higher than 20 characters, I tried augmenting it three times and if one of the outputs matched the format TOP_TEXT; BOTTOM_TEXT; it would be added to the dataset.

You can perform the same augmentation using this script.

Model training

I trained two models for text generation, one based on Dylan Wenzlau’s architecture (using the Keras tokenizer and with a spell-checking step during prediction) and one based on textgenrnn.

The respective model training code is available here and here.

The first model accepts a JSON dataset in the following format:

{
    "memes": [
        {
            "text": "LINE_1; LINE_2; ...;"
        }
    ]
}

Meanwhile the training data for textgenrnn must be provided as a text file with a meme per line, in the following format

LINE_1; LINE_2; ...

To convert a JSON dataset to text you can use this script. Once you have a text dataset textgenrnn can be trained with just a few lines of code:

from textgenrnn import textgenrnn

# Setup
textgen = textgenrnn(name="model_name")
train_func = textgen.train_from_file

# Define and train the model
train_func(
    file_path="path_to_dataset.txt",
    # Whether to train a model from scratch or start from the textgenrnn default weights
    new_model=False, 
    num_epochs=10,
    gen_epochs=20,
    batch_size=256,
    train_size=0.75,
    dropout=0.2,
    validation=True,
    is_csv=False,
    rnn_layers=3,
    rnn_size=128,
    rnn_bidirectional=False,
    max_length=32,
    dim_embeddings=64,
    word_level=False)

And generating text is a piece of cake:

# The temperature controls the model "creativity", a lower temperature will produce
# outputs more similar to the training data
textgen.generate(5, temperature=0.8)

Meme generation

Now that we have our models we can move on to generating memes. The prediction scripts for the convolutional and textgenrnn models can be respectively found here and here.

Taking the textgenrnn prediction script as an example we first generate a potential candidate:

sample = textgen.generate(
    n=1,
    return_as_list=True,
    temperature=0.5)[0]

We keep only samples with the right number of lines (2 for drake memes), each terminated by a semicolon.

segments = [w.strip() for w in sample[:-1].split(";")]

if len(segments) != 2:
    continue

And finally we use memegen.link to render an image!

def escape_meme_text(text):
    replacements = {
        " ": "_",
        "?": "~q",
        "%": "~p",
        "#": "~h",
        "/": "~s",
        "''": "\"",
    }

    for r in replacements.keys():
    text = text.replace(r, replacements[r])

    return text


def generate_meme(top_text, bottom_text, meme_type):
    top_text = escape_meme_text(top_text)
    bottom_text = escape_meme_text(bottom_text)
    url = f"https://memegen.link/{meme_type}/{top_text}/{bottom_text}.jpg"
    res = requests.get(url)

    return res.content

top_text, bottom_text = "I am the top text", "I am the bottom text"
meme_image_data = generate_meme(top_text, bottom_text, "drake")

# Write the image to file
with open(f"output.png", "wb") as out_image:
    out_image.write(meme_image_data)

Bonus: an API wrapper

If you want to serve your meme generation models via a REST API head to this repository. It contains a simple FastAPI wrapper with a single endpoint /meme/{template_id}/{temperature} that returns a meme in the following format:

{
    "top": top_text,
    "bottom": bottom_text,
    "image": base64_encoded_image
}

To ease deployment the API can be built and shipped as a Docker container, with optional HTTPS support.

Bonus #2

This project was built for a class on natural language processing, and I wrote a paper for it which you can download here