Description
Assignment 1:
Word Embeddings – Properties, Meaning and Training
Welcome to the first assignment of ECE 1786! This assignment engages you in the first major
topic of the course – the properties, meaning, viewing, and training of word embeddings (also called
word vectors). This assignment must be done individually.
The specific learning objectives in this
assignment are:
1. To set up the computing environment used in this course.
2. To learn word embedding properties, and use them in simple ways.
3. To translate vectors into understandable categories of meaning.
4. To understand how embeddings are created, using two methods – the Skip Gram method,
and the Skip Gram with Negative Sampling (SGNS) method, and to get a sense of trade-offs
in the creation of embeddings.
Late Penalty: There is a penalty-free grace period of one hour past the deadline. Any work that
is submitted between 1 hour and 24 hours past the deadline will receive a 20% grade deduction.
No other late work is accepted.
What To Submit
You should submit the following to the Quercus website for this course, under Assignment 1:
1. A single PDF document, called Assign1.pdf which answers every question posed in Sections
1 through 4 of this assignment. You should number your answer to each question in the form
Question X.Y, where X is the section of this assignment, and Y is the numbered element in
the question. You should include the specific written question itself and then provide
your answer.
2. You should submit separate code files for the code that you wrote for each section, as specified
in each section, with the specified file name. For example, Section 1 Part 2 asks you to submit
a python (text) file named A1P1_2.py. Note that you’re required to submit python files, not
notebook (.ipynb)files.
Before You Begin: Set Up Your Computing Environment
Set up your computing environment according to the document Course Software Infrastructure and Background provided with this assignment. Note that the pre-requisite of this course
requires that you have experience and knowledge of software systems like (or exactly the same) as
described in that document.
1 Properties of Word Embeddings [15 points]
In the first lecture of this course, we discuss properties of word embeddings/vectors, and use a
PyTorch notebook to explore some of their properties. You can retrieve that code in notebook
provided with this assignment called A1_Section1_starter.ipynb. It illustrates the basic properties of the GloVe embeddings [1]. As a way to gain familiarity with word vectors, do each of the
following:
1. Read the documentation of the Vocab class of Torchtext that you can find here: https://
torchtext.readthedocs.io/en/latest/vocab.html and then read the A1_Section1_starter.ipynb
code. Run the notebook and make sure you understand what each step does.
2. Write a new function, similar to print_closest_words called print_closest_cosine_words
that prints out the N-most (where N is a parameter) similar words using cosine similarity
rather than euclidean distance. Provide a table that compares the 10-most cosine-similar
words to the word ‘dog’, in order, alongside to the 10 closest words computed using euclidean
distance. Give the same kind of table for the word ‘computer.’ Looking at the two lists,
does one of the metrics (cosine similarity or euclidean distance) seem to be better than the
other? Explain your answer. Submit the specific code for the print_closest_cosine_words
function that you wrote in a separate Python file named A1P1_2.py. [2 points]
3. The section of A1_Section1_starter.ipynb that is labelled Analogies shows how relationships between pairs of words is captured in the learned word vectors. Consider, now, the
word-pair relationships given in Figure 1 below, which comes from Table 1 of the Mikolov[2]
paper. Choose one of these relationships, but not one of the ones already shown in the starter
notebook, and report which one you chose. Write and run code that will generate the second
word given the first word. Generate 10 more examples of that same relationship from 10
other words, and comment on the quality of the results. Submit the specific code that you
wrote in a separate Python file, A1P1_3.py. [4 points]
4. The section of A1_Section1_starter.ipynb that is labelled Bias in Word Vectors illustrates examples of bias within word vectors in the notebook, as also discussed in class. Choose
a context that you’re aware of (different from those already in the notebook), and see if you
can find evidence of a bias that is built into the word vectors. Report the evidence and the
conclusion you make from the evidence. [2 points]
5. Change the the embedding dimension (also called the vector size) from 50 to 300 and rerun the notebook including the new cosine similarity function from part 2 above. How does
the euclidean difference change between the various words in the notebook when switching
from d=50 to d=300? How does the cosine similarity change? Does the ordering of nearness
change? Is it clear that the larger size vectors give better results – why or why not? [5 points]
6. There are many different pre-trained embeddings available, including one that tokenizes
words at a sub-word level[3] called FastText. These pre-trained embeddedings are available
from Torchtext. Modify the notebook to use the FastText embeddings. State any changes
that you see in the Bias section of the notebook. [2 points]
Figure 1: Mikolov’s Pairwise Relationships
2 Computing Meaning From Word Embeddings [12 points]
Now that we’ve seen some of the power of word embeddings, we can also feel the frustration that
the individual elements/numbers in each word vector do not have a meaning that can be interpreted
or understood by humans. It would have preferable that each position in the vector correspond to
a specific axis of meaning that we can understand based on our ability to comprehend language.
For example the “amount” that the word relates to colour or temperature or politics. This is not
the case, because the numbers are the result of an optimization process that does not drive each
vector element toward human-understandable meaning.
We can, however, make use of the methods shown in Section 1 above to measure the amount of
meaning in specific categories of our choosing, such as colour. Suppose that we want to know
how much a particular word/embedding relates to colour. One way to measure that could be to
determine the cosine similarity between the word embedding for colour and the word of interest. We
might expect that a word like ‘sky’ or ‘grass’ might have elements of colour in it, and that ‘purple’
would have more.
However, it may also be true that there are multiple meanings to a single word,
such as ‘colour’, and so it might be better to define a category of meaning by using several words
that, all together, define it with more precision.
For example, a way to define a category such as colour would be to use that word itself, and together with several examples, such ‘red’, ‘green’, ‘blue’, ‘yellow.’ Then, to measure the “amount”
of colour in a specific word (like ‘sky’) you could compute the average cosine similarity between
sky and each of the words in the category.
Alternatively, you could average the vectors of all
the words in the category, and compute the cosine similarity between the embedding of sky and
that average embedding. In this section, use the d=50 GloVe embeddings that you used in Section 1.
Do the following:
1. Write a PyTorch-based function called compare words to category that takes as input:
sun moon winter
rain cow wrist
wind prefix ghost
glow heated cool
Table 1: Vocabulary to Test in Section 2
• The meaning category given by a set of words (as discussed above) that describe the
category, and
• A given word to ‘measure’ against that category.
The function should compute the cosine similarity of the given word in the category in two
ways:
(a) By averaging the cosine similarity of the given word with every word in the category,
and
(b) By computing the cosine similarity of the word with the average embedding of all of the
words in the category.
Submit the specific code that you wrote in a separate Python file, A1P2_1.py. [2 points]
2. Let’s define the colour meaning category using these words: “colour”, “red”, “green”, “blue”,
“yellow.” Compute the similarity (using both methods (a) and (b) above) for each of these
words: “greenhouse”, “sky”, “grass”, “azure”, “scissors”, “microphone”, “president” and
present them in a table. Do the results for each method make sense? Why or why not? What
is the apparent difference between method 1 and 2? [4 points]
3. Create a similar table for the meaning category temperature by defining your own set of
category words, and test a set of 10 words that illustrate how well your category works as
a way to determine how much temperature is “in” the words. You should explore different
choices and try to make this succeed as much as possible. Comment on how well your approach
worked. [4 points]
4. Use these two categories (colour & temperature) to create a new word vector (of dimension
2) for each of the words given in Table 1, in the following way: for each word, take its (colour,
temperature) cosine similarity numbers (try both methods and see which works better), and
apply the softmax function to convert those numbers into a probabilities. Plot each of the
words in two dimensions (one for colour and one for temperature) using matplotlib. Do the
words that are similar end up being plotted close together? Why or why not? [2 points]
3 Training A Word Embedding Using the Skip-Gram Method on
a Small Corpus [16 points]
In lecture 2, we described the Skip Gram method of training word embeddings [2]. In this Section
you are going to write code to use that method to train a very small embedding, for a very small
vocabulary on very small corpus of text. The goal is to gain some insight into the general notion
of how embeddings are produced in a neural net training context. The corpus you are going to use
was provided with this assignment, in the file SmallSimpleCorpus.txt.
Your task is to write complete code (given some starter code) to train, test and display the trained
embeddings, using the skip gram method. (In Section 4 you’ll use a different, more efficient method).
You can find the starter code in the provided file A1_Section3_starter.ipynb.
1. First, read the file SmallSimpleCorpus.txt so that you see what the sequence of sentences
is. Recalling the notion “you shall know a word by the company it keeps,” find three pairs
of words that this corpora implies have similar or related meanings. For example, ‘he’ and
‘she’ are one such example – which you cannot use in your answer! [1 point]
2. The prepare_texts function in the starter code is given to you and fulfills several key functions in text processing, a little bit simplified for this simple corpus. Rather than full tokenization (covered in Section 4 below, you will only lemmatize the corpus, which means converting
words to their root – for example the word “holds” becomes “hold”, whereas the word “hold”
itself stays the same (see the Jurafsky [4] text, section 2.1 for a discussion of lemmatization).
The prepare_texts function performs lemmatization using the spaCy library, which also performs parts of speech tagging. That tagging determines the type of each word such as noun,
verb, or adjective, as well as detecting spaces and punctuation. Jurafsky [4] Section 8.1 and
8.2 describes parts-of-speech tagging. The function prepare_texts uses the parts-of-speech
tag to eliminate spaces and punctuation from the vocabulary that is being trained.
Review the code of prepare_texts to make sure you understand what it is doing. Write
the code to read the corpus SmallSimpleCorpus.txt, and run the prepare_texts on it to
return the text (lemmas) that will be used next. Check that the vocabulary size is 11. Which
is the most frequent word in the corpus, and the least frequent word? What purpose do the
v2i and i2v functions serve? [2 points]
3. Write a new function called tokenize_and_preprocess_text the skeleton of which is given
in the starter code, but not written. It takes the lemmatized small corpus as input, along
with v2i (which serves as a simple, lemma-based tokenizer) and a window size window.
You
should write it so that its output should be the Skip Gram training dataset for this corpus:
pairs of words in the corpus that “belong” together, in the Skip Gram sense. That is, for
every word in the corpus a set of training examples are generated with that word serving as
the (target) input to the predictor, and all the words that fit within a window of size window
surrounding the word would be predicted to be in the “context” of the given word. The words
are expressed as tokens (numbers).
To be clear, this definition of window means that only
odd numbers of 3 or greater make sense. A window size of 3 means that there are maximum
2 samples per target word, using the one word before and the one word after.
For example, if the corpus contained the sequence then the brown cow said moo, and if
the current focus word was cow, and the window size was window=3, then there would be
two training examples generated for that focus word: (cow, brown) and (cow, said).
You
must generate all training examples across all words in the corpus within a window of size
window. Test that your function works, and show with examples of output (submitted) that
it does. [2 points]
4. Next you should define the model to be trained, the skeleton for which is give in the starter
code class Word2vecModel. Portions of the weights in this model, once trained, provides
the trained embeddings we are seeking. Recall that the input to the model is a token (a
number) representing which word in the vocabulary is being predicted from. The output of
the model is of size |V|, where V is the size of the vocabulary, and each individual output in
some sense represents the probability of that word being the correct output. That prediction
is based directly on the embedding for each word, and the embeddings are quantities being
determined during training. Set the embedding size to be 2, so that will be the size of
our word embeddings/vectors. What is the total number of parameters in this model with
an embedding size of 2 – counting all the weights and biases? Submit your code for the
Word2VecModel class in the file A1P3_4.py. [2 points]
5. Write the training loop function, given in skeleton form in the starter code as function
train_word2vec. It should call the function tokenize_and_preprocess_text to obtain
the data and labels, and split that data to be 80% training and 20% validation data. It
should use a Cross Entropy loss function, a batch size of 4, a window size of 5, and 50 Epochs
of training. Using the default Adam optimizer, find a suitable learning rate, and report what
that is.
Show the training and validation curves (loss vs. Epoch), and comment on the apparent success (or lack thereof) that these curves suggest. Submit your code for the training
function train_word2vec in the file A1P3_5.py [4 points]
6. For your best learning rate, display each of the embeddings in a 2-dimensional plot using
Matplotlib. Display both a point for each word, and the word itself. Submit this plot, and
answer this question: Do the results make sense, and confirm your choices from part 1 of
this Section? What would happen when the window size is too large? At what value would
window become too large for this corpus? [5 points]
7. Run the training sequence twice – and observe whether the results are identical or not. Then
set the random seeds that are used in, separately in numpy and torch as follows (use any
number you wish, not necessary 43, for the seed):
np.random.seed(43)
torch.manual_seed(43)
Verify (and confirm this in your report) that the results are always the same every time you
run your code if you set these two seeds. This will be important to remember when you are
debugging code, and you want it to produce the same result each time.
4 Skip-gram with Negative Sampling [14 points + 2 bonus points]
One big issue with the pure skip gram training method presented in the Section 3 is that it is quite
slow, largely because the output of the model is the size of the vocabulary, and normal vocabularies
are in the range of 5,000 or more words. An alternative is called skip gram with negative sampling,
or SGNS for short. Here, rather than predict a word that is contextually associated with another,
we build a binary predictor that says whether two words belong together as part of the same context. To make such a binary predictor, the training data has to contain both positive examples (as
in Section 3) and negative examples, hence the ‘negative sampling’ in the SGNS name. Section 6.8
of the Jurafsky text [4] gives a detailed description of this approach.
The goal in this Section to gain more understanding into how embeddings are made, using a larger
embedding size and a much larger corpus, with a much larger vocabulary. The corpus you are going
to use was provided with this assignment, in the file LargerCorpus.txt. It comes from a book
called “Illustrated History of the United States Mint.” We note that if you tried to train a word
embedding of reasonable size on this corpus, using the method of Section 3, it would be very slow!
Starter code is provided in the file A1_Section4_starter.ipynb.
1. Take a quick look through LargerCorpus.txt to get a sense of what it is about. Give a 3
sentence summary of the subject of the document. [1 point]
2. The prepare_texts function in the starter code is a more advanced version of that same
function given in Section 3. Read through it and make sure you understand what it does.
What are the functional differences between this code and that of the same function in
Section 3? [1 point]
3. Write the code to read in LargerCorpus.txt and run prepare_texts on it. Determine the
number of words in the text, and the size of the filtered vocabulary, and the most frequent
20 words in the filtered vocabulary, and report those. Of those top 20 most frequent words,
which one(s) are unique to the subject of this particular text? [1 point]
4. Write the function tokenize_and_preprocess_text which generates both positive and negative samples for training in the following way: first, use w2i to create a tokenized version of
the corpus. Next, for every word in the corpus, generate: a set of positive examples within a
window of size window similar to the example generation in Section 3. Set the label for each
positive example to be +1, in the list Y produced by the function.
Also, for each word, generate the same number of negative samples as positive examples, by choosing that number of
randomly-chosen words from the entire corpus. (You can assume randomly chosen words are
very likely to be not associated with the given word). Set the label of the negative examples
to be -1. Submit your code for this function in the file named A1P4_4.py. How many total
examples were created? [2 points]
5. OPTIONAL: The training can be made more efficient by reducing the number of examples
for the most frequent words, as the above method creates far more examples connected to
those words than are necessary for successful training. Revise your function to reduce the
number of examples. Submit your code for this function in the file named A1P4_5.py, and
state how many examples remain for the corpus using this reduction. [2 bonus points]
6. Write the model class, from the given skeleton class SkipGramNegativeSampling. The
model’s input are the tokens of two words, the word and the context. The model stores
the embedding that is being trained (just one embedding per token), which is set up in the
_init_ method. The output of the forward function is a binary prediction, but it should
only compute the raw prediction of the network (which is the dot product of the the two
embeddings of the input tokens). Submit your model class in the file A1P4_6.py. [2 points]
7. Write the training function, given in skeleton form in the starter code as the function
train_sgns. This function should call the tokenize_and_preprocess_text function to
obtain the data and labels, and split that data to be 80% training and 20% validation
data. It should use an embedding size of 8, a window size of 5, a batch size of 4, and
30 Epochs of training. The loss function should be log(σ(prediction)) for positive examples,
and log(σ(−prediction)) where σ is the sigmoid function. Since it is possible for the prediction to be 0, to prevent the log function from having an infinite result, we typically add a
small constant the output of the sigmoid to prevent this – typically 1e-5.
Using the default Adam optimizer, find a suitable learning rate, and report what that is.
Show the training and validation curves vs. Epoch, and comment on the apparent success (or
lack thereof) that these curves suggest. Submit your training function in the file A1P4_7.py.
[4 points]
8. Write a function that reduces the dimensionality of the embeddings from 8 to 2, using principle
component analysis (PCA) as shown in the partially-written function visualize_embedding.
Since we cannot visualize the embeddings of the entire vocabulary, the function is written to
select a range of the most frequent words in the vocabulary. (Too frequent are not interesting,
but too infrequent are also less interesting). Comment on how well the embeddings worked,
finding two examples each of embeddings that appear correctly placed in the plot, and two
examples where they are not. [3 points]
References
[1] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for
word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October 2014. Association for
Computational Linguistics.
[2] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word
representations in vector space. https://arxiv.org/abs/1301.3781, 2013.
[3] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics,
5:135–146, 2017.
[4] Dan Jurafsky and James H. Martin. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition, 3rd edition draft.
https://web.stanford.edu/ jurafsky/slp3/, 2023.
8
Assignment 2:
Subjective/Objective Sentence Classification Using MLP and
CNN
Learning Objectives
In this assignment you will:
1. Make use of pre-trained word vectors as a basis for classifying text
2. Implement an Multi-Layer Perceptron and a Convolutional Neural Network architecture for
text classification. You’ll explore some relevant hyperparameters, and look at the resulting
trained features for insights.
3. Practice doing qualitative analysis of results.
4. Explore quantitative methods to explain the results.
5. Build an interactive application using Gradio https://gradio.app.
What To Submit
You should hand in the following files:
• A PDF file assignment2.pdf containing answers to the written questions in this assignment.
You should number your answer to each question in the form Question X.Y.Z, where X is
the section of this assignment, Y is the subsection, and Z is the numbered element in the
question. You should include the specific written question itself and then provide
your answer.
• You should submit separate code and model files for the code and models that you wrote as
specified in Sections 4.8, 5.4 and 6.2.
1 Subjective/Objective Classification Problem Definition
The first lectures of this course, and the first assignment showed how textual words can be turned
into word vectors that represent the meaning of a word. These form the basis of all forms of modern deep-learning based NLP. In this assignment we will build models that classify a sentence as
objective (a statement of fact) or subjective (a statement based on opinion). To begin, make sure
you understand the distinction between these two words – look up a few definitions of these words,
when used as adjectives.
We will make use of word vectors that have already been created (as you now know, trained), and
use them as the basis for the two classifiers that you will build.
As in Assignment 1, each word (or portion of a word) is first tokenized, where the word is converted
into an identifying number. This number is referred to either as the word’s index or as the word
token. With this index, the word vector corresponding to the word can be retrieved from a lookup
table, which is referred to as the embedding matrix.
In this assignment, your code will pass these indices into two different neural network models to
achieve the classification of a sentence – that it is subjective or objective – as illustrated below:
“The fight scenes are fun” Tokenize 4 427 453 32 249 NN
Model 0.9
Text Sentence Discrete tokens
Output
Probability
(Subjective)
Figure 1: High Level diagram of the Assignment 2 Classifiers for Subjective/Objective
Note: the first ‘layer’ of the neural network model will actually be the step that converts the
index/token into a word vector. (This could have been done on all of the training examples, but
that would greatly increase the amount of memory required to store the examples). From the first
layer on, the neural network deals only with the word vectors.
2 Software Environment and Dataset
2.1 Software Environment
The software environment you’ll be using in this assignment is the same as that used in Assignment
1, including PyTorch, torchtext and SpaCy. The document given as part of Assignment 1,
“Course Software Infrastructure and Background” tells you what to use and install.
2.2 Dataset
We will use the Subjectivity dataset [2], introduced in the paper by Pang and Lee [5]. The data
comes from portions of movie reviews from the Rotten Tomatoes website [3] (which are assumed
all be subjective) and summaries of the plot of movies from the Internet Movie Database (IMDB)
[1] (which are assumed all be objective). This approach to labeling the training data as objective
and subjective may not be strictly correct, but will work for our purposes. Note that some of
these sentences may be labelled incorrectly, and discovering those incorrect labels is part of this
assignment.
3 Preparing the data (5 points)
3.1 Human Data Review
The data for this assignment was provided in the file data.tsv that you downloaded from Quercus.
This is a tab-separated-value (TSV) file that contains two columns, text and label. The text
column contains a text string (including punctuation) for each sentence (or fragment or multiple
sentences) that is a data sample. The label column contains a binary value {0,1}, where 0 represents the objective class and 1 represents the subjective class.
Do/answer each of the following questions:
1. Review the data to see how it is organized in the file. How many examples are in the file
data.tsv?
2. Select two random examples each from the positive set (subjective) and two from the negative
set. For all four examples, explain, in English, why it has the given label. [1 point]
3. Find one example from each of the positive and negative sets that you think has the incorrect
label, and explain why each is wrong [2 points].
3.2 Create train/validation/test splits and Overfit Dataset
You will need to divide the available data into three main datasets: training, validation and test
(make sure you know why all model training needs three datasets, and not 2!). You can use
train_test_split from the sklearn library (or some other splitter if you prefer) to split the
data.tsv into 3 files:
1. train.tsv: this file should contain 64% of the total data
2. validation.tsv: this file should contain 16% of the total data
3. test.tsv: this file should contain 20% of the total data
Finally, create a fourth dataset, called overfit.tsv also with equal class representation, that contains only 50 training examples for use in debugging, as discussed in Section 4.4 below. This dataset
will be used separately from the other three, so it can overlap with those datasets.
You should (programmatically) verify that there are equal number of examples in the two classes
in each of the datasets, and that you did not accidentally put the same sample in more than one of
the training, validation and test sets. Your code should report both of these checks. Submit your
code to perform these functions in the file A2P3_2.py. [2 points]
3.3 Processing the Training Data
We have provided starter code that processes the training, validation and test data for you, once
it has been split into the files described above. It can be found in the A2_Starter.py given
with this assignment. It uses the Pytorch DataLoader class to process the input data in this
assignment, as described in this tutorial. The code described below is present in the skeleton code
file A2_Starter.py. Read this section together with the code, and make sure you understand what
the code is doing.
1. The torchtext.vocab loads the GloVe vectors, in a manner similar to Assignment 1.
glove = torchtext.vocab.GloVe(name=”6B”, dim=args.emb_dim)
This loads word vectors into a GloVe class (see documentation https://torchtext.readthedocs.
io/en/latest/vocab.html#torchtext.vocab.GloVe) This GloVe model was trained with
six billion words to produce a word vector size of 100, as described in class.
This will download a rather large 862 MB zip file into the folder named .vector cache, which might
take some time; this file expands into a 3.3Gbyte set of files, but you will only need one of
those files, labelled glove.6B.100d.txt, and so you can delete the rest (but don’t delete the
file glove.6B.100d.txt.pt that will be created by A2_Starter.py, which is the binary form
of the vectors).
Note that .vector cache folder, because it starts with a ‘.’, is typically
not a visible folder, and you’ll have to make it visible with an operating system-specific view
command of some kind. (Windows, Mac) Once downloaded your code can now access the
vocabulary object within the text field object by calling .vocab attribute on the text field
object.
The glove object contains the index (also called word token) for most words in the data set.
For those words that do not occur in GloVe’s vocabulary range, we substitute it for the last
word in the vocabulary.
2. Next, the code below (also given in the starter code) loads the training, validation, and test
datasets to become TextDataset objects, a class extending the Pytorch Dataset class. This
method is designed specifically for text input. A2_Starter.py uses the following code, which
assumes that the tsv files are in the folder data:
train_dataset = TextDataset(glove, “train”)
val_dataset = TextDataset(glove, “validation”)
test_dataset = TextDataset(glove, “test”)
Additional details on the Pytorch Dataset class can be found at: https://pytorch.org/
tutorials/beginner/basics/data_tutorial.html Later, in Section 4.4 you’ll also need to
use the “overfit” dataset you created in Section 3.2.
3. Next, we need to create a DataLoader object that can be enumerated (Python-style) in the
training loop. The objects in each batch is specified by the “collate function”, collate fn.
This code is given to you in the starter code, A2 Starter.py. (You can review data loading in the Pytorch documentation to learn more about the DataLoader, and the function
collate fn, but the code is given to you.)
These DataLoaders are iterable objects that will produce the batches of batch_size samples
in each training inner loop step. In the given implementation of the collate function, a tuple
of two elements will be returned in each batch: An integer torch.tensor of size [length x
batch size], and a float torch.tensor of size [batch size].
Recall that, for the CNN model, the input strings of a given batch should all be of the
same length, and so shorter strings are padded with zeroes. The collate function returns a
tokenized and padded batch of text. Tokenization is done simply by splitting into words using
whitespace, and looking up the token using vocab.stoi.get.
4 Baseline Model and Training (10 points)
Using the above starter code, design and train the baseline model described below.
4.1 Embedding Layer
As mentioned in Section 1, we will make use of word vectors that have already been created/trained.
We will use the GloVe [6] pre-trained word vectors as an embedding matrix. The step that converts
the input words from an index number (the word token) into the word vector is usually done inside
the nn.Module model class. So, when defining the layers in your model class, you must add a layer
with the function nn.Embedding.from pretrained, and pass in vocab.vectors as the argument
where vocab is the Vocab object.
self.embedding = nn.Embedding.from_pretrained(vocab.vectors)
There is an optional argument, ‘freeze‘ (default=True) in the ‘from pretrained‘ method. When
set to False, the embedding vectors themselves are trained together with the model’s parameters.
This is an important notion that sometimes takes some time to get used to – that the inputs
presented to a neural net are themselves learned or updated through back propagation/gradient
descent.
We will make use of this feature later, in Section 5.2. More detail can be found here:
https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html.
4.2 Baseline Model
0.9
-0.4
0.2
1.2
The fight scenes are fun
4
0.2
0.1
0.9
-1.5
0.2
-0.1
1.1
0.3
0.1
-0.2
2.1
-1.3
0.5
-0.1
0.3
1.1
0.38
-0.14
0.92
-0.2
Word
Embedding
Average
0.9
Fully Connected
Vocab Index 427 453 32 249
Prediction
Figure 2: A simple baseline architecture
The baseline model was discussed in class and is illustrated in Figure 2. It first converts each of
the word tokens into a vector using the GloVe word embeddings. It then computes the average of
the word embeddings in the input sentence. The notion is that this average becomes the “average
meaning” of the entire sentence. This is fed to a fully connected layer which produces a scalar
output with sigmoid activation (which should be computed inside the BCEWithLogitsLoss losss
function) to represent the probability that the sentence is in the subjective class.
4.3 Training the Baseline Model
Using the A2 Starter.py code, write a training loop to iterate through the training dataset and
train the baseline model. Use the hyperparameters given in Table 1, but find a suitable batch size.
Hyperparameter Value
Optimizer Adam
Learning Rate 0.001
Number of Epochs 50
Loss Function BCEWithLogitsLoss()
Table 1: Hyperparameters to Use in Training the Models
4.4 Overfitting to debug
One very useful way to debug your model is to see if you can overfit your model and reach a much
higher training accuracy than validation accuracy. Use the overfit.tsv file that you created earlier to do this. The purpose of doing this is to be able to make sure that the input processing
and output measurement is working.
You will need more than 25 epochs to succeed in overfitting.
(Note that it is hard to overfit the baseline model to get an accuracy of 100% because it doesn’t
have that many parameters; the CNN model in the next section will have enough to reach 100%).
So, do not proceed into the next sections until you have some overfitting, because it will be harder
to debug with more data, and whatever the problem is can more easily be found here (such as
mislabelled data, or errors in the data handling).
It is also recommended that you include some useful logging in the loop to help you keep track of
progress, and help in debugging.
Provide the training loss and accuracy plot for the overfit data in your Report. [1 point]
4.5 Full Training Data
Now you should use the full training dataset to train your model, using the hyperparameters given
in Table 1. Although this model will get better with more epochs, 50 epochs is enough to illustrate
what it is doing.
Using your A2 Starter.py code (in notebook or raw python form) write an evaluation loop to
iterate through the validation dataset to evaluate your model. We recommend that you call the
evaluation function in the training loop (perhaps every epoch or two) to make sure your model isn’t
overfitting. Keep in mind if you call the evaluation function too often, it will slow down training.
Give the training and validation loss and accuracy curves vs. epoch in your report, and report
the final test accuracy. Evaluate the test data and provide the accuracy result in your report.
Answer this questions: In the baseline model, what information contained in the original sentence
is being ignored? [1 points]
4.6 Extracting Meaning from the Trained Parameters
The dimension of the parameters in the linear neuron is the same size as the word embedding,
which suggests that there is a meaning attributable to the learned parameters. You can explore
that meaning using the function print_closest_cosine_words from Assignment 1. Use that
function to determine the 20 closest words to those trained parameters of the neuron. You should
see some words that make it clear what the classifier is doing. Do some of the words that you
generated make sense? Explain. [4 points]
4.7 Saving and loading your model
Your trained model will be used later, in Section 6. You should save the parameters of your trained
model with the lowest validation error using torch.save(model.state dict(), ’model baseline.pt’).
See https://pytorch.org/tutorials/beginner/saving_loading_models.html for detail on saving and loading PyTorch networks.
4.8 Submit Baseline Code
Submit your full code for this section in either in a notebook file named A2_Baseline.ipynb or in
a zip file containing all your python files named A2_Baseline.zip. Your code should clearly state
how it should be run, and it should have easy-to-use arguments that allow any part of this Section
to be run. [4 points]
5 Convolutional Neural Network (CNN) Classifier (15 points)
Vector/Embedding Embedding Dim
0.9
Fully Connected
The fight scenes are fun
4 427 453 32 249
Prediction
Max-pooling
over outputs
from each kernel
Convolutional
Layers with
different filter width
Figure 3: A convolutional neural network architecture
The second architecture, described in class and illustrated in Figure 3, is to use a CNN-based
architecture that is inspired by Yoon et al. [4]. Yoon first proposed using CNNs in the context of
NLP.
You will write the code for the CNN model class from the following specifications:
1. Group together all the vectors of the words in a sentence to form a embedding dim * N
matrix. Here N is the number of words (and therefore tokens) in the sentence. Different
sentences will have different lengths, but the batch loading function pads each sentence in a
batch with zeroes to make them all the same length. Note that embedding dim is the size of
the word vector, 100.
2. The architecture consists of two convolutional layers that both operate on the word vector
group created above, but with different kernel sizes. The kernel sizes are [ki, embedding dim].
You should write your code to allow for the parameterization of k1 and k2 as well as the number of kernels n1 and n2.
Note that this organization of convolutional layers is different from
traditional computer vision-oriented CNNs in which one layer is fed into the next; these layers
are operating on the same input. Note also, that, even though the kernel sizes span the entire
embedding dimension, you can use the nn.Conv2d method, and explicitly specify the size of
a kernel using the kernel_size=(kx_size,ky_size) parameter.
3. Set the convolutional kernel to remove the bias term. This is necessary for Section 5.2
below, in which you explore the words the kernels are closest to. This is done by setting
the parameter bias=False when you instantiate the Conv2d layers in the network. This bias
setting is True by default.
4. Use the ReLU activation function on the convolution output.
5. The network uses a MaxPool operation on the convolution layer output (after activation),
along the sentence length dimension to reduce the amount of output produced from each
kernel. That is, it computes the maximum across the entire sentence length, to obtain one
output feature/number from each sentence for each kernel.
6. Concatenate the outputs from the maxpool operations above to form a vector of dimension
n1 + n2 – be sure you correctly construct the shape of the resulting tensor.
7. Finally, similar to the baseline architecture, use a fully connected layer to produce a scalar
output with Sigmoid activation to represent the probability that the sentence is in the subjective class. Note that the BCEwithLogitsLoss function computes the sigmoid as part of
the loss; to actually determine the probability when printing out an answer, you’ll need to
separately apply a sigmoid on the neural network output value.
5.1 Overfit
Once you’ve finished coding the model, use the overfit dataset, and the parameters k1 = 2, n1 =
50, k2 = 4, n2 = 50 to make sure that you can overfit the model, as discussed in Section 4.4. Report
the training accuracy that you were able to achieve with the overfit dataset. [1 point]
5.2 Training and Parameter Exploration
Explore the parameter space of the CNN in the following steps, using the full dataset:
1. Here you should explore the normal hyper parameters for neural networks along with the
specific ones in this CNN – k1, n1, k2 and n2. As a suggestion, start with k1 = 2, n1 =
10, k2 = 4, n2 = 10 and select the other hyperparameters. After that, explore different values
of k1, n1, k2, n2 to achieve the best accuracy that you can. Report the accuracy and the full
hyperparameter settings. Give the training and validation curves for that best model, and
describe your overall hyperparmeter tuning approach. [4 points]
2. Re-run your best model, but allow the embeddings to be fine-tuned during the training,
by setting the freeze parameter to False on the nn.Embedding.from_pretrained class.
Report the accuracy of the result, and comment on the result. Save this model in a .pt file
as you did in Section 4.7, for use below in Section 6. [2 points]
5.3 Extracting Meaning From Kernels
Observe that one of the kernel’s dimension is the same size as the word embedding. Since the
kernels are trained to learn values that should be present (or not present) in the input, they have
an interpretable meaning, as was the case in Section 4.6.
You can explore that meaning using the
function print_closest_cosine_words from Assignment 1. Use that function to determine the
five closest words to each of the words in the the kernels trained in your best classifier. Do those
words make sense? Do the set of words in each given kernel give a broader insight into what the
model is looking for? Explain. [4 points]
5.4 Submit CNN Code
Submit your full code for this section in either in a notebook file named A2_CNN.ipynb or in a
zip file containing all your python files named A2_CNN.zip. Your code should clearly state how it
should be run, and it should have easy-to-use arguments that allow any part of this Section to be
run. [4 points]
6 Web-based User Interface to Classify Input Sentences Using
Gradio (5 points)
In this section, you will write code that interacts with your best model. You will use the Gradio
software to build this user interface, as Gradio may come in handy during the course project. As
shown in class, your Gradio interface to the model should provide a box for an input sentence,
through a web page, and display the classification from each of the two models (your baseline and
the best cnn), and well as the output “probability” that this sentence is subjective.
First, install the library and read about the basics of Gradio here: https://gradio.app/getting_
started/.
Then, write a Gradio-based python script/notebook that takes in a sentence, and outputs the result of your best Baseline and CNN networks (that you saved in a .pt file, as described
in Sections 5.2 (part 5) and 4.7, and reload in a separate script/notebook. This output should be
two things: the subjective/objective classification and the associated output probability for each
of the two models.
Your code should process the input string in the following way, to create a tensor that can be input
into the two models:
1. Load the Vocab object as in Section 3.3 using the torchtext.vocab.GloVe object.
2. Load the saved parameters for models you’ve trained:
checkpoint = torch.load(‘filename.pt’)
model = torch.load_state_dict(checkpoint)
3. Using the code given below, split the input string into separate words, and convert the
individual words into their tokens, and then into a tensor:
tokens = sentence.split()
# Convert to integer representation per token
token_ints = [glove.stoi.get(tok, len(glove.stoi)-1) for tok in tokens]
# Convert into a tensor of the shape accepted by the models
token_tensor = torch.LongTensor(token_ints).view(-1,1)
4. Pass this tensor through your models, and display, using gradio, the prediction results.
6.1 Run and Compare
Run your two best stored models on 4 sentences that you come up with yourself, where two of the
sentences are definitely objective/subjective, and the other two are borderline subjective/objective,
according to your opinion. Include the input and output in your write up. Comment on
how the two models performed and whether they are behaving as you expected. Do they agree
with each other? Which model seems to be performing the best? [1 point]
6.2 Submit Gradio Code
Submit your full code for this section in either in a notebook file named A2_Gradio.ipynb or in a
zip file containing all your python files named A2_Gradio.zip. In addition, submit the two model
files that you used for this section in files named baseline.pt and cnn.pt. Your code should
clearly state how it should be run, and it should have easy-to-use arguments that allow any part
of this Section to be run. [4 points]
References
[1] Internet movie database. https://www.imdb.com/. Accessed: 2022-09-25.
[2] Movie review data. https://www.cs.cornell.edu/people/pabo/movie-review-data/. Accessed: 2022-09-25.
[3] Rotten tomatoes. https://www.rottentomatoes.com/. Accessed: 2022-09-25.
[4] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751.
Association for Computational Linguistics, 2014.
[5] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity. In
Proceedings of ACL, pages 271–278, 2004.
[6] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for
word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages
1532–1543, 2014.
11
Assignment 3: Training a Transformer Language Model and using
it for Classification
The goal of this assignment is to explore the use of Transformers as language models, and as
classifiers. You will be using code that has already been written, and so a big part of this assignment
will connecting the given code to the conceptual material from the lectures. This assignment must
be done individually.
The specific learning objectives in this assignment are:
1. Understand the language modeling objective.
2. Become familiar with a PyTorch code implementation of a Transformer model.
3. Pre-train the model using the language modeling objective (predicting the next word of a
sequence), first with a small corpus, then a larger corpus.
4. Observe the output probabilities of the pre-trained model on a few examples.
5. Fine-tune the pre-trained transformer model for sentiment classification.
6. Become familiar with the Huggingface model hub, and how to fine-tune large pre-existing
models with the Huggingface trainer class.
What To Submit
You should hand in the following files to this assignment on Quercus:
• A PDF file assignment3.pdf containing answers to the written questions in this assignment.
You should number your answer to each question in the form Question X.Y.Z, where X is
the section of this assignment, Y is the subsection, and Z is the numbered element in the
question. You should include the specific written question itself and then provide
your answer. Grades will be deducted if this format is not followed.
• Code files as specified in some individual questions.
1 Karpathy’s minGPT [12 pts]
We will make use of Andrej Karpathy’s code repository of the GPT-style Transformer code, which
is a simplified PyTorch implementation of GPT-2 [1] made more readable, but less powerful, as
Karpathy illustrates in Figure 1. A revised version of this code is provided in the assignment zip
file which builds and trains a full language model. Below we give an overview of the code that is
provided. This, together with the recent lectures, will help you answer the questions relating to
the code below.
Figure 1: Karpathy’s View of His Code vs. Others
1.1 Structure of Provided minGPT Code
In the zip file associated with this assignment, you will find a notebook LM.ipynb which imports
several python files from the folder mingpt. Below we explain the role of each of this notebook and
python code files. You may choose to convert your code into only python files, or perhaps one very
large notebook.
• LM.ipynb This notebook sets up the data used for training, sets the key hyperparameters
and performs the training. After storing the trained model in a file, it runs the ‘generate’
function from models.py to predict the words that come after a few starter words.
• mingpt/bpe.py This file contains the tokenizer that is used by minGPT, and most Transformers, in an approach called ‘byte-pair encoding.’
This method is described in the Jurafsky
[2] textbook, Section 2.4.3. You will not need to look too deeply at this code, except to note
that the vocabulary is actually downloaded the first time you use this code, and you can
review in your home folder ~/.cache/mingpt, where ~ is your home folder. One surprising
thing I found was that some words are repeated, with a special character in front in the
repetition.
• mingpt/model.py This file contains the transformer model definition, built up carefully on
a set of sub-modules, include GELU, CausalSelfAttention, Block, and finally GPT. It also
contains the generate function which will predict a sequence of words following one or more
seed words.
• mingpt/trainer.py This contains the code to train the model. It also has a method for
setting up the default hyperparameters of the training/network.
• mingpt/utils.py This file has some utility functions mostly related to setting up the configuration parameters/hyperparameters.
1.2 Questions
Read through all of the code to get a feel for it. This section helps you take in and learn the code
by asking you to find parts of code, or explain it, or to relate the code to the material in the lectures
of this course.
1. Which class does LanguageModelingDataset inherit from? (1 point)
2. What does the function lm collate fn do? Explain the structure of the data that results
when it is called. (2 points)
3. Looking the notebook block [6], (with comment “Print out an example of the data”) what
does this tell you about the relationship between the input (X) and output (Y) that is sent
the model for training? (1 point)
4. Given one such X,Y pair, how many different training examples does it produce? (1 point)
5. In the generate function in the file model.py what is the default method for how the generated word is chosen – i.e. based on the model output probabilities? (1 point)
6. What are the two kinds of heads that model.py can put on to the transformer model?
Show (reproduce) all the lines of code that implement this functionality and indicate which
method(s) they come from. (2 points)
7. How are the word embeddings initialized prior to training? (1 point)
8. What is the name of the object that contains the positional embeddings? (1 point)
9. How are the positional embeddings initialized prior to training? (1 point)
10. Which module and method implement the skip connections in the transformer block? Give
the line(s) of code that implement this code. (1 point)
2 Training and using a language model on a Small Corpus [12 pts]
After your review of the code, you’re ready to train the model on the file smallsimplecorpus.txt
from Assignment 1. The code as given in the notebook will train the model using that file as input.
1. Run the code up to the line trainer.run() and make sure it functions. Report the value of
the loss. (1 point)
2. Run the two code snippets following the training that calls the generate function. What is
the output for each? Why does the the latter parts of the generation not make sense? (2
points)
3. Modify the generate function so that it outputs the probability of each generated word.
Show the output along with these probabilities for the two examples, and then one of your
own choosing. (1 point)
4. Modify the generate function, again, so that it outputs, along with each word, the words
that were the 6-most probable (the 6 highest probabilities) at each word output. Show the
result in a table that gives all six words, along with their probabilities, in each column of the
table. The number of columns in the table is the total number of generated words. For the
first two words generated, explain if the probabilities in the table make sense, given the input
corpus. (5 points)
5. Submit your code for generate in the file A3_2.5.py. (3 points)
3 Training on the Bigger Corpus and Fine-Tuning Classification
[8 points]
In this section you will train the same model on a larger corpus, and then take the trained model
and turn it into a classifier. You will then fine-tune the classifier to do sentiment analysis.
1. Modify the notebook to train on the larger corpus, by un-commenting the other two dataset
lines as described in the comment at the end of the block [4] of LM.ipynb. This dataset needs
significantly more training, and as suggested in the notebook, set the number of iterations
(train config.max iters) to be 100,000 and the batch size to 16. Run the training; it will
take significantly longer.
If you’re using google colab, you should be able to make the training
run faster by switching the runtime type to be a GPU. You will have access to faster GPUs
if you use Google Colab Pro. This code will automatically use the GPU.
If you are short on time, you can instead just load a model that has already been trained from
the saved model file here. Report which of these two methods you used – trained yourself, or
loaded the saved model.
2. Check that this model can generate words by seeding the generate function with a few examples different from the ones given. Report the examples you used and the generation results,
and comment on the quality of the sentences. If you trained your own model, be sure to save
it for re-use in the fine-tuning process below. [2 points]
3. Next, the goal is to convert this trained model into a classifier, and fine-tune it on another
dataset to perform a new task. The classifier will be trained to determine the sentiment of a
sequence of words predicting whether the sequence is positive sentiment (label 1) or negative
(label 0). We will use the Stanford Sentiment Treebank dataset (sst2), which is part of the
General Language Understanding Dataset (referred to as glue) provided by Huggingface.
You’ll need to install the datasets library from Huggingface using: pip install datasets.
The command datasets.load dataset(“glue”, “sst2”) loads the Stanford Sentiment Treebank (binary classification) dataset from the Huggingface server. Use the first 1200 samples
from the train split, and then use the train test split method in scikit-learn to split them into
train and validation sets. Be sure to stratify the splits.
Report the training and validation curves for the fine-tuning, and the accuracy achieved on
the validation dataset. Submit your new code – the new notebook and all the .py files in a
zip file named A3_3_2.zip. [6 points]
Important: If you are able to write the code given to perform this classification task using
your own knowledge, just go ahead and do that. Below we give some general guidance on
what to do, but do not take this as a specification of what must be done. (Here I’m trying
to avoid the questions on Piazza ‘do we have to do it this way?’ The answer to that question
is no, as there are many ways to get this task done.
With that, here is the guidance:
• Create a new Dataset object (not to be confused with the very-similar-looking lower case
Huggingface library dataset) similar to the LanguageModelingDataset in the provided
code. This object should download the dataset as follows:
import datasets
sst = datasets.load_dataset(’glue’, ’sst2’)
You will need to create a training and validation split as indicated above.
• You will need to make use of the code in the given model that uses a classifier head rather
than a language modelling head to the model. As you can see in the code, the two heads
are both nn.Linear layers with the same input dimensions, but the output dimension
of the LM head equals the vocabulary size while the output dimension of the classifier
head equals the number of classes (in this assignment, 2). In the forward procedure of
model.py, the code that you were given for the pre-training uses the language modeling
head. The classifier head should be used during fine-tuning.
Note that in the data input shapes are different in the pre-training versus the fine-tuning,
as they are quite different tasks. During the language modeling pre-training, each token
gives rise to a corresponding label at the output – the word to predict. However, for
the classification fine-tuning each input sequence corresponds to only one label at the
output. To make this work, you will need to write a new collate fn that is used as
part of a new trainer object (from class Trainer).
The trainer.py script will need to be modified to allow language modeling training and
fine-tuning training as well.
4 Fine-Tuning Using Huggingface Models and Library [5 pts]
The fine-tuning work of in the previous section involved some complex code that needed to be
customized. Fortunately, there exists a library of models and code that make this process much
easier.
1. Read (and watch) the tutorial from Huggingface on how to use their model hub and their
datasets, here. This tutorial shows you how to both download a pre-trained model and to
fine-tune it, using Huggingface’s AutoModelForSequenceClassification class. (It also shows
how to do a more detailed-level training with lower-level pytorch).
2. Modify that tutorial to implement a fine-tuned sentiment classifier using the same dataset
that was used in Section 3. Rather than the one used in the tutorial, use the smallest version
of a pre-trained GPT2 you can find.
Report the classification accuracy on the validation set. Comment on the performance of this
model: is it better than the model you fine-tuned in the previous section? Submit your code
either as a notebook named A3 4 2.ipynb or a python file with a similar name. [5 points]
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models
are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
[2] Dan Jurafsky and James H. Martin. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition, 3rd edition draft.
https://web.stanford.edu/ jurafsky/slp3/, 2023.
5
Assignment 4: Generation Probability Trees, Prompt Engineering
for Generation & Classification, and Chain of Thought Prompting
The goal of the fourth assignment is to gain familiarity with the the probabilistic nature of language generation with Transformers, and to gain skill in prompt engineering for generation and
classification. This will be of specific help to those doing Class 2 projects. For those doing Class 1
projects, this may tempt to you to sitll make use of a large model (for synthetic data generation,
for example) but it is also an essential set of NLP skills moving forward in your career.
This assignment must be done individually. The specific learning objectives in this assignment are:
1. Learn about the generation (decoding) process and viewing the generation probability tree,
as well as the effect key generation-controlling hyperparameters.
2. Review the prompt engineering methodology from class and practice it – applying it to generation and classification.
3. experimenting with chain-of-thought prompting specifically.
What To Submit
You should hand in the following files to this assignment on Quercus:
• A PDF file assignment4.pdf containing answers to the written questions in this assignment.
You should number your answer to each question in the form Question X.Y.Z, where X is
the section of this assignment, Y is the subsection, and Z is the numbered element in the
question. You should include the specific written question itself and then provide
your answer.
• Code files as specified in individual questions.
1 Exploration of Text Generation Parameters and Probabilities
(12 points)
Review the Huggingface documentation on performing auto-regressive text generation, which you
encountered to a limited extent in Assignment 3, which can be found here. In particular take a
copy of the code titled Multinomial Sampling.
Make sure you can run that code, which downloads a (medium-sized) GPT2 model and uses it to
generate text given the specific input prompt.
The model.generate function (which has the same goal as the generate function you worked with
in Assignment 3) has many parameters, including temperature and top p as described in class.
These parameters influence the sampling of the output probabilities which are used to select each
word in turn in the auto-regressive generation. There are two other parameters to become familiar
with which control how many tokens are generated: max length and stopping criteria. There
are also parameters like repetition penalty which penalize repeated words, and reduce their
occurrence if the penalty is set above 1. You can review all of the parameters of the generate
function here.
Consider the following input sequence: “It is important for all countries to try harder to reduce
carbon emissions because”. In the following steps you are asked to both generate subsequent words
from that input context using that GPT2 model, and to explore the probabilities that are generated.
1. To get a sense of the influence of some of the generation parameters, explore at least 10
combinations of temperature and top p to generate a maximum of 30 tokens using autoregressive generation with the GPT2 model. Comment on how the generation differs across
the range of parameters that you have selected. You must choose your own range, and you’ll
have to do some exploration to do that; you are free to explore other parameters if you wish.
[5 points]
2. Modify the code to output the probabilities of the each word that is generated. You’ll need to
set these two generate parameters:return_dict_in_generate=True and, output_scores=True,
and extract the probabilities that come in the returned dictionary one call at a time. Provide
a table that shows these probabilities, similar to Assignment 3. Comment on the probabilities.
[2 points]
3. Write new code that generates the probability tree (like the one drawn on the board in
Lecture 6 and shown on page 6-5 of the notes in Lecture6 part1) using the treelib package
that you can find here. Generate the tree for the above sequence as input, providing the top 3
probabilities for each word position, as far as is practical to see, and submit that as part of the
answer to this question. (You’ll have to apply some common sense here to visualize the tree).
Comment on the what you see in the tree. Is the tree affected by the top p parameter or the
temperature parameter? Why or why not? Submit your full code that runs the generation
and builds and outputs the tree in the file A4_1_3.py [5 points]
2 Accessing OpenAI, Setting Limits, Learning the API
For the remainder of this assignment, you’ll be using models from OpenAI – through the OpenAI
Playground and also directly using the OpenAI API. If you do not have an OpenAI account already
should sign up for the OpenAI playground at here: https://platform.openai.com/playground. and
you will get a certain amount (I believe $5) of tokens for free. You will need to put in your own
credit card to use the playground, and so the work in this assignment will incur expenses.
However,
the cost per token, even for the most expensive model is 3 to 6 USD cents per 1000 tokens, and
none of the work in the sections below should incur more than $1 USD. That said, you will be
programmatically accessing the OpenAI API, which means you could accidentally put
your code into a loop that could incur significant expenses.
To prevent that from happening, go to playground page, and click your account in the upper right
hand side of the playground and then select Manage Account. Then select Billing from the
right hand side menu, and then select Usage Limits show on the screen. As you can see, there is
a soft limit to the amount of money spent (which generates a warning) and a hard limit that will
cause all requests to be rejected. Set your limits to be $1. Please talk to the instructor if this is
problematic in any way.
For some of your work in the following sections, you are required to use the OpenAI API to access
the models, rathing than the highly manual process of cutting and pasting to the playground. To
learn how to do that, read https://platform.openai.com/docs/introduction and https://platform.
openai.com/docs/quickstart?context=python, and perhaps more information in the documentation
section of OpenAI’s website.
3 Prompt Engineering Methodology
Recall the discussion in Lecture 7 about Prompt Engineering – the design of english-language statements to stimulate a large language model to perform a specific task. The methodology discussed
on page 7-6 of Lecture 7 is reprised below, and improved. The context of the methodology is one you
are familiar with: That you have a task (which could be generation of text or classification of text)
and an input data example that you wish to either generate from, or to classify.
Call that example
a training example, and if it is for classification, then it would come with a correct label. If it is for
generation then it won’t necessarily come with a label, but it might come with a ’correct’ generation.
Assume also that you’ve got 30 training examples, and divide them into 10 examples for ‘training’
as described below, and the remaining 20 as your hold-out test set. One essential insight is that
prompt engineering still requires that there be a training set, and a hold-out test-set, but the good
news is the number needed is much smaller than needed in either in fine-tuning a pre-trained
model, or training a model from scratch.
General Prompt Engineering Method
1. Write down your criterion for what makes the output acceptable, in English. This may involve
looking up the definition of one or more words of the goal, or exploring other online resources
so that you clearly understand it. This applies to both generation and classification.
2. Draft a Prompt that uses that criterion to direct the model to generate what you want.
3. Using just one input example (which in classical machine learning training would be called a
training example) run the prompt and the example to see how well it works, in the OpenAI
playground, on the model.
4. Keep evolving the prompt, using English, to make it work perfectly. This means changing the
prompt to correct anything that is wrong in the output. However, do not use language that is
specific to the input data, as that won’t generalize. Notice that the concept of generalization
will In the prompt engineering world, it will be helpful to cultivate an ability to write clearly,
with good use of language and meaning of words.
5. Once you’ve made it work for one input example, make it work for two, using the same
method – using a prompt with general words, not specific to example, but correcting any
issues seen on the second example. See the example used in Lecture 7 for some insights.
6. Then, try the prompt on five more training input examples all at once, and label the outputs
as good/not good with respect to your criterion. Evolve the prompt to succeed on all five.
7. Test. Run the prompt on your 20 hold-out test set. Measure success rate yourself, i.e. with
human labelling. Report your success rate.
8. If the success rate is below 100% you could choose to iterate once again, adjusting the prompt
to correct the non-successful outcomes.
There are many suggestions online about how to evolve prompts, including the Medium article
survey that was posted with Lecture 7 on Quercus (PromptEngineering Medium.pdf).
4 Prompt Engineering for Generation of Soft, Non-Expert Therapeutic Statements (14 points)
In therapeutic counseling, it is often important to make observations based on a patient’s words
that give insight to the patient. However, it is known that making direct and strong statements
(e.g. “You are addicted to cigarettes”) can be received as accusations or unwanted labels, which
a patient will likely reject.
This is made more problematic if the language used by the therapist
implies that they are superior or in a higher-up position of some kind – either as an expert or
simply their implicit position above the patient in some kind of hierarchy means their words are
correct simply because of that position. (For example, to say “You’re afraid of being judged by
your family,” is an authoritative statement of fact).
It is known that therapy is more effective if such statements are are expressed as possibilities,
not facts, perhaps converted to questions. In addition, statements that suggest the patient knows
better the truth better than the therapist (it is the patient’s life after all) are also better. In
general, encouraging a patient to come to their own conclusion is known to be more successful.
For example, rather than say “You’re afraid of being judged by your family,” a softened statement
might become “I might be wrong, but is it possible that you’re worried that your family will judge
you?” In the latter statement the speaker is both ceding expertise to the patient, and more gently
suggesting a possibility, rather than strongly stating a fact.
This softened statement is part of a broader therapeutic approach called Motivational Interviewing, which if you’re interested, you can read about here: https://www.ncbi.nlm.nih.gov/books/
NBK571068/.
Your task is to engineer a prompt for GPT-4 to convert direct statements to what we will call
softened, non-expert statements.
The dataset given in the file DirectStatements.csv, has a set of examples consisting of statements that are too direct, and are to be softened. These statements came from a variety of places
in behaviour change counseling, generally relating to addictions.
Follow the steps below, which are modelled on the methodology given in Section 3.
1. Write, in your own words (not those above), a clear definition of what it means to convert a
statement into a softened, non-expert version. Report your definition. [2 points]
2. Follow steps 2 through 4 of the methodology of Section 3 using the GPT4 model in the
OpenAI playground to develop a prompt for softening converstion. This means you work
only on the first training example (in row 1) in the dataset, until you are satisfied that the
result is good. Report the prompt that you arrived at in step 4. Produce three different
softened versions of the first example, and say for each why it meets your definition. You
may need to explore different parameters to get differrent results, or maybe just pushing the
button more than once gives a different answer.[4 points]
3. Generate a result on the second item (row 2), and explain how it meets your definition. [1
point]
4. Give the resulting on the next five (rows 3-7), and any changes you make to the prompt to
make them all succeed. [1 point]
5. Using the Open API that you read about in Section 2, (and not the playground) run the
remaining 23 examples and determine, by hand, if they meet your criterion. Submit a csv file
that contains three columns: the first input statement, the second column for the produced
output, and the third column that gives a label that indicates if output meets the criterion
(label 1) or not (label 0). Name that file A4_4_4.csv. Report your resulting success rate,
and which result you think is the best, and which is the worst. Provide the full code that you
used in a python file A4_4_4.py. [6 points]
5 Prompt Engineering for Classification of “Softness” (8 points)
In Lecture 7 (on page 7-5) we also discussed and illustrated a prompt that turned a large language
model into a classifier. In this section you will create a classifier that determines if a statement has
the softness feature described in Section 4.
1. Create a dataset that combines both the input statements from Section 4 and all of your
outputs for all 30 examples, with labels. Assume that all of the inputs in the original file
DirectStatements.csv would have a label 0, and use your assigned label from Part 4 of
Section 4. Submit the full dataset in a file named A4_5_1.csv [1 point].
2. Using the method described in Section 3, and the combined data set you just created, evolve
a prompt using 6 examples (3 negative, 3 positive). Show the prompt and give the success
rate across those 6 examples. DO NOT USE Chain of Thought Prompting, as this will be
the subject of Section 6. [3 points]
3. Using the OpenAI API, run the remaining examples and give the success rate. Attempt to
explain any incorrect results. [4 points]
6 Chain of Thought Prompting (5 points)
The chain-of-thought prompting method was also discussed in Lecture 7, and in the post from
Medium and the paper https://arxiv.org/pdf/2201.11903.pdf. There are two advantages of this
approach: it produces a higher accuracy, as the model traverses a ‘better’ path. Then, the explanation itself is of value, which we explore by revisiting the above classifier.
1. Using the same data set you created for Section 5, modify your classification prompt to elicit
the chain of thought reasoning. Evolve your prompt to achieve good explanations/chains of
thought. Report what your prompt is. [1 point]
2. Run your prompt on all of the example inputs from the dataset. Report on the accuracy,
and state whether it is different from the accuracy you achieved in Section 5.Choose the best
explanation you see across your dataset, and the worst one. Report each of these, and say
what is good/bad about each. [4 points]
5