Description
1 Overview
In this assignment, you will train a recurrent neural network for the task of sentiment analysis on natural language data. More specifically, we will analyze data from Twitter and learn to classify it as either ”positive” or ”negative”. For those who don’t know, Twitter is an online news and social networking site based on communication between users via ”tweets,” which are messages in natural language limited to 140 characters. These tweets can be viewed as sequences of words in natural language, and will form the sequential input to our RNN model. The following guide will take you through the downloading a preprocessed version of a sentiment dataset and creating a model in TensorFlow.
1.1 Sentiment Analysis
Sentiment analysis refers to the natural language processing task of classifying some collection of text by its polarity, i.e whether or not the text has a ”positive”, ”negative”, or ”neutral” attitude. The goal is to understand the attitude of the agent that generated the text. It can also attempt to assign a collection of text to more refined emotional states, such as ”angry”, ”happy”, or ”sad”. In our case we will be looking only at positive and negative polarity and classifying at the tweet level.
2 Dataset
Our dataset can be downloaded here. If you are interested in the dataset from which this originates, see section 2.1. Please read section 2.2 to get an
1
idea for the techniques used in preprocessing and to understand the dataset in its current form. There is also a question in section 2.2 which will need to be answered in the writeup. Please also download the associated vocabulary json here. You can also download the ”reverse vocabulary” json here. You need only download the reverse vocabulary if you want to ”decode” your tweets (go from an integer representation back to a string representation). More on this later.
2.1 Original Dataset
Our dataset comes from the Sentiment140 training set. This data contains 1.6 million tweets, classified as either positive or negative, that have been harvested by searching Twitter for emoticons. Tweets containing happy emoticons were naively labeled as positive, and those containing sad emoticons were labeled as negative. The emoticons were then removed from the tweets so that classifiers must rely on natural language features for classification. In addition to a column containing the text of the tweet, the original dataset also contains a column indicating the polarity of the tweets (0 for negative, 4 for positive), the time of the tweet, the associated Twitter users, a timestamp, and a flag indicating the query type used to get the data. We will not use any of these additional columns in our assignment except the labels. The original dataset is available for download here, but DO NOT USE THIS DATASET FOR THIS ASSIGNMENT. Instead we will be working with a pruned and preprocessed version of the dataset.
2.2 Preprocessing
For this assignment the data preprocessing has been handled for you. There are a number of potential problems with handling the raw CSV file containing tweets in string form. First let’s take a look at the trivial preprocessing that was done.
• Unnecessary columns were removed, leaving only tweets and associated labels. • Labels were converted from 0 and 4 to 0 and 1 in order to work more nicely with our binary classification setup. The reason that 0 and 4 were initially present in the data is the task was originally multiclass classification. Tweets were rated on a scale from 0 to 4, with 0 being most negative and 4 being most positive.
2
• Urls and links were removed. • Twitter handles were removed from tweets. When constructing tweets, users frequently start by addressing another user by their username, also known as a handle. We do not need to know who a user was addressing to classify a tweet. • Empty tweets were deleted A number of other proprocessing steps were taken in order to use a RNN on this data, some of which may be less intuitive. These steps are listed below, and might give you some insight into considerations that are made when undertaking a natural language processing task.
• Tokenization: In order to turn a string into a sequence of words, tokenization must be performed. Tokenization is the process of splitting natural language text into its discrete parts. For this assignment, each tweet was split on whitespace, which is a very simple approach. In more advanced applications, sophisticated tokenizers like those provided by Python’s NLTK can be used. To see more information about NLTK and tokenization, see here. • Removing Punctuation: Because of our naive tokenization process, many generated tokens(words) will contain trailing commas, question marks, etc. We can make an assumption that punctuation will not be helpful in determining the sentiment of a tweet, since both positive and negative tweets ought to use them equally. Note that this may not be true in practice, but is true enough for our purposes. Therefore, we can remove all punctuation and be left with words alone. If we wanted to use punctuation, a more sophisticated tokenization method would be necessary. • Lowercasing Words: All words were converted to lowercase. This was done to ensure that the RNN treats a word the same no matter what case a user used when typing it. Were words left in their original cases, many copies of the same word might appear in the vocabulary, e.g Sad, sad, SAD, etc. This is probably undesirable, though different cases of the same word may indicate different meanings. However, as we will see, limiting vocabulary size has more computational benefits than drawbacks.
3
• Removing ”uncommon” words: We remove words which appear less than 100 times in order to limit the amount of training data for this exercise, and also to control the vocabulary size. Many words which appear infrequently are typos or potentially just gibberish, and cost more to keep track of than they are worth in terms of improvements in accuracy. One way to include some information from uncommon words and still control vocabulary size is to use a special ”UNKNOWN” token to represent all words not in the vocabulary. We do not do this in this assignment. Rather, we simply remove tweets from the dataset which have words outside our vocabulary. • Strings to Ints: Now that all words have been normalized and we have settled on our vocabulary, each word can be represented by a unique integer. We will feed our RNN sequences of integers which represent words. This is necessary in order for the RNN to turn each word into an input vector, a process known as word embedding. The map from word to integer can be downloaded here, though this will be unnecessary unless you wish to re-use this trained model on your own tweets. However, the map from integer to word will be necessary and can be downloaded here. • Removing long tweets: In order to reduce the computational burden on you, we have removed tweets that are above 25 tokens long. This helps reduce the number of operations performed and memory use. Further, classification and other learning tasks become harder as inputs to an RNN become longer, thanks to the exploding and vanishing gradient problems. • Fixing sequence length and creating masks: In order for the RNN to accept our sequential inputs, we need to have a fixed number of tokens per sequence. Without knowing the number of inputs, we couldn’t declare the shape of the input tensor. However, tweets are clearly of varying length. In order to overcome this apparent conundrum, we determine the maximum length of a tweet in our dataset. This is 25 for our particular assignment, thanks the the previous preprocessing step. If a particular sequence is shorter than this maximum length, then we simply append dummy words to this sequence until it is the maximum length. At the same time, we construct a 25 entry long ”mask” for this sequence indicating which values are real and which values are dummy values to be ignored. At each index in this mask, a value of 1 indicates a ”real” entry, while a value of 0 indicates a
4
dummy entry. We will see how to apply this mask later.
2.3 A Word on Word Embeddings
We briefly touched on the process of word embedding, or converting a single word into an input vector for the RNN to work with. This embedding, or word-feature-vector, will be learned, similar to how the filters of a CNN are learned. Each word’s corresponding embedding is stored in a matrix of shape (vocabulary size, embedding size), where embedding size determines the length of the word-feature-vector. Without our previous preprocessing steps to limit the vocabulary, you could end up with a vocabulary size of up to 500000. In your assignment write-up, please include the calculations for how much memory a vocabulary size of 500000 would cost given an embedding size of 300 and using double precision floating point numbers. Give your answer in megabytes. You will see why modeling this large a vocabulary is a problem, as the embedding matrix is a single layer in our recurrent network.
2.4 Our Dataset
Our code applies all of the above preprocessing techniques to the original data, and saves the resulting 500000 tweets into a training, validation, and testing dataset. The training data contains 400000 tweets of fixed length and includes masks for each tweet. The validation data contains 50000 tweets also with masks. We reserve a test set of 50000 tweets for evaluating your model. After preprocessing, our vocabulary size is 8745. The training and evaluation datasets are compressed into one npz file which can be downloaded here. In order to unpack the npz file, use the following code (Figure 1), which should work for both python 2 and python 3. Next we will construct our model and see how to feed it our data.
3 Model
We will construct an RNN for this problem. Recall that an RNN accepts a sequence of inputs and produces a sequence of outputs as in Figure 2. Formally, our task is a binary classification task over the entire input sequence, so we need only consider the output of the RNN after it has received the final ”real” word (not the dummy words). Note that this will be a function of all ”real” input values in the sequence.
5
import numpy as np npzfile = np.load(“train_and_val.npz”) train_x = npzfile[“train_x”] train_y = npzfile[“train_y”] train_mask = npzfile[“train_mask”] #Validation filenames follow the same pattern val_x = npzfile[“val_x”] # etc …
Figure 1: Python code indicating how to read in the dataset for this assignment
Figure 2: An example RNN producing output at multiple timesteps.
3.1 Model Design
The general architecture will be as outlined in Figure 3. Our sequential input tensor of shape (batch size, max sequence length) will be fed into an embedding matrix (see section 3.2) of shape (vocabulary size, word embedding size). The choice of word embedding size is left up to you. With the word embedding done, the sequence of word embedding vectors is fed into the recurrent neural network cell. This cell can be a vanilla RNN, an LSTM, a Gated Recurrent Unit (GRU), or whatever recurrent object you want. The output of this RNN cell will be determined by the mask (see section 3.4), and will be multiplied by an output matrix to produce a single logit. You will then use the cross entropy loss to update your network parameters. I recommend using the tf.nn.sigmoid cross entropy with logits() function. Recall you must use this loss function on the unactivated output of that single final network node, i.e without applying a sigmoid first.
In terms of performance, shoot for 84-85% validation accuracy. You should be able to reach 80% quickly, i.e with one pass over all of the data. Please track your loss, training accuracy, and validation accuracy, and plot them. An example is shown in Figure 3. However, please generate a data
6
point after every n iterations as opposed to every epoch. 85% accuracy is very good for sentiment analysis tasks, though it is difficult to compare the results here to other results since we so carefully curate our data.
Figure 3: Our architecture.
Figure 4: Example loss plot. The losses were examined after each epoch. Please provide a more detailed plot by examining after every n iterations, where the choice of n is up to you.
3.2 Word Embeddings
In order to construct your word embedding layer, you must first declare a tensor variable of shape (vocab size, word embedding size). Exactly how to initialize this embedding matrix is up to you, simply keep in mind principles
7
of initialization that have been previously discussed in class. The output of a word embedding layer should pass onto the next layer (the RNN cell) without going through an activation function. This is because the word embedding can be considered an input to the network itself rather than a layer to be activated, since each embedding is supposed to be a representation of its corresponding word.
TensorFlow has an efficient built in function for generating an embedding of a particular word. Given a word embedding matrix w embed, simply run
rnn_input = tf.nn.embedding_lookup(w_embed, sequence_placeholder)
As the name suggests, rnn input, a tensor of shape (batch size, max sequence length, word embedding size), will then be fed into the RNN cell.
3.3 Layer Choice
Tensorflow provides access to a number of recurrent layers to choose from. Probably the most popular is the LSTM cell, which can be constructed in one line in tensorflow. See the tf.contrib.rnn.LSTMCell() function for details. This cell will accept the output from the embedding layer. Other popular recurrent cell choices can be found here, under the section ”Core RNN Cells for use with TensorFlow’s core RNN methods.” Each of these cells can be used with the tf.nn.dynamic rnn() operation, which will handle the feeding of the input sequence (the word embedding one) to your choice of RNN cell and generate outputs at each timestep. You can also use dropout with these cells by applying the tf.contrib.rnn.DropoutWrapper() to the cell.
More precisely, the dynamic rnn function generates an output, state pair, though we will only consider the output portion. Remember that the rnn generates an output for each input in the sequence, even the dummy ones. The output of the dynamic rnn operation will be a tensor of shape (batch size, max len, cell size). We only want the last relevant output, i.e no outputs that have been generated using dummy values. This is where the masking stage come in.
3.4 Masking
In order to select the last relevant output of the dynamic rnn cell, we take the output tensor of shape (batch size, max len, cell size) and perform the following series of operations on it (not necessarily the batch size and max
8
length computations if you have access to them somewhere else already).
length = tf.cast(tf.reduce_sum(mask_placeholder, reduction_indices=1), tf.int32) batch_size = tf.shape(output)[0] max_length = tf.shape(output)[1] out_size = int(output.get_shape()[2]) flat = tf.reshape(output, [-1, out_size]) index = tf.range(0, batch_size) * max_length + (length – 1) relevant = tf.gather(flat, index)
Here the true length of the sequence is computed by summing the number of ones in the mask data. Once this length has been determined, the output tensor is flattened to shape (total inputs, out size). An index into this tensor is constructed, first generating the start indices for each training example tf.range(0, batch size) * max length, and then adding the individual sequence lengths to those start indices. This link has more coverage of masking in TensorFlow.
Now that you have your relevant output tensor of shape (batch size, rnn cell size), you can construct an output weight tensor of shape (rnn cell size, 1), along with its corresponding bias. This produces your single output logit (per batch example) which you will then feed to your loss.
3.5 Hyperparameters and Optimization
The choices of initializations, regularization, optimizer, and other hyperparameters are left completely up to you. Recall previous principles we have discussed in the class and you should be fine. One tip: adaptive learning rate optimizers tend to work better with RNNs than plain SGD.
3.6 Computational Time
On a Lenovo t440s with 8GB of RAM, using a batch size of 1000, one iteration takes about 3.5 seconds. 10 epochs of training would then take about 3.5 hours. You may find, however, that fewer than 10 epochs are necessary to reach the target validation accuracy. I reached target accuracy in 3 epochs before I began overfitting. If you run into memory problems, please let us know. I don’t think this should be a problem unless you have under 4GB of RAM.
9
3.7 Saving
In order to run your model for grading, we will need access to your prediction operation, sequence placeholder, and mask placeholder. Your prediction operation should produce a row vector of predicted classifications for a collection of input sequences. I suggest using tf.round on the output sigmoid of your model. Please add these to a collection titled ”validation nodes” and save using the model name ”my model.” One example of doing so is pictured in Figure 3.
tf.get_collection(“validation_nodes”) tf.add_to_collection(“validation_nodes”, sequence_placeholder) tf.add_to_collection(“validation_nodes”, mask_placeholder) tf.add_to_collection(“validation_nodes”, predict_op) saver = tf.train.Saver() save_path = saver.save(sess, “my_model”)
Figure 5: Python code indicating how to read in the dataset for this assignment
3.8 Word Vector Visualization
We have briefly touched on word embeddings and the purpose of learning them from discrete inputs. Once you have trained your recurrent network on the given data, your word embedding matrix will no longer be a random matrix, but rather a matrix in which each row contains a representation of the corresponding word in a vector space. The learned representation can give a sense of the meanings the model has learned to attribute to each word. In good embeddings, words with similar meanings will often have vectors close to one another in the embedding space. Such relationships can be visualized using dimensionality reduction, as we will do here.
For this assignment, we will use the t-distributed Stochastic Neighbor Embedding algorithm to visualize the our embeddings t-SNE for short. t-SNE . As described in Wikipedia t-SNE ”is a nonlinear dimensionality reduction technique that is particularly well-suited for embedding highdimensional data into a space of two or three dimensions, which can then be visualized in a scatter plot. Specifically, it models each high-dimensional object by a two- or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by
10
distant points.” An introduction for interested readers can be viewed here.
For our purposes, t-SNE will simply be a nice way of visualizing our learned word embedding vectors. Please install the sklearn package e.g via pip install sklearn. This package contains an implementation of t-SNE that we will use.
Figure 6: An example t-SNE visualization of some learned word embeddings from Turian et al. 2010. Notice how words with similar functions, the days, months, and years, are grouped together.
We are going to visualize a simple set of words that should give us pretty clear groupings: fruits and days of the week. We will be using the vocab.json file we downloaded earlier.
import json with open(“vocab.json”, “r”) as f: vocab = json.load(f) s = [“monday”, “tuesday”, “wednesday”, “thursday”, “friday”, “saturday”, “sunday”, “orange”, “apple”, “banana”, “mango”, “pineapple”, “cherry”, “fruit”] words = [(i, vocab[i]) for i in s]
Figure 7: Python code indicating how to generate common words in our vocabulary, along with their indices.
With the top 100 words generated, we can now select the corresponding vectors from the embedding matrix and embed them in our two dimensional
11
space for visualizing.
from sklearn.manifold import TSNE model = TSNE(n_components=2, random_state=0) #Note that the following line might use a good chunk of RAM tsne_embedding = model.fit_transform(word_embedding_matrix) words_vectors = tsne_embedding[np.array([item[1][0] for item in words])]
Figure 8: Python code indicating how to train a t-SNE embedding on our common words
Now that you have an array containing all our example words embedded in a two dimensional space, you can create a scatter plot of these points like Figure 5 and 9. See, for example, this stackoverflow question showing how to annotate a scatter plot with text. The tsne embedding array preserves the order of points, so your words vectors word order will match that of the tsne embedding array. You should be able to get a fairly clean grouping in this collection of words. See Figure 9 for an example.
4 Submission
To recap, you must submit for this assignment
• Your memory calculation (see Section 2.3). • Your graph collection containing the input and prediction nodes (see Section 3.7). • Your loss and accuracy plots (see Section 3.1). • Your visualization of word vectors (see Section 3.8). You can validate that your graph is OK by using the script at a link to be uploaded soon. If you saved your collection as ”my model”, simply run python validate.py my model.
12
Figure 9: An example t-SNE visualization of the word list given