## Description

1 Softmax (10 points)

(a) (5 points) Prove that softmax is invariant to constant osets in the input, that is, for any input vector

x and any constant c,

softmax(x) = softmax(x + c)

where x + c means adding the constant c to every dimension of x. Remember that

softmax(x)i =

exi

P

j exj

Note: In practice, we make use of this property and choose c = ?maxi xi when computing softmax

probabilities for numerical stability (i.e. subtracting its maximum element from all elements of x).

(b) (5 points) Given an input matrix of N rows and d columns, compute the softmax prediction for each row.

Write your implementation in q1 softmax.py. You may test by executing python q1 softmax.py.

Note: The provided tests are not exhaustive. Later parts of the assignment will reference this code so it is

important to have a correct implementation. Your implementation should also be ecient and vectorized

whenever possible. A non-vectorized implementation will not receive full credit!

2 Neural Network Basics (30 points)

(a) (3 points) Derive the gradients of the sigmoid function and show that it can be rewritten as a function

of the function value (i.e. in some expression where only (x), but not x, is present). Assume that the

input x is a scalar for this question. Recall, the sigmoid function is

(x) =

1

1 + e?x

(b) (3 points) Derive the gradient with regard to the inputs of a softmax function when cross entropy loss

is used for evaluation, i.e. nd the gradients with respect to the softmax input vector , when the

prediction is made by ^y = softmax(). Remember the cross entropy function is

CE(y; ^y) = ?

X

i

yi log(^yi)

1

CS 224d: Assignment #1

where y is the one-hot label vector, and ^y is the predicted probability vector for all classes. (Hint: you

might want to consider the fact many elements of y are zeros, and assume that only the k-th dimension

of y is one.)

(c) (6 points) Derive the gradients with respect to the inputs x to an one-hidden-layer neural network (that

is, nd @J

@x where J is the cost function for the neural network). The neural network employs sigmoid

activation function for the hidden layer, and softmax for the output layer. Assume the one-hot label

vector is y, and cross entropy cost is used. (feel free to use 0(x) as the shorthand for sigmoid gradient,

and feel free to dene any variables whenever you see t)

x

h

^y

Recall that the forward propagation is as follows

h = sigmoid(xW1 + b1) ^y = softmax(hW2 + b2)

Note that here we’re assuming that the input vector (thus the hidden variables and output probabilities)

is a row vector to be consistent with the programming assignment. When we apply the sigmoid function

to a vector, we are applying it to each of the elements of that vector. Wi and bi (i = 1; 2) are the weights

and biases, respectively, of the two layers.

(d) (2 points) How many parameters are there in this neural network, assuming the input is Dx-dimensional,

the output is Dy-dimensional, and there are H hidden units?

(e) (4 points) Fill in the implementation for the sigmoid activation function and its gradient in q2 sigmoid.py.

Test your implementation using python q2 sigmoid.py. Again, thoroughly test your code as the pro-

vided tests may not be exhaustive.

(f) (4 points) To make debugging easier, we will now implement a gradient checker. Fill in the implementa-

tion for gradcheck naive in q2 gradcheck.py. Test your code using python q2 gradcheck.py.

(g) (8 points) Now, implement the forward and backward passes for a neural network with one sigmoid

hidden layer. Fill in your implementation in q2 neural.py. Sanity check your implementation with

q2 neural.py.

3 word2vec (40 points + 5 bonus)

(a) (3 points) Assume you are given a predicted word vector ^r (vwI in the case of skip-gram or the sum of

vw’s of the context words for CBOW), and word prediction is made with the softmax function found in

word2vec models

^yi = Pr(wi j ^r;uw1:::jV j ) =

exp(uw

i ^r)

PjV j

j=1 exp(uw

j ^r)

where wj denotes the j-th word and uwj (j = 1; : : : ; jV j) are the \output” word vectors for all words in

the vocabulary. Assume cross entropy cost is applied to this prediction and word i is the expected word

(the i-th element of the one-hot label vector is one), derive the gradients with respect to ^r.

Page 2 of 4

CS 224d: Assignment #1

Hint: It will be helpful to use notation from question 2. For instance, letting ^y be the vector of softmax

predictions for every word, y as the expected word vector, and the loss function

Jsoftmax?CE(wi; ^r;uw1:::jV j ) = CE(y; ^y)

(b) (3 points) In the previous problem, derive gradients for the \output” word vectors uwj ‘s (including uwi ).

(c) (6 points) Repeat part (a) and (b) assuming we are using the negative sampling loss for the predicted

vector ^r, and the expected output word is wi. Assume that K negative samples (words) are drawn, and

they are w1; ;wK, respectively for simplicity of notation (i =2 f1; : : : ;Kg). Again, for a given word,

wo, denote its output vector as uwo . The negative sampling loss function in this case is

Jneg?sample(wi; ^r;uwi ;uw1:::K ) = ?log((uw

i ^r)) ?

XK

k=1

log((?uw

k ^r))

where () is the sigmoid function.

After you’ve done this, describe with one sentence why this cost function is much more ecient to

compute than the softmax-CE loss (you could provide a speed-up ratio, i.e. the runtime of the softmax-

CE loss divided by the runtime of the negative sampling loss).

Note: the cost function here is the negative of what Mikolov et al had in their original paper, because we

are doing a minimization instead of maximization in our code.

(d) (8 points) Derive gradients for all of the word vectors for skip-gram and CBOW given the previous parts

and given a set of context words [wordi?C; : : : ;wordi?1;wordi;wordi+1; : : : ;wordi+C], where C is the

context size. Denote the \input” and \output” word vectors for wordk as vwk and uwk respectively.

Hint: feel free to use F(wo; ^r) (where wo is the expected word) as a placeholder for the Jsoftmax?CE(wo; ^r; :::)

or Jneg?sample(wo; ^r; :::) cost functions in this part | you’ll see that this is a useful abstraction for the

coding part. That is, your solution may contain terms of the form @F(wo;^r)

@::: .

Recall that for skip-gram, the cost for a context is

Jskip-gram(wordi?C:::i+C) =

X

?CjC;j6=0

F(wi+j ; vwi )

For (a simpler variant of) CBOW, we sum up the input word vectors in the context

^r =

X

?CjC;j6=0

vwi+j

then the CBOW cost is

JCBOW(wordi?C:::i+C) = F(wi; ^r)

(e) (12 points) In this part you will implement the word2vec models and train your own word vectors

with stochastic gradient descent (SGD). First, write a helper function to normalize rows of a matrix in

q3 word2vec.py. In the same le, ll in the implementation for the softmax and negative sampling cost

and gradient functions. Then, ll in the implementation of the cost and gradient functions for the skip-

gram model. When you are done, test your implementation by running python q3 word2vec.py.

Note: If you choose not to implement CBOW (part h), simply remove the NotImplementedError so that

your tests will complete.

(f) (4 points) Complete the implementation for your SGD optimizer in q3 sgd.py. Test your implemen-

tation by running python q3 sgd.py.

Page 3 of 4

CS 224d: Assignment #1

(g) (4 points) Show time! Now we are going to load some real data and train word vectors with everything

you just implemented! We are going to use the Stanford Sentiment Treebank (SST) dataset to train

word vectors, and later apply them to a simple sentiment analysis task. There is no additional code to

write for this part; just run python q3 run.py.

Note: The training process may take a long time depending on the eciency of your implementation

(an ecient implementation takes approximately an hour). Plan accordingly!

When the script nishes, a visualization for your word vectors will appear. It will also be saved as

q3 word vectors.png in your project directory. Include the plot in your homework write up.

Brie y explain in at most three sentences what you see in the plot.

(h) Extra credit (5 points) Implement the CBOW model in q3 word2vec.py. Note: This part is optional

but the gradient derivations for CBOW in part (d) are not!.

4 Sentiment Analysis (20 points)

Now, with the word vectors you trained, we are going to perform a simple sentiment analysis. For each

sentence in the Stanford Sentiment Treebank dataset, we are going to use the of all the word vectors in that

sentence as its feature, and try to predict the sentiment level of the said sentence. The sentiment level of

the phrases are represented as real values in the original dataset, here we’ll just use ve classes:

\very negative”, \negative”, \neutral”, \positive”, \very positive”

which are represented by 0 to 4 in the code, respectively. For this part, you will learn to train a softmax

regressor with SGD, and perform train/dev validation to improve generalization of your regressor.

(a) (10 points) Implement a sentence featurizer and softmax regression. Fill in the implementation in

q4 softmaxreg.py. Sanity check your implementation with python q4 softmaxreg.py.

(b) (2 points) Explain in fewer than three sentences why we want to introduce regularization when doing

classication (in fact, most machine learning tasks).

(c) (4 points) Fill in the hyperparameter selection code in q4 sentiment.py to search for the \optimal”

regularization parameter. What value did you select? Report your train, dev, and test ac-

curacies. Justify your hyperparameter search methodology in at most one sentence. Note:

you should be able to attain at least 30% accuracy on dev.

(d) (4 points) Plot the classication accuracy on the train and dev set with respect to the regularization

value, using a logarithmic scale on the x-axis. This should have been done automatically. Include

q4 reg acc.png in your homework write up. Brie y explain in at most three sentences what you

see in the plot.

Page 4 of 4