Description
Overview 1.1 Canadian Hansards The main corpus for this assignment comes from the official records (Hansards) of the 36th Canadian Parliament, including debates from both the House of Representatives and the Senate. This corpus is available at /u/cs401/A2/data/Hansard/ and has been split into Training/ and Testing/ directories. This data set consists of pairs of corresponding files (*.e is the English equivalent of the French *.f) in which every line is a sentence. Here, sentence alignment has already been performed for you. That is, the n th sentence in one file corresponds to the n th sentence in its corresponding file (e.g., line n in fubar.e is aligned with line n in fubar.f). Note that this data only consists of sentence pairs; many-to-one, many-to-many, and one-to-many alignments are not included. 1.2 Seq2seq We will be implementing a simple seq2seq model, without attention, with single-headed attention, and with multi-headed attention based largely on the course material. You will train the models with teacherforcing and decode using beam search. We will write it in PyTorch version 1.9.1 (https://pytorch.org/ docs/1.9.1/), and Python version 3.9.7, which are the versions installed on the teach.cs servers. For those unfamiliar with PyTorch, we suggest you first read the PyTorch tutorial (https://pytorch.org/ tutorials/beginner/deep_learning_60min_blitz.html). 1.3 Tensors and batches PyTorch, like many deep learning frameworks, operate with tensors, which are multi-dimensional arrays. When you work in PyTorch, you will rarely if ever work with just one bitext pair at a time. You’ll instead be working with multiple sequences in one tensor, organized along one dimension of the batch. This means that a pair of source and target tensors F and E actually correspond to multiple sequences F = (F (m) 1:S(m) )m∈[1,M] , E = (E (m) 1:T(m) )m∈[1,M] . We work with batches instead of individual sequences because: a) backpropagating the average gradient over a batch tends to converge faster than single samples, and b) sample computations can be performed in parallel. For example, if we want to multiply source sequences F (m) and F (m+1) with an embedding matrix W, we can tell one CPU core to compute the result for F (m) and another for F (m+1), halving the overall time it would take to multiply them independently. Learning to work with tensors can be difficult at first, but is integral to efficient computation. We suggest you read more about it in the NumPy docs (https://numpy.org/doc/stable/user/basics.broadcasting.html), which PyTorch borrows for tensors. Copyright © 2022 University of Toronto. All rights reserved. 1 1.4 Differences from the lectures There are three changes to the seq2seq architectures that we make for this assignment. First, instead of scaled dot-product attention scores score(a, b) = |a| −1/2 ⟨a, b⟩, we’ll use the cosine similarity between vectors a and b: score(a, b) = ⟨a, b⟩ max (∥a∥2 ∥b∥2 , ε) Where 0 < ε ≪ 1 ensures score(a, b) = 0 when a = 0 or b = 0. The second relates to how we calculate the first hidden state for the decoder when we don’t use attention. Recall that a bidirectional recurrent architecture processes its input in both directions separately: the forward direction processes (x1, x2, . . . , xS) whereas the backward direction processes (xS, xS−1, . . . , x1). The bidirectional hidden state concatenates the forward and backward hidden states for the same time ht = [h f orward t , hbackward t ]. This implies hS has processed all the input in the forward direction, but only one input in the backward direction (and vice versa for h1). To ensure the decoder gets access to all input from both directions, you should initialize the first decoder state h˜ 1 = [h f orward S , hbackward 1 ] When you use attention, set h˜ 1 = 0. Our final “change” isn’t so much a change as a clarification. In multi-headed attention, recall we have N heads such that h˜ (n) t = W˜ (n)h˜ t and h (n) s = W(n)hs. W(n) and W˜ (n) need not be square matrices; the size of h˜ (n) t need not be the size of h˜ t nor h (n) s the size of hs. For this assignment, we will be setting the size of |h˜ (n) t | = |h˜ t |/N and |h (n) s | = |hs|/N (you may assume N evenly divides the hidden state size). 2 Your tasks 2.1 Setup You are expected to run your solutions on teach.cs. Download the starter code from MarkUs. You can download the starter files by clicking the “Download” button in the “Starter Files” section of the Assignment page. The Hansards parallel text data are located in the /h/u1/cs401/A2/data/ directory on teach.cs. You don’t need to copy the text data to your own teach.cs directory. Download the starter code MarkUs into your working directory to get started. You should have 8 files: a2 abcs.py, a2 bleu score.py, a2 encoder decoder.py, a2 dataloader.py, a2 run.py, a2 training and testing.py, test a2 bleu score.py, and test a2 encoder decoder.py. 2.2 Calculating BLEU scores Modify a2 bleu score.py to be able to calculate BLEU scores on single reference and candidate strings. We will be using the definition of BLEU scores from the lecture slides: BLEU = BPC × (p1p2 . . . pn) (1/n) To do this, you will need to implement the functions grouper(…), n gram precision(…), brevity penalty(…), and BLEU score(…). Make sure to carefully follow the doc strings of each function. Do not re-implement functionality that is clearly performed by some other function. Your functions will operate on sequences (e.g., lists) of tokens. These tokens could be the words themselves (strings) or an integer ID corresponding to the words. Your code should be agnostic to the type of token used, though you can assume that both the reference and candidate sequences will use tokens of the same type. 2 2.3 Building the encoder/decoder You are expected to fill out a number of methods in a2 encoder decoder.py. These methods belong to sub-classes of the abstract base classes in a2 abcs.py. The latter defines the abstract classes EncoderBase, DecoderBase, and EncoderDecoderBase, which implement much of the boilerplate code necessary to get a seq2seq model up and running. Though you are welcome to read and understand this code, it is not necessary to do so for this assignment. You will, however, need to read the doc strings in a2 abcs.py to understand what you’re supposed to fill out in a2 encoder decoder.py. Do not modify any of the code in a2 abcs.py. A high-level description of the contents of the requirements for a2 encoder decoder.py follows here. More details can be found in the doc strings in a2 {abcs,encoder decoder}.py. 2.3.1 Encoder a2 encoder decoder.Encoder will be the concrete implementation of all encoders you will use. The encoder is always a multi-layer neural network with a bidirectional recurrent architecture. The encoder gets a batch of source sequences as input and outputs the corresponding sequence of hidden states from the last recurrent layer. Encoder.forward pass defines the structure of the encoder. For every model in PyTorch, the forward function defines how the model will run, and the forward function of every encoder or decoder will first clean up your input data and call forward pass to actually define the model structure. Now you need to implement the forward pass function that defines how your encoder will run. Encoder.init submodules(…) should be filled out to initialize a word embedding layer and a recurrent network architecture. Encoder.get all rnn inputs(…) accepts a batch of source sequences F (m) 1:S(m) and lengths S (m) and outputs word embeddings for the sequences x (m) 1:S(m) . Encoder.get all hidden states(…) converts the word embeddings x (m) 1:S(m) into hidden states for the last layer of the RNN h (m) 1:S(m) (note we’re using (m) here for the batch index, not the layer index). 2.3.2 Decoder without attention a2 encoder decoder.DecoderWithoutAttention will be the concrete implementation of the decoders that do not use attention (so-called “transducer” models). Method implementations should thus be tailored to not use attention. In order to feed the previous output into the decoder as input, the decoder can only process one step of input at a time and produce one output. Thus DecoderWithoutAttention is designed to process one slice of input at a time (though it will still be a batch of input for that given slice of time). The goal, then, is to take some target slice from the previous time step E (m) t−1 and produce an un-normalized log-probability distribution over target words at time step t, called logits(m) t . Logits can be converted to a categorical distribution using a softmax: P(y (m) t = i| . . .) = exp(logits(m) t,i ) P j exp(logits(m) t,j ) DecoderWithoutAttention.forward pass defines the structure of network. Similar to what you did for your encoder, you need to assemble the model here. DecoderWithoutAttention.init submodules(…) should be filled out to initialize a word embedding layer, a recurrent cell, and a feed-forward layer to convert the hidden state to logits. DecoderWithoutAttention.get first hidden state(…) produces h˜ (m) 1 given the encoder hidden states h (m) 1:S(m) . 3 DecoderWithoutAttention.get current rnn input(…) takes the previous target E (m) t−1 (or the previous output y (m) t−1 in testing) and outputs word embedding ˜x (m) t for the current step. DecoderWithoutAttention.get current hidden state(…) takes ˜x (m) t and h˜ (m) t−1 and produces the current decoder hidden state h˜ (m) t−1 . DecoderWithoutAttention.get current logits(…) takes h˜ (m) t and produces logits(m) t . 2.3.3 Decoder with (single-headed) attention a2 encoder decoder.DecoderWithAttention will be the concrete implementation of the decoders that use single-headed attention. It inherits from DecoderWithoutAttention to avoid re-implementing get current hidden states(…) and get current logits(…). The remaining methods must be reimplemented, but slightly modified for the attention context. Two new methods must be implemented for DecoderWithAttention. DecoderWithAttention.get attention scores(…) takes in a decoder state h˜ (m) t and all encoder hidden states h (m) 1:S(m) and produces attention scores for that decoder state but all encoder hidden states: e (m) t,1:S(m) . DecoderWithAttention.attend(…) takes in a decoder state h˜ (m) t and all encoder hidden states h (m) 1:S(m) and produces the attention context vector c (m) t . Between get attention scores and attend, use get attention weights(…) to convert e (m) t,1:S(m) to α (m) t,1:S(m) , which has been implemented for you. 2.3.4 Decoder with multi-head attention a2 encoder decoder.DecoderWithMultiHeadAttention implements a multi-headed variant of attention. It inherits from DecoderWithAttention. Two methods must be re-implemented for this variant. DecoderWithMultiHeadAttention.init submodules(…) should initialize new submodules for the matrices W, W˜ , and Q. DecoderWithMultiHeadAttention.attend(…) should “split” hidden states h˜ (m) t into h˜ (m,n) t and h (m) s into h (m,n) s , where m still indexes the batch number and n indexes the head. Then it should call super().attend(…) to do the attention, and combine c (m,n) t of the N heads. We want you to do this without ever actually “splitting” any tensors! The key is to reshape the a full hidden output into N chunks. If it’s a little bit too tricky, try starting from writing the case where N = 1, i.e. when there’s no need for splitting. 2.3.5 Putting it together: the Encoder/Decoder a2 encoder decoder.EncoderDecoder coordinates the encoder and decoder. Its behaviour depends on whether it’s being used for training or testing. In training, it receives both F (m) 1:S(m) and E (m) 1:T(m) and outputs logits(m) 1:T(m) un-normalized log-probabilities over y (m) 1:T(m) . In testing, it receives only F (m) 1:S(m) and outputs K paths from beam search per batch element n: y (n,k) 1:T(n,k) . EncoderDecoder.init submodules(…) initializes the encoder and decoder. EncoderDecoder.get logits for teacher forcing(…) provides you the encoder output h (m) 1:S(m) and the targets E (m) 1:T(m) and asks you to derive logits(m) 1:T(m) according to the MLE (teacher-forcing) objective. EncoderDecoder.update beam(…) asks you to handle one iteration of a simplified version of the beam search from the slides. While a proper beam search requires you to handle the set of finished paths 4 f, update beam doesn’t need to. Letting (n, k) indicate the n th batch elements’ k th path: ∀n, k, v.b(n,k→v) t,0 ← h˜ (n,k) t+1 b (n,k→v) t,1 ← [b (n,k) t,1 , v] log P(b (n,k→v) t ) ← log P(b (n,k) t ) + log P(yt+1 = v|h˜ (n,k) t+1 ) ∀n, k.b(n,k) t+1 ← argmaxk b (n,k′→v) t log P(b (n,k′→v) t ) In short, extend the existing paths, then prune back to the beam width. A greedy update function update greedy is provided for you in a2 abcs.py. You can use the option –greedy to switch to greedy update. This option might be handy when you want to test the correctness of the rest of your assignment. 2.3.6 Padding An important detail when dealing with sequences of input and output is how to deal with sequence lengths. Individual sequences within a batch F (m) and E(m) can have unequal lengths S (m) ̸= S (n+1) , T (m) ̸= T (n+1) , but we pad the shorter sequences to the right to match the longest sequence. This allows us to parallelize across multiple sequences, but it’s important that whatever the network learns (i.e., the error signal) is not impacted by padding. We’ve mostly handled this for you in the functions we’ve implemented, with three exceptions: first, no word embedding should be learned for padding (which you’ll have to guarantee); second, you’ll have to ensure the bidirectional encoder doesn’t process the padding; and third, the first hidden state of the decoder (without attention) should not be based on padded hidden states. You are given plenty of warning in the starter code when these three cases ought to be considered. The decoder uses the end-of-sequence symbol as padding, which is entirely handled in a2 training and testing.py. 2.4 The training and testing loops After following the PyTorch tutorial, you should be familiar with how models are trained and tested in PyTorch. You are expected to implement training and testing loops in a2 training and testing.py. In a2 training and testing.compute batch total bleu(…), you are given reference (from the dataset) and candidate batches (from the model) in the target language and asked to compute the total BLEU score over the batch. You will have to convert the PyTorch tensors in order to use a2 bleu score.BLEU score(…). In a2 training and testing.compute average bleu over dataset(…), you are to follow instructions in the doc string and use compute batch total bleu(…) to determine the average BLEU score over a data set. In a2 training and testing.train for epoch(…), once again follow the doc strings to iterate through a training data set and update model parameters using gradient descent. 2.5 Running the models Once you have completed the coding portion of the assignment, it is time you run your models. In order to do so in a reasonable amount of time, you’ll have to train your models using a machine with a GPU. There are a few ways you can do this: 1. You can ssh to teach.cs and use srun to run your code on a GPU on the department’s cluster. See more details below about how to use srun. 5 2. A number of teaching labs in the Bahen building have GPUs (listed in https://www.teach.cs. toronto.edu/faq.html#ABOUT4), but you must log in at the physical machines to use them (as opposed to remote access). 3. If you have access to your own GPU, you may run this code locally and report the results. However, any modifications you make to run the code locally must be reverted to work on teach before you submit! Even on a GPU, the code can take upwards of 2 hours to complete in full. Be sure to plan accordingly! You are going to interface with your models using the script a2 run.py. This script glues together the components you implemented previously. The only meaningful remaining code is in a2 dataloader.py, which converts the Hansard sentences into sequences of IDs. Suffice to say that you not need to know how either a2 run.py nor a2 dataloader.py works, only use them (unless you are interested). Run the following code block line-by-line from your working directory. In order, it: 1. Builds maps between words and unique numerical identifiers for each language. 2. Splits the training data into a portion to train on and a hold-out portion. 3. Trains the encoder/decoder without attention and stores the model parameters. 4. Trains the encoder/decoder with single-headed attention and stores the model parameters. 5. Trains the encoder/decoder with multi-headed-attention and stores the model parameters. 6. Returns the average BLEU score of the encoder/decoder without attention on the test set. 7. Returns the average BLEU score of the encoder/decoder with single-headed attention on the test set. 8. Returns the average BLEU score of the encoder/decoder with multi-headed attention on the test set. export TRAIN=/h/u1/cs401/A2/data/Hansard/Training/ export TEST=/h/u1/cs401/A2/data/Hansard/Testing/ # 1. Generate vocabularies python3.9 a2_run.py vocab $TRAIN e vocab.e.gz python3.9 a2_run.py vocab $TRAIN f vocab.f.gz # 2. Split train and dev sets python3.9 a2_run.py split $TRAIN train.txt.gz dev.txt.gz # 3. Train a model without attention srun -p csc401 –gres gpu \ python3.9 a2_run.py train $TRAIN \ vocab.e.gz vocab.f.gz \ train.txt.gz dev.txt.gz \ model_wo_att.pt.gz \ –device cuda # 4. Train a model with attention srun -p csc401 –gres gpu \ python3.9 a2_run.py train $TRAIN \ vocab.e.gz vocab.f.gz \ 6 train.txt.gz dev.txt.gz \ model_w_att.pt.gz \ –with-attention \ –device cuda # 5. Train a model with multi-head attention srun -p csc401 –gres gpu \ python3.9 a2_run.py train $TRAIN \ vocab.e.gz vocab.f.gz \ train.txt.gz dev.txt.gz \ model_w_mhatt.pt.gz \ –with-multihead-attention \ –device cuda # 6. Test the model without attention srun -p csc401 –gres gpu \ python3.9 a2_run.py test $TEST \ vocab.e.gz vocab.f.gz model_wo_att.pt.gz \ –device cuda # 7. Test the model with attention srun -p csc401 –gres gpu \ python3.9 a2_run.py test $TEST \ vocab.e.gz vocab.f.gz model_w_att.pt.gz \ –with-attention –device cuda # 8. Test the model with multi-head attention srun -p csc401 –gres gpu \ python3.9 a2_run.py test $TEST \ vocab.e.gz vocab.f.gz model_w_mhatt.pt.gz \ –with-multihead-attention –device cuda Steps 1 and 2 should not fail and need only be run once. Step 3 onward depend on the correctness of your code. The srun -p csc401 –gres gpu is necessary to run on a GPU on teach. You do not need a GPU for the first two steps. If you are running the training/testing locally, you will not need srun -p csc401 when running steps 3-6. The srun prefix is only needed when running on the (remote) teach server that uses SLURM1 to schedule processes on the department’s cluster. Because all students in this class or any class requiring GPUs will be running their jobs on the cluster please only run steps 3-6 after you have debugged your code. We discuss how you can train a smaller network below that will only take a fraction of the time anyway. In a file called analysis.txt, provide the following: • The printout after every epoch of the training loop of both the model for the model trained without, with single-headed, and with multi-headed attention. Clearly indicate which is which. • The average BLEU score reported on the test set for each model. Again, clearly indicate which is which. 1 https://en.wikipedia.org/wiki/Slurm_Workload_Manager 7 • A brief discussion on your findings. Was there a discrepancy in between training and testing results? Why do you think that is? If one model did better than the others, why do you think that is? 2.6 Bonus [up to 15 marks] We will give bonus marks for innovative work going substantially beyond the minimal requirements. However, your overall mark for this assignment cannot exceed 100%. Submit your write-up in bonus.pdf. You may decide to pursue any number of tasks of your own design related to this assignment, although you should consult with the instructor or the TA before embarking on such exploration. Certainly, the rest of the assignment takes higher priority. Some ideas: • Perform substantial data analysis of the error trends observed in each method you implement. This must go well beyond the basic discussion already included in the assignment. • There are many possible ways to assemble attentions for the encoder / decoder. For example, dotproduct attention similar to (Vaswani et al., 2017)2 , additive attention (Bahdanau et al., 2014)3 and structured self attention (Lin et al., 2017)4 . Analyze several attention mechanisms, compare their performances, and include discussions towards their reasons of the performance differences. • Explore the effects of using different ‘attention score function’ for computing attention score as discussed above and include attention visualization of the different attention functions. 2 https://arxiv.org/abs/1706.03762 3 https://arxiv.org/abs/1409.0473 4 https://arxiv.org/abs/1703.03130 8 3 Submission requirements This assignment is submitted electronically. Submit your assignment on MarkUs. Do not tar or compress your files, and do not place your files in subdirectories. You should submit: 1. The files a2 bleu score.py, a2 encoder decoder.py, and a2 training and testing.py that you filled out according to assignment specifications. We will not accept a2 abc.py, a2 dataloader.py, nor a2 run.py – your assignment must be compatible with the versions we provided you. 2. Your write-up on the experiment in analysis.txt. 3. If you are submitting a bonus, tell us what you’ve done by submitting a write-up in bonus.pdf. Please distribute bonus code amongst the above *.py files, being careful not to break functions, methods, and classes related to the assignment requirements. You should not submit any additional files that you generated to train and test your models. For example, do not submit your model parameter files *.pt.gz or vocab files *.{e,f}.gz. Only submit the above files. Additional source files containing helper functions are not permitted. 9 A Suggestions A.1 Check Piazza regularly Updates to this assignment as well as additional assistance outside tutorials will be primarily distributed via Piazza (https://piazza.com/class/kx9cacio1zf5rh). It is your responsibility to check Piazza regularly for updates. A.2 Run cluster code early and at irregular times Because GPU resources are shared with your peers, your srun job may end up on a weaker machine (or even postponed until the resources are available) if too many students are training at once. To help balance resource usage over time, we recommend you finish this assignment as early as possible. You might find that your peers are more likely to run code at certain times in the day. To check how many jobs are currently queued or running on our partition, please run squeue -p csc401. If you decide to run your models right before the assignment deadline, please be aware that we will be unable to request more resources or run your code sooner. We will not grant extensions for this reason. By the way, if you end up scheduling your job on one of the slower machines, you might end up with slightly different results than on the fastest ones. This is an unfortunate but normal by-product of how the GPUs calculate results in slightly different ways. Your BLEU score should not differ by more than 3% absolute. A.3 Connection persistence: keep training after disconnecting Training a model can take hours. If your internet connection is weak, you run the risk of losing your progress. You can use the Linux screen (https://linux.die.net/man/1/screen) command to create a persistent shell that is only destroyed when you exit from it. This will allow training to continue even if disconnected. The most basic usage of screen is as follows. To start a new shell, run the command screen. You should see the same shell interface as before (i.e. wolf:$) but you are now in a new shell instance. You can type Ctrl+D or the command exit to kill the screen, or type Ctrl+A followed by Ctrl+D to detach from the screen. A detached screen persists between SSH sessions. If you want to reconnect to your shell, try the command screen -r. Tmux users can find the ‘Tmux cheatsheet’ at https://tmuxcheatsheet.com/ helpful. A.4 Using your own computer If you want to do some or all of this assignment on your laptop or other computer, you will have to do the extra work of downloading and installing the requisite software and data. You take on the risk that your computer might not be adequate for the task. You are strongly advised to upload regular backups 10 of your work to teach.cs, so that if your machine fails or proves to be inadequate, you can immediately continue working on the assignment at teach.cs. When you have completed the assignment, you should try your programs out on teach.cs to make sure that they run correctly there. A submission that does not work on teach.cs will get zero marks. That said, due to concerns of limited resources, we will allow you to report the results of training/testing your model on your local machine. The code must still conform to the teach.cs environment, but the contents of analysis.txt can be based on your local environment. A.5 Unit testing We strongly recommend you test the methods and functions you’ve implemented prior to running the training loop. You can test actual output against expected input for complex methods like update beam(…). The Python 3.7 teach environment has installed pytest (https://docs.pytest.org/en/5.1.2/). We have included some preliminary tests for BLEU score(…) and update beam(…). You can run your test suite on teach by calling python3.9 -m pytest While the test suite will execute initially, the provided tests will fail until you implement the necessary methods and functions. While passing these initial tests is a necessity for full marks, they are not sufficient on their own. Please be sure to add your own tests. Unit testing is not a requirement, nor will you receive bonus marks for it. A.6 Debugging task Instead of re-running the entire task on GPU when debugging your code, we recommend that you run a much smaller version of the task until you are confident that you are error free. Ideally, you should only need to run the full task once. The following commands may be used to set up such a task. # Variables TRAIN and TEST as before. export OMP_NUM_THREADS=4 # avoids a libgomp error on teach # create an input and output vocabulary of only 100 words python3.9 a2_run.py vocab $TRAIN e vocab_tiny.e.gz –max-vocab 100 python3.9 a2_run.py vocab $TRAIN f vocab_tiny.f.gz –max-vocab 100 # only use the proceedings of 4 meetings, 3 for training and 1 for dev python3.9 a2_run.py split $TRAIN train_tiny.txt.gz dev_tiny.txt.gz –limit 4 # use far fewer parameters in your model python3.9 a2_run.py train $TRAIN \ vocab_tiny.e.gz vocab_tiny.f.gz \ train_tiny.txt.gz dev_tiny.txt.gz \ model.pt.gz \ 11 –epochs 2 \ –word-embedding-size 51 \ –encoder-hidden-size 101 \ –batch-size 5 \ –cell-type gru \ –beam-width 2 # use with the flags –with-attention and –with-multihead-attention to test single- # and multi-headed attention, respectively. We request that you first see if running directly on the teach.cs server (wolf) runs quickly enough before attempting to use the cluster. Tested at low occupancy, each epoch took about 30 seconds with welloptimized code directly on wolf. At high occupancy, each epoch took 4 minutes. Note that your BLEU will likely be high in the reduced vocabulary condition – even using very few parameters – since your model will end up learning to output the out-of-vocabulary symbol. Do not report your findings on the toy task in analysis.txt. A.7 Beam search not finished warning You might come across a warning like this during training: a2_abcs.py:882: UserWarning: Beam search not finished by t=100. Halted This just means your model failed to output an end-of-sequence token after t = 100. This may not mean you’ve made a mistake. On certain machines in the cluster and certain network configurations, even our solutions give this warning. However, it should not occur over all epochs. Your BLEU score shouldn’t change much whether or not you get this warning because it should only occur on a few sentences. If the warning keeps popping up or your BLEU scores are close to zero, you probably have an error. A.8 Recurrent cell type You’ll notice that the code asks you to set up a different recurrent cell type depending on the setting of the attribute self.cell type. This could be an LSTM, GRU, or RNN (the last refers to the simple linear weighting ht = σ(W[x, ht−1] + b) that you saw in class). The three cell types act very similarly – GRUs and RNNs are largely interchangeable from a programming perspective – except the LSTM cell often requires you to carry around both a cell state and a hidden state. Pay careful attention to the documentation for when h t, htilde t, etc. might actually be a pair of the hidden state and cell state as opposed to just a hidden state. Sometimes no change is necessary to handle the LSTM cell. Other times you might have to repeat an operation on the elements individually. Take advantage of the following pattern: 12 if self.cell_type == ’lstm’: # do something else: # do something else Be sure to rerun training with different cell types (i.e. use the flag –cell-type) to ensure your code can handle the difference. 13 B Variable names and slides We try to match the variable names in a2 abc.py to those in the lecture slides on machine translation. The table below serves as reference for converting between the two. In general, we denote a specific indexed value of a variable from the slides, such as Et , with an underscored variable name, e.g. E t. When an index is omitted, it means the variable contains all the indexed values at once, e.g. F corresponds to F1:S. Note that the slides are 1-indexed, whereas code is 0-indexed. Also, all variables in the PyTorch code are batched, but the slides only look at one sequence at a time. 14 Variable Slides Notes F F1:S Source sequence. F lens S In the code, the maximum-length source sequence in the batch is said to have length S. The actual length of each sequence in the batch (before padding) is stored in F lens. x x Encoder RNN inputs. h h1:S Encoder hidden states. Always refers to the last encoder layer’s hidden states, with both directions concatenated. htilde 0 h˜ 1 The first decoder hidden state. E tm1 Et−1 The target token at t − 1 (previous). xtilde t x˜t Decoder RNN input at time t (current). htilde t h˜ t Decoder hidden state at time t (current). For the LSTM architecture, this can be a pair with the cell state. Note in the beam search update htilde t also includes paths, i.e. h˜ (1:K) t . logits t log P(yt | . . .) + C Un-normalized log-probabilities over the target vocabulary at time t (current). Pre-softmax. E E1:T Target sequence. logits log P(yt | . . .) 1:T + C Un-normalized log-probabilities over the the target vocabulary across each time step. Pre-softmax. b tm1 1 b (1:K) t−1,1 All prefixes in the beam search at time t − 1 (previous). logpb tm1 log P(b (1:K) t−1 ) The log-probabilities of the beam search prefixes up to time t − 1 (previous). logpy t log P(yt | . . .) Valid (normalized) log-probabilities over the target vocabulary at time t (current). Post-softmax. b t 0 b (1:K) t,0 Decoder hidden states at time t (current). The difference between b (1:K) t,0 and h˜ (1:K) t is contextual: the latter points to the paths in the beam before the update, the former after the update. b t 1 b (1:K) t,1 All prefixes in the beam seach at time t (current). logpb t log P(b (1:K) t ) The log-probabilities of the beam search prefixes up to time t (current). c t ct Context vector (for attention) at time t (current). alpha t αt,1:S Attention weights over all source times at target time t (current). e t et,1:S Attention scores over all source times at target time t (current). 15
