Description
In this section, we are going to implement an agent which can play the Frozen Lake game (Version 1) via deep Q-learning (DQL). You may take a look at the documentation of this game at the Gymnasium portal; see gymlibrary.dev/environments/toy_text/frozen_lake. We mainly implement the deep Q-network (DQN) and the DQL algorithm. You have been given with source files that include the implementation for most parts of the algorithms. You should only complete the lines that are indicated. Attention: In the source files, you only complete lines within modules Agent.py and DQL.py . There, you only need to complete ## COMPLETE ## parts. These parts are also provided with descriptions on what you should complete. There are several functions and loops that have been already implemented. Feel free to use them as a reference code to implement the other parts or in your project. Preparation To get started, open the source files and check the main.py and Hyperparameters.py files. The former lets you train and test your implementation once you are over with completing Agent.py and DQL.py files, and the latter provides you with all hyperparameters you need. Go over the code in both files to get familiar with the class Hyperparameters() . Note that once you finish training your agent, the code will generate two learning curves and a .pth file in the corresponding folder. The .pth file includes the weights of the trained DQN that you could use to play with later. Sample outputs have been included for 4×4 case in the folder 4×4 . Take a look at them to get familiar with them. To run the code you need to have the libraries gymnasium , numpy , collections , matplotlib , and torch installed. To install any of these libraries, you can run 1 pip install 2 in the terminal. Question 1 [10 Points] (DQN) We want to implement a multi-layer perceptron (MLP) with two hidden layers. The details of the architecture are given below • The input layer takes the feature vector. For feature representation we use tokenization. This means that for each state we have a one-hot vector representation. • First hidden layer has width 16 with all neurons activated by ReLU (·). • Second hidden layer has width 8 with all neurons activated by ReLU (·). • Output layer is of the size of number of actions and is a simple linear layer with no activation. Complete the class DQN() in file Agent.py to implement the above network. Question 2 [15 Points] (Control Loop) In the file Agent.py go through the four methods __init__() , epsilon_greedy() , greedy() and apply_SGD and complete the annotated lines. This will complete the control loop of the agent. Question 3 [15 Points] (DQL Algorithm) Now, go through the file DQL.py and complete the missing lines in methods feature_representation() , train() and play() in class DQL() to complete the DQL algorithm.
Question 4 [10 Points] (Experimenting) After finishing with the code, try the following items 1. Train your agent for 4 × 4 and 8 × 8 maps. Play with your trained agents to see their performance. 2. Change hyperparameter epsilon_decay to 0.99 and train your agent once again. You can do this by uncommenting the given line in the file main.py . Explain why you see such a poor training. Attention: Submit the .pth files, the learning curves, and the final codes for both map sizes considering the original hyperparameters given in the source files. Be careful that your last experiment, i.e., chaning epsilon_decay to 0.99, does not overwrite your previous .pth files.
