Description
Introduction
This assignment is an introduction to using PyTorch for training simple neural net models. Two different datasets will be used:
- MNIST digits [handwritten digits]
- CIFAR-10 [32×32 resolution color images of 10 object classes].
Requirements
You should perform this assignment in PyTorch, modify this ipython notebook
To install PyTorch, follow instructions at https://pytorch.org/
Please email your assignment to the graders: Yi-Hsiang Kao (ykh342@nyu.edu) and Anshul Sharma (as10950@nyu.edu).
Warmup [10%]
It is always good practice to visually inspect your data before trying to train a model, since it lets you check for problems and get a feel for the task at hand.
MNIST is a dataset of 70,000 grayscale hand-written digits (0 through 9). 60,000 of these are training images. 10,000 are a held out test set.
CIFAR-10 is a dataset of 60,000 color images (32 by 32 resolution) across 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). The train/test split is 50k/10k.
Use matplotlib and ipython notebook’s visualization capabilities to display some of these images. See this PyTorch tutorial page for hints on how to achieve this.
** Relevant Cell: “Data Loading” **
Training a Single Layer Network on MNIST [20%]
Start by running the training on MNIST. By default if you run this notebook successfully, it will train on MNIST.
This will initialize a single layer model train it on the 50,000 MNIST training images for 10 epochs (passes through the training data).
The loss function cross_entropy computes a Logarithm of the Softmax on the output of the neural network, and then computes the negative log-likelihood w.r.t. the given target.
The default values for the learning rate, batch size and number of epochs are given in the “options” cell of this notebook. Unless otherwise specified, use the default values throughout this assignment.
Note the decrease in training loss and corresponding decrease in validation errors.
Paste the output into your report. (a): Add code to plot out the network weights as images (one for each output, of size 28 by 28) after the last epoch. Grab a screenshot of the figure and include it in your report. (Hint threads: #1 #2 )
(b): Reduce the number of training examples to just 50. [Hint: limit the iterator in the train function]. Paste the output into your report and explain what is happening to the model.
Training a Multi-Layer Network on MNIST [20%]
- Add an extra layer to the network with 1000 hidden units and a
tanhnon-linearity. [Hint: modify theNetclass]. Train the model for 10 epochs and save the output into your report. - Now set the learning rate to 10 and observe what happens during training. Save the output in your report and give a brief explanation
Training a Convolutional Network on CIFAR [50%]
To change over to the CIFAR-10 dataset, change the options cell’s dataset variable to 'cifar10'.
- Create a convolutional network with the following architecture:
- Convolution with 5 by 5 filters, 16 feature maps + Tanh nonlinearity.
- 2 by 2 max pooling.
- Convolution with 5 by 5 filters, 128 feature maps + Tanh nonlinearity.
- 2 by 2 max pooling.
- Flatten to vector.
- Linear layer with 64 hidden units + Tanh nonlinearity.
- Linear layer to 10 output units.
Train it for 20 epochs on the CIFAR-10 training set and copy the output into your report, along with a image of the first layer filters.
Hints: Follow the first PyTorch tutorial or look at the MNIST example
- Give a breakdown of the parameters within the above model, and the overall number.
from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable# options
dataset = 'cifar10' # options: 'mnist' | 'cifar10'
batch_size = 64 # input batch size for training
epochs = 10 # number of epochs to train
lr = 0.01 # learning rate# Data Loading
# Warning: this cell might take some time when you run it for the first time,
# because it will download the datasets from the internet
if dataset == 'mnist':
data_transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])
trainset = datasets.MNIST(root='.', train=True, download=True, transform=data_transform)
testset = datasets.MNIST(root='.', train=False, download=True, transform=data_transform)
elif dataset == 'cifar10':
data_transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
trainset = datasets.CIFAR10(root='.', train=True, download=True, transform=data_transform)
testset = datasets.CIFAR10(root='.', train=True, download=True, transform=data_transform)
train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=0)
test_loader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=0)## network and optimizer
if dataset == 'mnist':
num_inputs = 784
elif dataset == 'cifar10':
num_inputs = 3072
num_outputs = 10 # same for both CIFAR10 and MNIST, both have 10 classes as outputs
class Net(nn.Module):
def __init__(self, num_inputs, num_outputs):
super(Net, self).__init__()
self.linear = nn.Linear(num_inputs, num_outputs)
def forward(self, input):
input = input.view(-1, num_inputs) # reshape input to batch x num_inputs
output = self.linear(input)
return output
network = Net(num_inputs, num_outputs)
optimizer = optim.SGD(network.parameters(), lr=lr)def train(epoch):
network.train()
for batch_idx, (data, target) in enumerate(train_loader):
data, target = Variable(data), Variable(target)
optimizer.zero_grad()
output = network(data)
loss = F.cross_entropy(output, target)
loss.backward()
optimizer.step()
if batch_idx % 100 == 0:
print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
epoch, batch_idx * len(data), len(train_loader.dataset),
100. * batch_idx / len(train_loader), loss.data[0]))
def test():
network.eval()
test_loss = 0
correct = 0
for data, target in test_loader:
data, target = Variable(data, volatile=True), Variable(target)
output = network(data)
test_loss += F.cross_entropy(output, target, size_average=False).data[0] # sum up batch loss
pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
correct += pred.eq(target.data.view_as(pred)).cpu().sum()
test_loss /= len(test_loader.dataset)
print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
test_loss, correct, len(test_loader.dataset),
100. * correct / len(test_loader.dataset)))
for epoch in range(1, epochs + 1):
train(epoch)
test()