Description
ECE472 Deep Learning – Assignment 1
tldr: Perform linear regression of a noisy sinewave using a set of gaussian basis
functions with learned location and scale parameters. Model parameters are
learned with stochastic gradient descent. Use of automatic differentiation is
required. Hint: note your limits!
Problem Statement Consider a set of scalars {x1, x2, . . . , xN } drawn from U(0, 1)
and a corresponding set {y1, y2, . . . , yN } where:
yi = sin (2πxi) + ϵi (1)
and ϵi
is drawn from N (0, σnoise). Given the following functional form:
yˆi =
∑
M
j=1
wjϕj (xi
| µj
, σj ) + b (2)
with:
ϕ(x | µ, σ) = exp
−(x − µ)
2
σ
2
(3)
find estimates ˆb, {µˆj}, {σˆj}, and {wˆj} that minimize the loss function:
J(y, yˆ) = 1
2
(y − yˆ)
2
(4)
for all (xi
, yi) pairs. Estimates for the parameters must be found using stochastic
gradient descent. A framework that supports automatic differentiation must be
used. Set N = 50, σnoise = 0.1. Select M as appropriate. Produce two plots. First,
show the data-points, a noiseless sinewave, and the manifold produced by the
regression model. Second, show each of the M basis functions. Plots must be of
suitable visual quality.
−4 −2 0 2 4
x
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
y
Fit 1
−4 −2 0 2 4
x
0.0
0.2
0.4
0.6
0.8
1.0
y
Bases for Fit 1
−4 −2 0 2 4
x
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
y
Fit 2
−4 −2 0 2 4
x
0.0
0.2
0.4
0.6
0.8
1.0
y
Bases for Fit 2
Figure 1: Example plots for models with equally spaced sigmoid and gaussian basis functions.
ECE472 Deep Learning – Assignment 2
tldr: Perform binary classification on the spirals dataset using a multi-layer
perceptron. You must generate the data yourself.
Problem Statement Consider a set of examples with two classes and distributions as
in Figure 1. Given the vector x ∈ R
2
infer its target class t ∈ {0, 1}. As a model
use a multi-layer perceptron f which returns an estimate for the conditional
density p(t = 1 | x):
f : R
2 → [0, 1] (1)
parametrisized by some set of values θ. All of the examples in the training set
should be classified correctedly (i.e. p(t = 1 | x) > 0.5 if and only if t = 1).
Impose an L
2 penalty on the set of parameters. Produce one plot. Show the
examples and the boundary corresponding to p(t = 1 | x) = 0.5. The plot must be
of suitable visual quality. It may be difficult to to find an appropriate functional
form for f, write a few sentences discussing your various attempts.
−10 −5 0 5 10
−15
−10
−5
0
5
10
15
Spirals
Figure 1: Sample spiral data.
ECE472 Deep Learning – Assignment 3
tldr: Classify mnist digits with a (optionally convoultional) neural network. Get
at least 95.5% accuracy on the test test.
Problem Statement Consider the mnist dataset consisting of 50,000 training
images, and 10,000 test images. Each instance is a 28 × 28 pixel handwritten digit
zero through nine. Train a (optionally convolutional) neural network for
classification using the training set that achieves at least 95.5% accuracy on the test
set. Do not explicitly tune hyperparameters based on the test set performance, use
a validation set taken from the training set as discussed in class. Use dropout and
an L
2 penalty for regularization. Note: if you write a sufficiently general program
the next assignment will be very easy.
Do not use the built in mnist data class from tensorflow.
Extra challenge (optional) In addition to the above, the student with the fewest
number of parameters for a network that gets at least 80% accuracy on the test set
will receive a prize. There will be an extra prize if any one can achieve 80% on the
test set with a single digit number of parameters. For this extra challenge you can
make your network have any crazy kind of topology you’d like, it just needs to be
optimized by a gradient based algorithm.
ECE472 Assignment 4
tldr: Classify cifar. Acheive performance similar to the state of the art. Classify
cifar. Achieve a top-5 accuracy of 80%.
Problem Statement Consider the cifar and cifar datasets which contain
32 × 32 pixel color images. Train a classifier for each of these with performance
similar to the state of the art (for cifar. It is your task to figure out what is state
of the art. Feel free to adapt any techniques from papers you read. I encourage you
to experiment with normalization techniques and optimization algorithms in this
assignment. Write a paragraph or two summarizing your experiments. Hopefully
you’ll be able to resuse your mnist program.
