Description
1
• Overfitting and Deterministic Noise
1. Deterministic noise depends on H, as some models approximate f better than
others. Assume that H0 ⊂ H and that f is fixed. In general (but not necessarily in all cases), if we use H0
instead of H, how does deterministic noise
behave?
[a] In general, deterministic noise will decrease.
[b] In general, deterministic noise will increase.
[c] In general, deterministic noise will be the same.
[d] There is deterministic noise for only one of H and H0
.
• Regularization with Weight Decay
In the following problems use the data provided in the files
https://work.caltech.edu/data/in.dta
https://work.caltech.edu/data/out.dta
as a training and test set respectively. Each line of the files corresponds to a twodimensional input x = (x1, x2), so that X = R
2
, followed by the corresponding
label from Y = {−1, 1}. We are going to apply Linear Regression with a non-linear
transformation for classification. The nonlinear transformation is given by
Φ(x1, x2) = (1, x1, x2, x2
1
, x2
2
, x1x2, |x1 − x2|, |x1 + x2|).
Recall that the classification error is defined as the fraction of misclassified points.
2. Run Linear Regression on the training set after performing the non-linear transformation. What values are closest (in Euclidean distance) to the in-sample and
out-of-sample classification errors, respectively?
[a] 0.03, 0.08
[b] 0.03, 0.10
[c] 0.04, 0.09
[d] 0.04, 0.11
[e] 0.05, 0.10
3. Now add weight decay to Linear Regression, that is, add the term λ
N
P7
i=0 w
2
i
to the squared in-sample error, using λ = 10k
. What are the closest values to
the in-sample and out-of-sample classification errors, respectively, for k = −3?
Recall that the solution for Linear Regression with Weight Decay was derived
in class.
2
[a] 0.01, 0.02
[b] 0.02, 0.04
[c] 0.02, 0.06
[d] 0.03, 0.08
[e] 0.03, 0.10
4. Now, use k = 3. What are the closest values to the new in-sample and out-ofsample classification errors, respectively?
[a] 0.2, 0.2
[b] 0.2, 0.3
[c] 0.3, 0.3
[d] 0.3, 0.4
[e] 0.4, 0.4
5. What value of k, among the following choices, achieves the smallest out-ofsample classification error?
[a] 2
[b] 1
[c] 0
[d] −1
[e] −2
6. What value is closest to the minimum out-of-sample classification error achieved
by varying k (limiting k to integer values)?
[a] 0.04
[b] 0.06
[c] 0.08
[d] 0.10
[e] 0.12
• Regularization for Polynomials
Polynomial models can be viewed as linear models in a space Z, under a nonlinear
transform Φ : X → Z. Here, Φ transforms the scalar x into a vector z of Legendre
3
polynomials, z = (1, L1(x), L2(x), …, LQ(x)). Our hypothesis set will be expressed as
a linear combination of these polynomials,
HQ =
(
h | h(x) = w
T
z =
X
Q
q=0
wqLq(x)
)
,
where L0(x) = 1.
7. Consider the following hypothesis set defined by the constraint:
H(Q, C, Qo) = {h | h(x) = w
T
z ∈ HQ;wq = C for q ≥ Qo},
which of the following statements is correct:
[a] H(10, 0, 3) ∪ H(10, 0, 4) = H4
[b] H(10, 1, 3) ∪ H(10, 1, 4) = H3
[c] H(10, 0, 3) ∩ H(10, 0, 4) = H2
[d] H(10, 1, 3) ∩ H(10, 1, 4) = H1
[e] None of the above
• Neural Networks
8. A fully connected Neural Network has L = 2; d
(0) = 5, d(1) = 3, d(2) = 1. If only
products of the form w
(l)
ij x
(l−1)
i
, w
(l)
ij δ
(l)
j
, and x
(l−1)
i
δ
(l)
j
count as operations (even
for x
(l−1)
0 = 1), without counting anything else, which of the following is the
closest to the total number of operations in a single iteration of backpropagation
(using SGD on one data point)?
[a] 30
[b] 35
[c] 40
[d] 45
[e] 50
Let us call every ‘node’ in a Neural Network a unit, whether that unit is an input
variable or a neuron in one of the layers. Consider a Neural Network that has 10
input units (the constant x
(0)
0
is counted here as a unit), one output unit, and 36
hidden units (each x
(l)
0
is also counted as a unit). The hidden units can be arranged
in any number of layers l = 1, · · · , L−1, and each layer is fully connected to the layer
above it.
4
9. What is the minimum possible number of weights that such a network can have?
[a] 46
[b] 47
[c] 56
[d] 57
[e] 58
10. What is the maximum possible number of weights that such a network can
have?
[a] 386
[b] 493
[c] 494
[d] 509
[e] 510
5
