Description
1 Written Problems (6 points)
1. MLE minimizes KL divergence to the empirical distribution (Exercise
2.15 of Murphy’s book) (1 point)
2. Centering and ridge regression (Exercise 7.3 of Murphy’s book) (1
point)
3. Symmetric version of `2 regularized multinomial logistic regression (Exercise 8.5 of Murphy’s book) (1 point)
4. Elementary properties of `2 regularized logistic regression (Exercise 8.6
of Murphy’s book) (1 point)
5. Given the following denominator layout derivatives, (2 points)
• Differentiation of a scalar function w.r.t. a vector: If f(w) is a
scalar function of d variables, w is a d×1 vector, then differentiation
of f(w) w.r.t. w results in a d × 1 vector
df(w)
dw
=
∂f
∂w1
.
.
.
∂f
∂wd
• Differentiation of a vector function w.r.t. a vector: If f(w) is a
vector function of size h×1 and w is a d×1 vector, then differentiation
of f(w) w.r.t. w results in a d × h vector
df(w)
dw
=
∂f1
∂x1
. . .
∂fh
∂w1
.
.
.
.
.
.
.
.
.
∂f1
∂wd
. . .
∂fh
∂wd
1
Please prove the following derivatives, and X and y are not functions of
w:
d(X>w)
dw
= X,
d(y
>Xw)
dw
= X>y
d(w>Xw)
dw
= (X + X>)w
2 Programming (5 points)
Given Xw = y where X =
1 2
4 3
5 6
3 8
9 10
and y =
−1
0
1
0
0
which constitute an exemplary problem. Write a Python routine to find the least squares solution
for w given arbitrary X ∈ R5×2 and y ∈ R5×1
.
Submit your Python codes
as a function routine (“def A1 MatricNumber(X,y)”) that takes in X and y as
inputs and generate (X>X)
−1 and w as outputs in a single file with filename
“A1 StudentMatriculationNumber.py”. Your Python routine should return the
least squares solution vector w (as numpy array) and two matrices X> and
(X>X)
−1
. Hint: you will need “import numpy as np” and its matrix manipulation functions.
Please use the python template provided to you. Remember to rename both
“A1 StudentMatriculationNumber.py” and “A1 MatricNumber” using your student matriculation number.
For example, if your matriculation ID is 123456789,
then you should submit “A1 123456789.py” that contains the function “A1 123456789”.
The way we would run your code might be something like this:
import A1 123456789 as grading
w, XT, InvXTX = grading.A1 123456789(X,y)
Marks allocation is based on the three outputs: w (3 points), XT (1 point),
InvXTX (1 point)
NOTE: Please do NOT zip/compress your file. Please make sure you replace
“StudentMatriculationNumber” and “MatricNumber” with your matriculation
number! Because of the large class size, points will be deducted if instructions are not followed.