Description
Problem 1: Poisson Maximum Likelihood Estimation (20pts)
Consider a nonnegative, integer-valued random variable X that is distributed according to a Poisson
distribution X ∼
λ
xe−λ
x!
for some real-valued parameter λ > 0.
1. Given data samples x
(1), . . . , x(m)
, what is the maximum likelihood estimate for λ?
2. Suppose now that you introduce a prior probability distribution, λ ∼
1
5 max{−λ/10 + 1, 0}.
What is the MAP estimate under this prior probability distribution?
3. Why might you not prefer a prior probability distribution like the above for this estimation
task? What might be a better prior?
Problem 2: Log-Normal Maximum Likelihood Estimation (20pts)
Consider a positive, real-valued random variable X that is distributed according to a log-normal
distribution: X ∼
1
xσ√
2π
exp(−
(ln x−µ)
2
2σ2 ) for real-valued parameters µ and σ with σ > 0.
Suppose
that you are given M data observations x
(1), . . . , x(M)
.
1. For a given µ and σ, what is the log-likelihood of the data observations?
2. Find the maximum likelihood estimators for µ and σ. Note that the log-likelihood is not a
concave function of µ and σ.
3. Are the maximum likelihood estimators for µ and σ unbiased?
4. Why might a Gaussian distribution not be a good choice for a prior distribution over µ?
Problem 3: Logistic Regression (30pts)
For this problem, consider the Sonar data sets from Homework 4.
1
1. Fit a logistic regression classifier to training data set. What is the accuracy on the test set?
Explain why in standard logistic regression, without any type of regularization, the weights
may not converge (even though the predicted label for each data point effectively does) if the
input data is linearly separable.
2. Fit a logistic regression classifier with an `2 penalty on the weights to this data set using the
validation set to select a good choice of the regularization constant. Report your selected
constant, the learned weights and bias, and the accuracy on the test set.
3. Fit a logistic regression classifier with an `1 penalty on the weights to this data set using the
validation set to select a good choice of the regularization constant. Report your selected
constant, the learned weights and bias, and the accuracy on the test set.
4. Does `1 or `2 tend to produce sparser weight vectors?
Problem 4: Gaussian Na¨ıve Bayes (30pts)
For this problem, consider the Sonar data set from Homework 4. Suppose that you want to fit a
Gaussian NB model to this data. That is, assume that the probability distribution is of the form
p(x1, . . . , xn, y) = p(y)
Yn
i=1
p(xi
|y)
where p(xi
|y) is a distinctly parameterized normal distribution, i.e., p(xi
|y) is parameterized by
µi,y and σi,y.
1. Given a data set with m continuous features, what is the log-likelihood of the Gaussian NB
model? Compute the MLE for each of the model parameters.
2. Fit a Gaussian NB model to the training data. What is the accuracy of your trained model
on the test set?
3. What kind of prior might make sense for this model? Explain.
4. Do you think the NB assumption is reasonable here?
2
