Description
Consider the high dimensional linear regression model
y = β0 + β1×1 + β2×2 + · · · + βpxp + , ∼ N(0, σ2
) (1)
Let x = (x1, …, xp)
0 ∼ N(0, Σ), where Σ is a p × p positive definite matrix. Assume that we
have observed a sample of size n, (yi
, xi)
n
i=1 and assume σ
2 = 1. Consider simulating data
by taking various combinations of (n, p, Σ, β) as follows
(a) n = 50, 500
(b) p = 100, 1000
(c) Σ = I,S0.1,S0.6,
(d) (i) β1 = · · · = β5 = 3, βj = 0 for any other j; (ii) β1 = · · · = β5 = 5, β6 = · · · = β10 = −2,
β11 = · · · = β15 = 0.5, βj = 0 for any other j; (iii)βj = 1 for all j,
where Sρ,ii = 1,Sρ,ij = ρ
|i−j|
for i 6= j. Altogether they give rise to 36 different combinations.
• Simulate data for all 36 combinations described as above.
• Run Lasso and Ridge regression for all 36 combinations and compare the results.
• Run Bayesian models with spike and slab, Bayesian Lasso and Generalized double
pareto prior distributions on β respectively for all 36 combinations. (Write your own
code)
1
• Numerically obtain E(βj
|y) for all the competing Bayesian models for all j. Discuss
accuracy of the Bayesian models w.r.t a metric.
• Compare lasso and spike and slab prior as methods for selecting variables.
• Let Lj be the length of 95% credible interval for the jth predictor. Let Mzero =
mean(Lj
: β
0
j 6= 0) and Mzero = mean(Lj
: β
0
j = 0) where β
0
j
is the true value of βj
.
Calculate Mzero and Mnonzero for all the competing Bayesian shrinkage priors.
• Take a particular combination n = 50, p = 100, S0.6 and (i), out of the 36 combinations. Simulate 50 additional responses and predictors (ypred,i, xpred,i)
50
i=1 for this
combination with (1). Draw 1000 samples from the posterior predictive distribution
π(y|y1, …, yn, xpred,i), ∀ i = 1, …, 50 for all the shrinkage priors. Calculate posterior
predictive mean yest,i at every xpred,i for Bayesian lasso and generalized double pareto.
Calculate MSPE = 1
50
P50
i=1(ypred,i − yest,i)
2
for both shrinkage priors.
2