STAT451 HW03: Practice with SVM, kNN, gradient descent, feature engineering solution

1. Visualize classifier decision boundaries.

1a. Complete the function in the next cell that plots a classifier’s decision boundary.

Hint: My solution used 9 lines:

• Make linspaces of grid_resolution points in xlim and grid_resolution points in ylim. e.g. For xlim=(-1, 1), ylim=(0, 2) and grid_resolution=3, make the linspace (-1, 0, 1) of x coordinates and the linspace (0, 1, 2) of y coordinates.
• Use np.tile() to repeat the x grid points grid_resolution times (e.g. (-1, 0, 1, -1, 0, 1, -1, 0, 1)) and np.repeat() to repeat each of the y grid points grid_resolution times (e.g. (0, 0, 0, 1, 1, 1, 2, 2, 2)).
• Use np.stack() to combine the x grid points and y grid points into a 2D array of size grid_resolution${}^{}$ x 2. (e.g. [[-1, 0], [0, 0], [1, 0], [-1, 1], [0, 1], [1, 1], [-1, 2], [0, 2], [1, 2]] )
• Make a dictionary keyed by -1 and 1 with values ‘pink’ and ‘lightskyblue’.
• Use clf.predict() on the 2D array of points to get predicted y values.
• For each y in {-1, 1}, use plt.plot() to plot those points in your 2D array with that predicted y value in the color specified by your dictionary.

Visualize the decision boundary for an SVM.

Here I have provided test code for your function to visualize the decision boundary for the SVM under the header “Now try 2D toy data” inhttps://pages.stat.wisc.edu/~jgillett/451/burkov/01/01separatingHyperplane.html.

Recall: That SVM’s decision boundary was $y=-x+\frac{1}{2}$, so your function should make a plot with lightskyblue above that line and pink below that line. Then my code adds the data points in blue and red.

There is nothing for you to do in this step, provided you implemented the required function above.

Note: It is ok if you get a warning about calling clf.fit() on input that does not have feature names. (I haven’t figured out a satisfactory way to design the function to exclude this warning easily.)

1b. Visualize the decision boundary for a decision tree.

• Make a decision tree classifier on the same df used above. (Use criterion=’entropy’, max_depth=None, random_state=0.)
• Use print(export_tree()) to print a text version of your tree.
• Copy the last few lines of the cell above to make the plot.
• Study the tree and plot until you understand how the plot represents the decisions in the tree.

1c. Visualize the decision boundary for kNN with $k=3$.

• Make a kNN classifier on the same df used above. (Use n_neighbors=3 and metric=’euclidean’.)
• Copy the plotting code again.

(Experiment with $k=1$ and $k=2$ to see how the decision boundary varies with $k$ before setting $k=3$.)

1d. Visualize the decision boundary for an SVM with a nonlinear boundary.

Use the example under the header “Nonlinear boundary: use kernel trick” in https://pages.stat.wisc.edu/~jgillett/451/burkov/03/03SVM.html.

• Read the data from http://www.stat.wisc.edu/~jgillett/451/data/circles.csv. This “.csv” file has y in {0, 1}, so change the 0 values to -1.
• Fit an SVM with kernel=’rbf’, C=1, gamma=1/2.
• Copy the last few lines of my plotting code, above, again to make the boundary plot.

(Experiment with $\gamma =2$, $\gamma =10$, and $\gamma =30$ to see how the decision boundary varies with gamma before setting gamma to 1/2.)

2. Run gradient descent by hand.

Run gradient descent with $\alpha =0.1$ to minimize $z=f(x,y)=(x+1{)}^2+(y+2{)}^2$. Start at (0, 0) and find the next two points on the descent path.

Hint: The minimum is at (-1, -2), so your answer should be approaching this point.

… your answer in a Markdown cell here …

3. Practice feature engineering

by exploring the fact that rescaling may be necessary for kNN but not for a decision tree.

3a. Read and plot a toy concentric ellipses data set.

• Read the data from http://www.stat.wisc.edu/~jgillett/451/data/ellipses.csv into a DataFrame.
• Display the first five rows.
• Plot the data.
  • Put x0 on the $x$ axis and x1 on the $y$ axis.
  • Plot points with these colors:
    • $y=0$: red
    • $y=1$: blue
  • Use $x$ and $y$ axis limits of $(-6,6)$.
  • Include a legend.

3b. Train a $k$NN classifier and report its accuracy.

• Use $k=3$ and the (default) euclidean metric.
• Report the accuracy on the training data by writing a line like Training accuracy is 0.500 (0.500 may not be correct).

3c. Now rescale the features using standardization; plot, train, and report accuracy again.

• Fit the scaler to the training features.
• Transform the training features.
• Plot the rescaled data.
• Train kNN again and report its accuracy as before. (Notice that rescaling helped.)

3d. Train a decision tree classifier on the original (unscaled) data and report its accuracy.

• Train on the training data.
• Report the accuracy as before.

3e. Why is feature scaling unnecessary for an ID3 decision tree? Answer in a markdown cell.

… your answer here in a Markdown cell …