Description
Overview
In this assignment, you will work with a real-world dataset containing information about
different cars. Your main objective is to predict each car’s fuel efficiency, measured in
miles per gallon (mpg), based on its features.
The original dataset comes from the StatLib library at Carnegie Mellon University and was
later cleaned by Ross Quinlan to remove entries with missing mpg values. The data has
been further refined by removing six entries with unknown horsepower values.
So far in class, we have mostly worked with examples involving only two features. In this
assignment, you will extend that knowledge to a multidimensional feature space and
explore how different feature choices affect learning performance, particularly for
classification and multivariate regression tasks.
Dataset Description
The dataset contains 392 car entries. Each entry includes the following 9 field names:
• mpg – Miles per gallon (output/target value)
• cylinders – Number of engine cylinders (4 to 8)
• displacement – Engine displacement in cubic inches (read more here)
• horsepower
• weight – Vehicle weight in pounds
• acceleration – Time (in seconds) to accelerate from 0 to 60 mph
• model year
• origin – 1 = USA, 2 = Europe, 3 = Asia
• car name
Part A – Conceptual Questions
All written responses for this section must be submitted in report.pdf.
1.a) Defining the Problem
We begin with a simple labeling step using only the mpg column. The goal is to label
each car based on the question: “Is this car fuel-efficient or not?”
In your response, explain:
• The mpg threshold you choose to label a car as fuel-efficient
• Why this threshold is reasonable based on the distribution and characteristics of
the data
1.b) Feature Representation and Preprocessing
Preprocessing is a crucial step in any machine learning pipeline. In this question, you will
analyze how to represent each feature in a way that improves learning and generalization.
For each of the 8 input features (excluding mpg), choose one of the following
representations:
• drop – Leave the feature out
• standard – Standardize values by subtracting out average value and dividing by
standard deviation
• one-hot – Use a one-hot encoding, i.e. a vector of 0’s and 1’s where the presence
of the feature is indicated by 1, and nonexistence by 0
For each feature:
• State your chosen representation
• State whether it is a continuous or discrete representation
• Discuss any tradeoffs with other possible choices
There may be multiple reasonable answers. The goal is to demonstrate your
understanding of how feature representation affects model performance.
1.c) Short Answer Questions
Provide short written answers to the following:
1.c.1) Can a model trained on this dataset be used to predict whether cars from
2019 are fuel-efficient? Explain your reasoning.
1.c.2) Since features like weight and cylinders are on very different numerical
scales, discuss whether this affects prediction performance for following
methods. If scale does matter, explain how the issue can be addressed.
• K-Nearest Neighbors (KNN)
• Logistic Regression
• Decision Trees
Part B – Coding Questions
2) KNN Classification (Python Implementation)
In this part, you will complete and run a K-Nearest Neighbors (KNN) classifier using the
provided Python code file (q2_classification.py). The goal is to classify cars as fuelefficient or not fuel-efficient, based on the design choices you made in Part A, 10-fold
cross validation, with at least 3 values of K. Follow the steps below carefully:
Step 1 – Load the Dataset
The code already loads the car dataset from a .tsv file. Make sure the dataset file is in
the same directory as the Python script.
Step 2 – Define the Efficiency Threshold (From 1.a)
Update the following line using the mpg threshold you selected in Part I – Question 1.a:
MPG_THRESHOLD = 0.0
This value will be used to convert mpg into a binary label: 1 = fuel-efficient or 0 = not
fuel-efficient.
Step 3 – Set Feature Preprocessing (From 1.b)
Modify the feature_plan dictionary to match your preprocessing decisions from
Question 1.b.
Step 4 – Implement the KNN Classifier
Complete the knn_predict() function so that it:
• Computes the distance between each test point and all training points
• Finds the K nearest neighbors
• Predicts the class using majority voting
Step 5 – Implement the Evaluation Metrics
You must implement two metrics discussed in class: Accuracy and F1-score. Hence,
complete the accuracy(y_true, y_pred) and f1(y_true, y_pred) functions.
Step 6 – Choose K Values
Select at least three different values of K using k_values list.
Step 7 – Run 10-Fold Cross-Validation
The code already performs 10-fold cross-validation. You do not need to modify this part.
Step 8 – Report and Discuss Results
In your report.pdf, include:
• A results table showing Accuracy and F1-score for each K
• A short explanation of:
o Why you chose these two metrics
o Which K performed best
o Any trends you observed
• A brief discussion connecting your preprocessing choices (Part A) to the results
• Snapshots of your modified parts in code
3) Predicting MPG with Ridge Regression using SGD (Python Implementation)
In this part, you will implement Ridge Regression to predict a car’s fuel efficiency (mpg)
as a continuous value, using stochastic gradient descent (SGD). You will work with the
same dataset and feature preprocessing decisions you made in Part A.
You are provided with two Python files:
• q3_helper.py (helper functions)
• q3_regression.py (main script)
Specific parts of the code are left for you to complete, as described below.
Objective Function
Your model should minimize the following objective:
𝐽 = TrainLoss(𝐰) +
𝜆
2
‖𝐰‖
2 =
1
2
(𝐱
𝑇𝐰 − 𝑦)
2 +
𝜆
2
‖𝐰‖
2
where the training loss is the mean squared error. The optimization is performed using
stochastic gradient descent.
Step 1 – Load the Dataset
The code already loads the car dataset from a .tsv file. Make sure the dataset file is in
the same directory as the Python script.
Step 2 – Set Feature Preprocessing (From 1.b)
Modify the feature_plan dictionary to match your preprocessing decisions from
Question 1.b.
Step 3 – Implement the Ridge Objective and Gradient
In q3_helper.py, complete the function make_ridge_J_dJ(lam).
This function should return:
• J: the Ridge Regression cost for one training example
• dJ: the gradient of this cost with respect to the weight vector
Note: The cost and gradient are computed for a single data point, since SGD is used.
Step 4 – Implement Regression Metrics
In q3_regression.py, complete the following two functions:
• rmse(y_true, y_pred): RMSE (Root Mean Squared Error) measures the typical
magnitude of prediction errors.
• mae(y_true, y_pred): MAE (Mean Absolute Error) measures the average absolute
prediction error.
These metrics will be used to evaluate model performance during cross-validation.
Step 5 – Choose Regularization Parameters
Select at least three different values of the regularization parameter 𝜆. These values are
used to control the strength of L2 regularization.
The code will automatically evaluate each value using 10-fold cross-validation.
Step 6 – Run 10-Fold Cross-Validation
The provided code performs 10-fold cross-validation using SGD to train the model and
reports the average RMSE and MAE across folds.
You do not need to modify this part.
Step 7 – Report and Discuss Results
In your report.pdf, include:
• A table showing RMSE and MAE for each value of 𝜆
• A short explanation of:
o Why RMSE and MAE are appropriate metrics for regression
o Which value of 𝜆 performed best
o Any trends you observed as 𝜆 increased
• A brief discussion connecting your feature preprocessing choices (Part A) to the
regression results
• Snapshots of your modified parts in code
