Assignment 4 Neural Networks and Deep Learning CSCI 5922 solution

The goal of this assignment is to use tensorflow to explore convolutional nets and to optimize a network for a challenging visual classification task.

You will experiment with CIFAR-10, a data set of tiny (32×32) color images in 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. This task is nontrivial: an error rate of 15-25% is likely.

The data set is divided into 50,000 training images and 10,000 test images. For part 1 of the assignment, do not peek at the test images. Instead, split your 50k training examples into a training set and a validation set. Decide how many examples to use for each.

TensorFlow has a very nice tutorial using the CIFAR-10 data with a sample architecture and code for reading the data and annealing learning rates, etc. It is fair to start with this demo and modify it to optimize your architecture.

(1a) Split your 50k training examples into a training set and a validation set. Decide how many examples to use for each. Describe your split into training and validation. (You may wish to do k-fold cross validation instead of just a single fold of validation, or you may want to do many-fold validation where you resample the training and validation sets each time. Simply describe the strategy  you have selected.)

(1b) Use tensorflow to build a convolutional neural net for this task. You can use any of the tricks we’ve discussed in the course, including pooling, max-pooling, batch normalization, data augmentation, residual networks, drop out, etc. Of course there is a huge literature on CIFAR-10 on the web (including results from various approaches), but I ask you not to pay much attention to this literature and try to invent your own architecture from scratch. You will want to experiment with different variations of your architecture and the combination of tricks you use. I want you to report the history of experiments you’ve performed by presenting the following information for each variant:

• Describe the architecture in enough detail that another member of the class could replicate the work.  For example, you may vary the number of hidden layers or whether you are using dropout. But report details such as the receptive field size of each convolutional layer and the stride, and if you’re using dropout, what dropout % you chose.
• Report training classification accuracy (regardless of the loss function you use for training) and the validation classification accuracy.

You will undoubtably play with some minor variations (e.g., changing learning rates, loss function, etc.). You needn’t report every such tweek. My goal is for you to convey what you believe are the most important architectural manipulations needed to get good performance on this task.

(2a) For the variation that obtained the best performance in Part 1, compute error on the test set (the 10k examples you had set aside until now).

It might be interesting to pull some examples of images from the 80 million tiny images data set from classes other than the 10 classes you trained on, e.g., buildings or goats or mailboxes. Run these images through your trained network and see whether you can use the output entropy (or some similar measure) to determine whether the test image can be rejected as not belonging to any object category. If you pick an entropy threshold, you will not only reject some of the out-of-class examples, but you will also falsely reject some in-class examples.

(3a) Describe the rejection criterion you use, and for that criterion, report on the % of out-of-class examples you’ve correctly rejected, but also the % of in-class examples (from the test set) that you erroneously rejected.

Present objects from an untrained class.