Solved [CS639] Homework 3 Spring 2026

Description

1 VIT
1.1 Setup
The image data we will use for this homework is CIFAR100. The code used to
create the dataset objects is already included in the notebook. Also provided is
code for attention and transformer blocks.
1.2 [Code] VIT Implementation (15 points)
The transformer building blocks are provided for you in this homework. It
is recommended you look at the TransformerBlock class, which now has an
argument that controls whether it will use causal masking or not.
Using these building blocks, complete the ViTImageEncoder class.
1.3 [Writeup] VIT Understanding (15 points)
Explain in detail all the important components of the VIT architecture, and
how it processes a 2D image input to return the final logits.
1.4 [Code, Writeup] VIT Training (10 points)
The training code is provided for you. Train the VIT classifier and report the
hyperparameters used, the final training and validation metrics, and the plots
of the trajectories.
2 CLIP
In this question, you will use your VIT implementation as the image encoder
to train a toy CLIP model. The text encoder is (nearly) implemented for you
already.
1
2.1 [Code] Text Encoder Implementation (2.5 points)
You will need to implement one final piece of the TextEncoder class’s forward()
function. Most of the provided model is the same as the character-level GPT
from Homework 2, the only change you need to make is to return the text
embeddings instead of logits.
2.2 [Writeup] Text Encoder Understanding (2.5 points)
Give a brief explanation of how the final text embedding is extracted from the
transformer features.
2.3 [Code] CLIP Implementation (15 points)
Complete the forward() and clip loss functions in the ClipModel class.
2.4 [Writeup] CLIP Understanding (15 points)
Explain in detail the training objective of CLIP. Make sure to include descriptions of how the batch of image and text features are used in the computation
of the CLIP loss.
2.5 [Code, Writeup] CLIP Training (10 points)
The training code is provided for you. Train the CLIP model and report the
hyperparameters used, the final training and validation losses, and the plots of
the trajectories.
2.6 [Code, Writeup] CLIP Classification (15 points)
Use your trained CLIP model to perform classification on the training and test
sets. Report the accuracies and compare them to the VIT model’s in Question
1.