Description
1 Introduction
1.1 Description
In this homework, you will finetune the SAM model to segment the tumor area fromthe MRI
images of brasins. As shown in Fig.1, there are many MRI images of brain and their masks of
tumor as groundtruth. Segment Anything Model (SAM) is a large model that can be used for
image segmentation. But in this task, we need to fine-tune the model in specific areas of braintumor segmentation. This homework will be implemented on the GPU server we provided.
Figure 1: The MRI of brain and its mask
1.2 Provided File List
The provided files for this homework including:
0-The code for SAM: https://github.com/facebookresearch/segment-anything
1-SAM: The code for SAM fine-tune
2-MRI dataset: The MRI dataset
3-HW3_Assignment.pdf: The introduction and description of homework 3. 4-Report Template: A template for report. 1.3 Submission File List
H3_code: Put the whole code you edited in one fold. HW3report.pdf: Report for homework 3.
2 Task 1: Prompt encoder of box[70points]
There are three types of sparse prompts: points, boxes, and text. In task 1, You need to prompt
the model in boxes. The boxes are the envelope of the maskes, and you can also write the code
yourself to generate the boxes. You can find the code to prompt the model in box in the code for
SAM fine-tune. You need to finetune the model on training dataset, and validate it on testing
dataset.
Figure 2: The prompt encoder of SAM
You should calculate the IOU to show the results of your tasks. You need to write codes to
calculate the IOU of the masks we provided and your model predicted. IOU(Intersection over
Union) is used to compare the similarity between the predicted output of the model and the real
ground truth. The calculation formula for IOU of two areas A and B is as follows:
IOU =
S(A ∩ B)
S(A ∪ B)
Figure 3: The IOU
Where S(A∩B) means the the area of A∩B，and S(A∪B) means the the area of A∪B.
3 Task 2: Prompt encoder of point[30points]
There are three types of sparse prompts: points, boxes, and text. In task 2, You need to prompt
the model in points. Points are generated by masks. Just one point should be generated for
each image. You need to write the code yourself to generate the points. You can also find the code
to prompt the model in points in the code for SAM fine-tune. You need to finetune the model on
training dataset, and validate it on testing dataset. You also need to calculate IOU to showyour
result.
4 Code and Report
Code:
Before editing your code, you need to use the following commands to configure the environment
on the GPU server we provided:
conda install numpy
conda install pandas
conda install scikit-image
conda install tqdm
pip install ipykernel –upgrade
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 –index-url
https://download.pytorch.org/whl/cu118
conda install matplotlib
conda install monai
For the task, you need to use pre_grey_rgb2D.py to preprocess your dataset. Then you need to
edit finetune_and_inference_tutorial_2D_dataset.ipynb to fine-tune the SAMmodel. You canfollow the tips in code annotation.
Report:
Summarize the process and results of the homework, including but not limited to:
The description of the implementation of SAM. The formulation and implementation of the prompt of SAM. The comparison between points and box of prompt. The description, formulation and implementation of your algorithm to improve IOU.
5 Discussion and Question
You are encouraged to discuss your ideas, and ask and answer questions about homework 3. Anew post for this assignment “Homework 3 Discussion” is opened in the Discussion Forumon
Canvas. If you encounter any difficulty with the assignment, try to post your problemfor help. The classmates and the course staff will try to reply.
6 Submission Instructions
1.Zip your whole edited code and your report file HW3report.pdf to a zip file named as
HW3_GoupID.zip for a group. Write the GroupID in the name of zip file and all the names and
student IDs on the cover of the report, and submit it on the team leader’s account. 2.Upload the file to “Homework 3: Segmentation” on Canvas.