Description
Image Mosaics For Assignment 2, you will implement an image stitcher that uses image warping and homographies to automatically create an image mosaic. We will focus on the case where we have two input images that should form the mosaic, where we warp one image into the plane of the second image and display the combined views. This problem will give some practice manipulating homogeneous coordinates, computing homography matrices, and performing image warps. For simplicity, we’ll specify corresponding pairs of points manually using mouse clicks. Note: There are some built-‐in Matlab functions that could do much of the work for this project. However, to get practice with the workings of the algorithms, we want you to write your own code. Specifically, you may not use any of these functions in your implementation: cp2tform, imtransform, tformarray, tformfwd, tforminv, maketform. Provided files: Two image files that can be used for the mosaic. What You Have to Do Task 1 (15 points) Getting correspondences: Write code to get manually identified corresponding points from two views. Look at Matlab’s ginput function for an easy way to collect mouse click positions. The results will be sensitive to the accuracy of the corresponding points; when providing clicks, choose distinctive points in the image that appear in both views. Task 2 (20 points) Computing the homography parameters: Write a function that takes a set of corresponding image points and computes the associated 3 x 3 homography matrix H. This matrix transforms any point pi in one view to its corresponding homogeneous coordinates in the second view, pi’, such that λ* pi = H * pi’ Note that pi and pi’ are both vectors with 3 elements. The function should take a list of pairs of corresponding points (what is the minimum number of points you need?) from the two views, where each point is specified with its 2d image coordinates. Useful Matlab functions: ‘\’ operator (help mldivide), reshape
Verify that the homography matrix your function computes is correct by mapping the clicked image points from one view to the other, and displaying them on top of each respective image. Be sure to handle homogenous and non-‐homogenous coordinates correctly. Task 3 (20 points) Warping between image planes: Write a function that can take the recovered homography matrix and an image, and return a new image that is the warp of the input image using H . Since the transformed coordinates will typically be sub-‐pixel values, you will need to sample the pixel values from nearby pixels. Feel free to use the sampleBilinear.m you developed for the first assignment. For color images, warp each RGB channel separately and then stack together to form the output. To avoid holes in the output, use inverse warp rather than direct mapping. To compute the bounding box of the destination image, you will need to warp the points from the source image into the reference frame of the destination. Then sample all points in that destination bounding box from the proper coordinates in the source image. Note that transforming all the points will generate an image of a different shape / dimensions than the original input. It is ok to have some areas of the new image be black (0). Useful Matlab functions: round, interp2, meshgrid, isnan. Task 4 (20 points) Create the output mosaic: Once we have the source image warped into the destination image’s frame of reference, we can create a merged image showing the mosaic. Create a new image large enough to hold both (registered) views; overlay one view onto the other, simply leaving it black wherever no data is available. Don’t worry about artifacts that result at the boundaries. You are free to use a method/convention of your own choosing for the overlap areas. Task 5 (25 points) After writing and debugging your system: 1. [5 pts] Apply your system to the provided pair of images, and display the output mosaic. 2. [10 pts] Show two additional examples of mosaics you created using images that you have taken. You can make a mosaic from two or more images of a broad scene that requires a wide angle view to see well. Or, make a mosaic using two images from the same room where the same person appears in both. 3. [10 pts] Warp one image into a “frame” region in the second image. To do this, let the points from the one view be the corners of the image you want to insert in the frame, and the let the corresponding points in the second view be the clicked points
of the frame (rectangle) into which the first image should be warped. Use this idea to replace one surface in an image with an image of something else. For example -‐-‐ overwrite a billboard with a picture of your dog, or project a drawing from one image onto the street in another image, or replace a portrait on the wall with someone else’s face, or paste a Powerpoint slide onto a movie screen, … For all examples, play around a bit with the choice of points for the correspondence pairs until you get a reasonable alignment. [OPTIONAL] Extra credit Implement RANSAC for robustly estimating the homography matrix from noisy correspondences. Your function should be able to handle more than 4 point correspondences. Show with an example where it successfully gives good results even when there are outlier (bad) correspondences given as input. Compare the robust output to the original (non-‐RANSAC, only 4 point correspondences) implementation where all correspondences are used. Note: feel free to attempt using David Lowe’s SIFT Matlab demo (https://www.cs.ubc.ca/~lowe/keypoints/siftDemoV4.zip). Warning: it only works in Linux and Windows, it does not work on Mac. The result of calling the sift.m function will be 2 vectors, one of features and one of keypoints. Make sure you understand what is saved in these two variables so that you can potentially use them for the RANSAC implementation. If you can’t get SIFT to work, you will need to select make than 4 pairs of points using the function developed in Task 1. Submitting the assignment: Make sure each script or function file is well commented and it includes a block comment with your name, course number, assignment number and instructor name. Zip all the .m and image files together and submit the resulting .zip file through Moodle as Assignment 2 by Friday, September 30th, by 11:55pm. Acknowledgement: project description and test images courtesy of Kristen Grauman, University of Texas at Austin. Tips: • It can be useful when debugging to plot the corners and clicked points from one view on top of the second view after transforming them via H. Use axis([minx, maxx, miny, maxy]); to adjust the viewing window so that you can see all points. • You will need the inverse of the homography matrix to transform “backwards”.
• Be aware that Matlab’s image (matrix) indices are specified in (row,col) order, i.e., (y,x), whereas the plot and ginput functions use (col,row) order, i.e., (x,y). • Check the order of the clicked corresponding points, to make sure your code uses the intended corresponding point pairs. • As usual, be careful with how images are cast for computations and display (double vs. uint8). In particular, for your sampleBilinear.m function, be sure to pass a matrix of doubles for the image input. • When collecting your own images, be sure to either maintain the same center of projection (hold the camera at one location, but rotate between views), or else take shots of a scene with a large planar component (a building, maybe). In either case, use a static scene. Textured images that have distinctive points you can click on are good.
