Description
Introduction In this homework, we will be exploring the usefulness of homographies. Robots often deal with planes, whether in the form of walls, ground, or some other flat surface. When two cameras observe a plane, there exists a relationship between the images captured. This relationship is defined by a 3 × 3 transformation matrix, called a planar homography. A planar homography allows us to compute how a planar scene would look from second camera location, given only the first image. In fact, we can compute how images of the plane will look from any camera at any location without knowing any internal camera parameters and without actually taking the pictures, all with the planar homography. Suppose we have two cameras C1 and C2 looking at a common plane Π in 3D space. Any 3D point P on Π generates a projected 2D point located at p1 ≡ (x1, y1, 1)T on the first camera C1 and p2 ≡ (x2, y2,1 ) T on the second camera C2. Since P is confined to the plane Π, we expect that there is a relationship between p1 and p2. In particular, there exists a common 3 × 3 matrix H, so that for any p1 and p2, the following conditions holds: p1 ≡ Hp2 (1) We call this relationship planar homography. Recall that both p1 and p2 are in homogeneous coordinates and the equality ≡ means p1 is proportional to Hp2. 1 Planar Homographies: Theory (40pts) The Q1 questions should not take more than a couple of lines each. In particular, the “proofs” do not require any lengthy calculations. If you are lost in many lines of complicated algebra you are doing something much too complicated (or wrong). Question 1.1 (10pts) Prove that there exists an H that satisfies Equation 1 given two 3 × 4 camera projection matrices M1 and M2 corresponding to cameras C1, C2 and a plane Π. You do not need to produce an actual algebraic expression for H. All we are asking for is a proof of the existence of H. Note: A degenerate case happens when the plane Π contains both cameras centers, in which case there are infinite choices of H satisfying Equation 1. You can ignore this special case in your answer. Question 1.2 (10pts) Prove that there exists an H that satisfies Equation 1 given two cameras separated by a pure rotation. That is, for C1, p1 = K1[I 0]P and for C2, p2 = K2[R 0]P . Note that K is not the same for the two cameras. Question 1.3 (5pts) Suppose that a camera is rotating about its center C, keeping the intrinsic parameters K constant. Let H be the homography that maps the view from one camera orientation to the view at a second orientation. Let θ be the angle of rotation between the two. Show that H2 is the homography corresponding to a rotation of 2θ. 2 Question 1.4 (5pts) Why is the planar homography not sufficient to map any arbitrary scene image to another viewpoint? State your answer concisely in one or two sentences. Question 1.5 (10pts) We have a set of points p i 1 in an image taken by camera C1 and points p i 2 in an image taken by C2. Suppose we know there exists an unknown homography H such that p i 1 ≡ Hpi 2 (2) Assume the points are homogeneous coordinates in the form p i j = (x i j , yi j , 1)T . For a single point pair, write a matrix equation of the form Ah = 0 (3) Where h is a vector of the elements of H and A is a matrix composed of the point coordinates. 1. How many elements are there in h? 2. How many point pairs are required to solve this system? 3. Show how to estimate the elements in h using the Rayleigh Quotient Theorem mentioned in class to find a solution to minimize this homogeneous linear least squares system. 2 Planar Homographies: Implementation (30pts) Question 2.1 (15pts) Implement a function H2to1=computeH(p1, p2). Inputs: p1 and p2 should be 2×N matrices of corresponding (x, y) T coordinates between two images. Outputs: H2to1 should be a 3 × 3 matrix encoding the homography that best matches the linear equation derived above in Equation 2 (in the least squares sense). Hint: Remember that a homography is only determined up to scale. The Matlab function eig will be useful. Question 2.2 Mom! I am on TV!! (15pts) Have you been to a Pirates game at PNC park? Have you ever wondered how cool it would be to have your face show up on the big screen as shown in Figure 1a1 ? With what you have learned so far, this is no longer a fantasy. Assuming the the big screen is planar (which it most likely is), there exist a homography between the plane of the big screen and any (planar) image. Therefore, if we are able to find this homography, then we can map an image on your computer to the big screen as shown in Figure 1d. Please do the following: a) Create an image with at least your name on it. You can modify the provided pnc tomap.jpg as shown in Figure 1b. Please save this image as pnc tomap.jpg. 1Photo taken by Shoou-I Yu, Padres v.s. Pirates, 8/10/2012. 3 b) Find the point correspondences p1 and p2 (2×N matrices) between the big screen of PNC park and the image with your name. You can use Matlabs cpselect to select matching points on each image. Hint: What is the minimum amount of point correspondences required to compute a homography? Which point correspondences should you choose? Please save p1 and p2 in Q4.2.p1p2.mat with this command: save(Q4.2.p1p2.mat, p1, p2). c) Implement a function: [img_yourname_warped, img_PNCpark_yourname] = warp2PNCpark(img_PNCpark, img_yourname, p1, p2) which takes Figure 1a (img PNCpark) and Figure 1b (img yourname) as input, and outputs Figure 1c (img yourname warped) and Figure 1d (img PNCpark yourname). The point correspondences p1 and p2 are the ones you found in the previous step. Apply your computeH to compute a homography H. Then use the provided warpH function to warp the image with your name to the coordinates of the big screen of PNC park as shown in Figure 1c. warp im=warpH(im, H, out size, fill value) warps image im using the homography transform H. In this function, the pixels in warp im are sampled at coordinates in the rectangle (1, 1) to (out size(2), out size(1)). The coordinates of the pixels in the source image are taken to be (1, 1) to (size(im,2), size(im,1)) and transformed according to H. The empty regions of warp im are filled with fill value. Setting an appropriate fill value can help you quickly find empty regions. The final step will be to combine the output of warpH with img PNCpark to get the final output. A common problem: There are different data types to represent images for Matlab. When you read an image with imread, the data type used is usually uint8. The range of values for uint8 is 0 to 255, so negative numbers are not available for the uint8 type. If you want to have negative numbers in your matrix, then you need to use a data type of single or double. However, if you are using single or double typed images, then for imshow to display correctly, the values of the image should be between 0 and 1. On the other hand, if you want to use imshow to view an uint8 image, the values of the image should range from 0 to 255. You can convert between single and uint8 data types with the functions im single = single(im) and im uint8 = uint8(im). d) Run the provided q42checker.m. The q42checker.m makes sure that the format of the functions and files follow the specifications of this homework. Please make sure that your code can run through the checker without any errors. The checker will generate an image with a structure very similar to Figure 1d. There will be additional crosses on the images in the top row showing the point correspondences p1 and p2. Please include this image in your PDF submission. Make sure that the resolution is HIGH ENOUGH (at least 1280 x 908) for the TAs to clearly see the image. 3 Panoramas (30pts) We can also use homographies to create a panorama image from multiple views of the same scene. This is possible for example when there is no camera translation between the views (e.g., only rotation about the camera center), as we saw in Question 1.2. You will need to use the provided warpH function again which you used in Question 2.2(c). Use the provided q5checker.m, and make sure that your code runs without any error. 4 (a) Big screen showing players name (b) Image with your name (c) Image with your name warped (d) Big screen with Shoou’s name Question 3.1 (15pts) For this problem, use the provided images taj1r.jpg and taj2r.jpg along with the data in tajPts.mat. Use Matlabs imread function to load an image and the load function to load a mat file. tajPts is a 4 × 1048 matrix containing the coordinates of features points in taj1 in the first two rows and matching feature points for taj2 in the bottom two rows. You can visualize a sample of these correspondences using plotMatches(img1,img2,pts) Within a few weeks, we will see how to automatically generate them. Figure 2: Matched features from tajPts.mat. a) Apply your computeH using these correspondences to compute H2to1, which is the homography from taj2 onto taj1. To validate your result, try to warp all the feature points in taj2 to the image coordinate of the taj1. And check the error between corresponding points by calculating the 2D distance between them. Save H2to1 as q5 1.mat using Matlabs save function. Save all the transformed feature points of taj1 by H2to1 as q5 1 warpedFeatures.mat using Matlabs save function. Note that this should be a 2 × 1048 matrix. Also, write the Root Mean Squared Error between corresponding points in your PDF submission. 5 Figure 3: Original images and warped (a) taj1.jpg (b) taj2.jpg (c) taj2 warped into taj1 b) Apply the computed homography H2to1 to taj2 using warpH. Note that since the warped image will be translated to the right, you will need a larger target image. Use outSize = [size(taj1,1),3000]; Save this figure as q5 1.jpg. Produce a simple panorama by laying taj1 on top of the warped taj2. Save this figure as q5 1 pan.jpg. Save your code in a Matlab function [H2to1,warpedImg,panoImg] = q5 1(img1, img2, pts) which accepts the images and feature points and returns homography, warped image, and panorama image. Question 3.2 (15pts) Suppose we want to display all the visible pixels in the panorama. Write a few lines of Matlab code to find a matrix M, and out size in terms of H2to1 and size(im1) and size(im2) such that, warp_im1=warpH(im1, M, out_size); warp_im2=warpH(im2, M*H2to1, out_size); produces images in a common reference frame where all points in im1 and im2 are visible and out size(2)=1280 (out size(1) should be determined by out size(2)). The matrix M should perform scaling and translation only, and the resulting image should fit snugly around the corners of the panorama. Do not select M manually. It should be computed by your code, considering the size of the panorama image. Save this image as q5 2 pan.jpg. Save the code used to generate and display the panorama as [H2to1, panoImg] = q5 2(img1, img2, pts) where img1 is taj1 and img2 is taj2. 6 Figure 4: Final mosaic view. Border lines are only for reference and need not be present in your solution. 7