Geometry of image formation

Transkript

1 eometry of image formation Tomáš Svoboda, Czech Technical University in Prague, Center for Machine Perception Last update: July 4, 2008 Talk Outline Pinhole model Camera parameters Estimation of the parameters Camera calibration

2 Motivation 2/47 parallel lines window sizes image units distance from the camera

3 What will we learn 3/47 how does the 3D world project to 2D image plane? how is a camera modeled? how can we estimate the camera model?

4 Pinhole camera 4/47 camera

5 Camera Obscura 5/ obscura

6 Camera Obscura room-sized 6/47 Used by the art department at the UNC at Chapel Hill obscura

7 D Pinhole camera 7/47

8 D Pinhole camera projects 2D to D 8/47 image plane x f C z optical axis x Z Z 2 Z 3 x f = Z x = f Z

9 Problems with perspective I 9/47 image plane x 2 f x 2 C z Z Z 2 Z 3 optical axis = 2 x x 2

10 Problems with perspective II 0/47 image plane x 2 3 f x 23 C z optical axis Z 2 Z x 2 = x 3

11 et rid of the ( ) sign /47 image plane x image plane 2 3 f x 23 x 23 C f z Z Z 2 optical axis Z 3

12 2/47 How does the 3D world project to the 2D image plane?

13 A 3D point in a world coordinate system 3/47 z [0, 0, 0] x y C

14 A pinhole camera observes a scene 4/47 z [0, 0, 0] x y C

15 Point projects to the image plane, point x 5/47 z [0, 0, 0] x y x C

16 Scene projection 6/47 z [0, 0, 0] x y x C

17 Scene projection 7/47 z [0, 0, 0] x y C

18 3D Scene projection observations 8/47 3D lines project to 2D lines but the angles change, parallel lines are no more parallel. area ratios change, note the front and backside of the house C

19 Put the sketches into equations 9/47

20 3D 2D Projection 20/47 We remember that: x = [ f Z, fy Z ] [ x ] f fy Z [ x ] f 0 f 0 0 [ ] x Use the homegeneous coordinates 4 λ [ ] x [3 ] = K [3 3] [ I 0 ] [4 ] but... C 4 for the notation conventions, see the talk notes

21 ... we need the in camera coordinate system otate the vector: z 2/47 = ( w C w ) is a 3 3 rotation matrix. The point coordinates are now in the camera frame. Use homogeneous coordinates to get a matrix equation cam w x [0, 0, 0] y [ ] = [ Cw 0 ] [ w ] x C w The camera center C w is often replaced by the translation vector C t = C w

22 External (extrinsic parameters) The translation vector t and the rotation matrix are called External parameters of the camera. x K [ I 0 ] [ ] w z [0, 0, 0] 22/47 λx = K [ t ] [ w ] cam x y C w Camera parameters (so far): f,, t Is it all? What can we model? x C

23 What is the geometry good for? 23/47 video: Zoom out vs. motion away from scene How would you characterize the difference? Would you guess the motion type?

24 What is the geometry good for? 24/47 video: Zoom out vs. motion away from scene

25 25/47 Enough geometry 5, look at real images 5 just for a moment

26 From geometry to pixels and back again 26/

27 Problems with pixels 27/

28 Is this a stright line? 28/

29 Problems with pixels 29/

30 What are we looking at? 30/

31 Did you recognize it? 3/

32 Pixel images revisited 32/ There are no negative coordinates. Where is the principal point? Lines are not lines any more. Pixels, considered independently, do not carry much information.

33 Pixel coordinate system Assume normalized geometrical coordinates x = [x, y, ] 33/47 u = m u ( x) + u 0 v = m v y + v 0 where m u, m v are sizes of the pixels and [u 0, v 0 ] are coordinates of the principal point. u v x x [u 0, v 0 ] y C

34 Put pixels and geometry together From 3D to image coordinates: λx λy λ From normalized coordinates to pixels: Put them together: λ u v = = u v f f = fm u 0 u 0 0 fm v v [ t ] [4 ] m u 0 u 0 0 m v v [ t ] x y 34/47 Finally: u K [ t ] Introducing a 3 4 camera projection matrix P: u P

35 Non-linear distortion Several models exist. Less standardized than the linear model. We will consider a simple on-parameter radial distortion. x n denote the linear image coordinates, x d the distorted ones. 35/47 x d = ( + κr 2 )x n where κ is the distortion parameter, and r 2 = x 2 n + y 2 n is the distance from the principal point. Observable are the distorted pixel coordinates u d = Kx d Assume that we know κ. How to get the lines back?

36 Undoing adial Distortion 36/47 From pixels to distorted image coordinates: x d = K u d From distorted to linear image coordinates: x n = x d +κr 2 Where is the problem? r 2 = x 2 n + y 2 n. We have unknowns on both sides of the equation. Iterative solution:. initialize x n = x d 2. r 2 = x 2 n + y 2 n 3. compute x n = x d +κr 2 4. go to 2. (and repeat few times) And back to pixels u n = Kx n

37 Undoing adial Distortion 37/47 video

38 Estimation of camera parameters camera calibration The goal: estimate the 3 4 camera projection matrix P and possibly the parameters of the non-linear distortion κ from images. Assume a known projection [u, v] of a 3D point with known coordinates λu P λv = P Y 2 λ P Z 3 λu λ = P λv P and 3 λ = P 2 P 3 e-arrange and assume 6 λ 0 to get set of homegeneous equations u P 3 P = 0 v P 3 P 2 = 0 38/47 6 see some notes about λ = 0 in the talk notes

39 Estimation of the P matrix 39/47 u P 3 P = 0 v P 3 P 2 = 0 e-shuffle into a matrix form: [ 0 u } 0 {{ v } A [2 2] ] P P 2 P 3 } {{ } p [2 ] = 0 [2 ] A correspondece u i i forms two homogeneous equations. P has 2 parameters but scale does not matter. We need at least 6 2D 3D pairs to get a solution. We constitute A [ 2 2] data matrix and solve p = argmin Ap subject to p = which is a constrained LSQ problem. p minimizes algebraic error

40 Decomposition of P into the calibration parameters 40/47 P = [ K Kt ] and C = t We know that should be 3 3 orthonormal, and K upper triangular. P = P./norm(P(3,:3)); [K,] = rq(p(:,:3)); t = inv(k)*p(:,4); C = - *t; See the slide notes for more details.

41 An example of a calibration object 4/47

42 42/47 2D projections localized

43 eprojection for linear model 43/

44 eprojection for full model 44/

45 eprojection errors comparison between full and linear model 45/ sorted 2D reprojection errors full model linear model 4 pixels

46 eferences The book [2] is the ultimate reference. It is a must read for anyone wanting use cameras for 3D computing. Details about matrix decompositions used throughout the lecture can be found at [] [] ene H. olub and Charles F. Van Loan. Matrix Computation. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, USA, 3rd edition, 996. [2] ichard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge University, Cambridge, 2nd edition, /47

47 End 47/47

48

49

50

51

52

53

54 image plane x f x C z Z Z 2 Z 3 optical axis x f = Z x = f Z

55 image plane x 2 f x 2 C z optical axis Z Z 2 = 2 x x 2 Z 3

56 image plane x 2 3 f x 23 C z optical axis Z 2 Z x 2 = x 3

57 image plane x image plane 2 3 f x 23 x 23 C f z Z Z 2 optical axis Z 3

58 C z [0, 0, 0] x y

59 C z [0, 0, 0] x y

60 C z [0, 0, 0] x y x

61 C z [0, 0, 0] x y x

62 C z [0, 0, 0] x y

63

64 C

65 C x

66 C z w [0, 0, 0] x y cam C w x

67 C z w [0, 0, 0] x y cam C w x

68

69

70

71

72

73

74

75

76

77

78 C u v x x [u 0, v 0 ] y

79

80

81

82

83

84 sorted 2D reprojection errors full model linear model 4 pixels 3 2