Patch tracking based on comparing its pixels 1

Transkript

1 Patch tracking based on comparing its pixels 1 Tomáš Svoboda, svoboda@cmp.felk.cvut.cz Czech Technical University in Prague, Center for Machine Perception Last update: March 23, 2015 Talk Outline comparing patch pixels normalized cross-correlation, ssd... KLT - gradient based optimization good features to track 1 Please note that the lecture will be accompanied be several sketches and derivations on the blackboard and few live-interactive demos in Matlab

2 What is the problem? 2/38 video: CTU campus, door of G building

3 Tracking of dense sequences camera motion 3/38 T - Template I - Image Scene static, camera moves.

4 Tracking of dense sequences object motion 4/38 T - Template I - Image Camera static, object moves.

5 Alignment of an image (patch) 5/38 Goal is to align a template image T (x) to an input image I(x). x column vector containing image coordinates [x, y]. The I(x) could be also a small subwindow within an image.

6 How to measure the alignment? What is the best criterial function? 6/38 How to find the best match, in other words, how to find extremum of the criterial function?

7 How to measure the alignment? What is the best criterial function? 6/38 How to find the best match, in other words, how to find extremum of the criterial function? Criterial function What are the desired properties (on a certain domain)?

8 How to measure the alignment? What is the best criterial function? 6/38 How to find the best match, in other words, how to find extremum of the criterial function? Criterial function What are the desired properties (on a certain domain)? convex (remember the optimization course?) discriminative...

9 Normalized cross-correlation You may know it as correlation coefficient (from statistics) 7/38 ρ X,Y = cov(x, Y ) σ X σ Y = E[(X µ X)(Y µ Y )] σ X σ Y where σ means standard deviation.

10 Normalized cross-correlation You may know it as correlation coefficient (from statistics) 7/38 ρ X,Y = cov(x, Y ) σ X σ Y = E[(X µ X)(Y µ Y )] σ X σ Y where σ means standard deviation. Having template T (k, l) and image I(x, y), r(x, y) = k k ( ) ( ) l T (k, l) T I(x + k, y + l) I(x, y) ( I(x + k, y + l) I(x, y) l ( T (k, l) T ) 2 k l ) 2

11 Normalized cross-correlation in picture 8/38 criterial function ncc

12 Normalized cross-correlation in picture 8/38 criterial function ncc well, definitely not convex but the discriminability looks promising very efficient in computation, see [3]2. 2 check also normxcorr2 in Matlab

13 Sum of squared differences 9/38 ssd(x, y) = k l (T (k, l) I(x + k, y + l)) criterial function ssd x

14 Sum of absolute differences 10/38 sad(x, y) = k l T (k, l) I(x + k, y + l) criterial function sad x

15 SAD for the door part 11/38 50 criterial function sad x

16 SAD for the door part truncated 12/38 criterial function sad_truncated Differences greater than 20 intensity levels are counted as 20.

17 Normalized cross-correlation: how it works 13/38 live demo for various patches

18 Normalized cross-correlation: tracking 14/38 video

19 Normalized cross-correlation: tracking 14/38 video What went wrong? Why did it failed? Suggestions for improvement?

20 Tracking as an optimization problem finding extrema of a criterial function... 15/38

21 Tracking as an optimization problem finding extrema of a criterial function... 15/38... sounds like an optimization problem Kanade Lucas Tomasi (KLT) tracker Iteratively minimizes sum of square differences. It is a Gauss-Newton gradient algorithm.

22 Importance in Computer Vision Firstly published in 1981 as an image registration method [4]. 16/38 Improved many times, most importantly by Carlo Tomasi [5, 6] Free implementation(s) available3. Also part of the OpenCV library4. After more than two decades, a project5 at CMU dedicated to this single algorithm and results published in a premium journal [1]. Part of plethora computer vision algorithms. Our explanation follows mainly the paper [1]. It is a good reading for those who are also interested in alternative solutions Lucas-Kanade 20 Years On id=515&menu_id=261

23 Original Lucas-Kanade algorithm I Goal is to align a template image T (x) to an input image I(x). x column vector containing image coordinates [x, y]. The I(x) could be also a small subwindow withing an image. Set of allowable warps W(x; p), where p is a vector of parameters. For translations [ ] x + p1 W(x; p) = y + p 2 W(x; p) can be arbitrarily complex The best alignment, p, minimizes image dissimilarity [I(W(x; p)) T (x)] 2 x 17/38

24 Original Lucas-Kanade algorithm II 18/38 [I(W(x; p)) T (x)] 2 x I(W(x; p) is nonlinear! The warp W(x; p) may be linear but the pixels value are, in general, non-linear. In fact, they are essentially unrelated to x. Linearization of the image: It is assumed that some p is known and best increment p is sought. The modified problem [I(W(x; p + p)) T (x)] 2 x is solved with respect to p. When found then p gets updated p p + p...

25 Original Lucas-Kanade algorithm III 19/38 x [I(W(x; p + p)) T (x)] 2 linearization by performing first order Taylor expansion 6 x [I(W(x; p)) + I W p T (x)]2 p I = [ I x, I W y ] is the gradient image computed at W(x; p). The term p is the Jacobian of the warp. 6 Detailed explanation on the blackboard.

26 Original Lucas-Kanade algorithm IV Differentiate x [I(W(x; p)) + I W p p T (x)]2 with respect to p 20/38

27 Original Lucas-Kanade algorithm IV Differentiate x [I(W(x; p)) + I W p p T (x)]2 with respect to p 20/38 2 x [ ] [ ] I W I(W(x; p)) + I W p T (x) p p setting equality to zero yields

28 Original Lucas-Kanade algorithm IV Differentiate x [I(W(x; p)) + I W p p T (x)]2 with respect to p 20/38 2 x [ ] [ ] I W I(W(x; p)) + I W p T (x) p p setting equality to zero yields p = H 1 x [ ] I W [T (x) I(W(x; p))] p where H is (Gauss-Newton) approximation of Hessian matrix. H = x [ ] [ ] I W I W p p

29 Iterate: The Lucas-Kanade algorithm Summary 21/38 1. Warp I with W(x; p) 2. Warp the gradient I with W(x; p) 3. Evaluate the Jacobian W p image I W p 4. Compute the H = x 5. Compute p = H 1 x [ at (x; p) and compute the steepest descent I W p [ ] [ I W p 6. Update the parameters p p + p until p ɛ I W p ] ] [T (x) I(W(x; p))]

30 Example of convergence 22/38 video

31 Example of convergence 23/38 Convergence video: Initial state is within the basin of attraction

32 Example of divergence 24/38 Divergence video: Initial state is outside the basin of attraction

33 Example on-line demo 25/38 Let play and see...

34 What are good features (windows) to track? 26/38

35 What are good features (windows) to track? How to select good templates T (x) for image registration, object tracking. 27/38 p = H 1 x [ ] I W [T (x) I(W(x; p))] p

36 What are good features (windows) to track? How to select good templates T (x) for image registration, object tracking. 27/38 p = H 1 x [ ] I W [T (x) I(W(x; p))] p where H is the matrix H = x [ ] [ ] I W I W p p

37 What are good features (windows) to track? How to select good templates T (x) for image registration, object tracking. 27/38 p = H 1 x [ ] I W [T (x) I(W(x; p))] p where H is the matrix H = x [ ] [ ] I W I W p p The stability of the iteration is mainly influenced by the inverse of Hessian. We can study its eigenvalues. Consequently, the criterion of a good feature window is min(λ 1, λ 2 ) > λ min (texturedness).

38 What are good features for translations? Consider translation W(x; p) = [ ] W 1 0 p = 0 1 [ x + p1 y + p 2 ]. The Jacobian is then 28/38 H = x = x = x [ I W p [ ( I x I I x y ] [ ] [ I x I I W p ] y ) 2 I I x y ( I y ) 2 ] [ I x, I x ] [ ] The image windows with varying derivatives in both directions. Homeogeneous areas are clearly not suitable. Texture oriented mostly in one direction only would cause instability for this translation.

39 What are the good points for translations? The matrix H = x ( I ) 2 I x x I I x y ( I y I y ) 2 Should have large eigenvalues. We have seen the matrix already, where? 29/38

40 What are the good points for translations? The matrix H = x ( I ) 2 I x x I I x y ( I y I y ) 2 Should have large eigenvalues. We have seen the matrix already, where? 29/38 Harris corner detector [2]! The matrix is sometimes called Harris matrix.

41 Experiments - no occlusions 30/38 video

42 Experiments - occlusions 31/38 video

43 Experiments - occlusions with dissimilarity 32/38 video

44 Experiments - object motion 33/38 video

45 Experiments door tracking 34/38 video

46 Experiments door tracking smoothed 35/38 video

47 Comparison of ncc vs KLT tracking 36/38 video

48 References 37/38 [1] Simon Baker and Iain Matthews. Lucas-Kanade 20 years on: A unifying framework. International Journal of Computer Vision, 56(3): , [2] C. Harris and M. Stephen. A combined corner and edge detection. In M. M. Matthews, editor, Proceedings of the 4th ALVEY vision conference, pages , University of Manchaster, England, September on-line copies available on the web. [3] J.P. Lewis. Fast template matching. In Vision Interfaces, pages , Extended version published on-line as "Fast Normalized Cross-Correlation" at [4] Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Conference on Artificial Intelligence, pages , August [5] Jianbo Shi and Carlo Tomasi. Good features to track. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages , [6] Carlo Tomasi and Takeo Kanade. Detection and tracking of point features. Technical Report CMU-CS , Carnegie Mellon University, April 1991.

49 End 38/38

50

51

52

53

54

55

56

57 criterial function ncc

58

59 criterial function ssd x

60

61 criterial function sad x

62

63 criterial function sad x

64

65 criterial function sad_truncated

66