Abstract
Motion in the image plane is ultimately a function of 3D motion in space. We propose to compute optical flow using what is ostensibly an extreme overparameterization: depth, surface normal, and frame-to-frame 3D rigid body motion at every pixel, giving a total of 9 DoF. The advantages of such an overparameterization are twofold: first, geometrically meaningful reasoning can be called upon in the optimization, reflecting possible 3D motion in the underlying scene; second, the ‘fronto-parallel’ assumption implicit in the use of traditional matching pixel windows is ameliorated because the parameterization determines a plane-induced homography at every pixel. We show that optimization over this high-dimensional, continuous state space can be carried out using an adaptation of the recently introduced PatchMatch Belief Propagation (PMBP) energy minimization algorithm, and that the resulting flow fields compare favorably to the state of the art on a number of small- and large-displacement datasets.
Chapter PDF
Similar content being viewed by others
References
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. Intl. J. of Comp. Vis. (2011)
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (2009)
Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized patchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010)
Besse, F., Rother, C., Fitzgibbon, A., Kautz, J.: PMBP: PatchMatch belief propagation for correspondence field estimation. In: Proc. BMVC (2012)
Bleyer, M., Rhemann, C., Rother, C.: PatchMatch stereo-Stereo matching with slanted support windows. In: Proc. BMVC (2011)
Brox, T., Bregler, C., Malik, J.: Large displacement optical flow. In: Proc. CVPR (2009)
Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Intl. J. of Comp. Vis. (2012)
Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM (1981)
Hartley, R., Zisserman, A.: Multiple view geometry in computer vision, vol. 2. Cambridge University Press (2000)
Heise, P., Klose, S., Jensen, B., Knoll, A.: PM-Huber: PatchMatch with Huber regularization for stereo matching. In: Proc. CVPR (2013)
Horn, B., Schunck, B.: Determining optical flow. Artificial Intelligence (1981)
Hornáček, M., Fitzgibbon, A., Rother, C.: SphereFlow: 6 DoF scene flow from RGB-D pairs. In: Proc. CVPR (June 2014)
Li, G., Zucker, S.W.: Surface geometric constraints for stereo in belief propagation. In: Proc. CVPR (2006)
Lowe, D.: Object recognition from local scale-invariant features. In: Proc. ICCV (1999)
Mac Aodha, O., Humayun, A., Pollefeys, M., Brostow, G.: Learning a confidence measure for optical flow. IEEE T-PAMI (2012)
Moisan, L., Stival, B.: A probabilistic criterion to detect rigid point matches between two images and estimate the fundamental matrix. Intl. J. of Comp. Vis. (2004)
Morel, J.M., Yu, G.: Asift: A new framework for fully affine invariant image comparison. SIAM Journal on Imaging Sciences (2009)
Nir, T., Bruckstein, A., Kimmel, R.: Over-parameterized variational optical flow. Intl. J. of Comp. Vis. (2008)
Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE T-PAMI (2004)
Rosman, G., Shem-Tov, S., Bitton, D., Nir, T., Adiv, G., Kimmel, R., Feuer, A., Bruckstein, A.: Over-parameterized optical flow using a stereoscopic constraint. Scale Space and Variational Methods in Computer Vision (2012)
Rother, C., Kolmogorov, V., Lempitsky, V., Szummer, M.: Optimizing binary MRFs via extended roof duality. In: Proc. CVPR (2007)
Sun, D., Roth, S., Black, M.: Secrets of optical flow estimation and their principles. In: Proc. CVPR (2010)
Trobin, W., Pock, T., Cremers, D., Bischof, H.: An unbiased second-order prior for high-accuracy motion estimation. Pattern Recognition (2008)
Valgaerts, L., Bruhn, A., Weickert, J.: A variational model for the joint recovery of the fundamental matrix and the optical flow. Pattern Recognition (2008)
Vogel, C., Schindler, K., Roth, S.: Piecewise rigid scene flow. In: Proc. ICCV (2013)
Wedel, A., Pock, T., Braun, J., Franke, U., Cremers, D.: Duality TV-L1 flow with fundamental matrix prior. Image and Vision Computing (2008)
Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second-order smoothness priors. IEEE T-PAMI (2009)
Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE T-PAMI (2012)
Yedidia, J.S., Freeman, W.T., Weiss, Y.: Generalized belief propagation. In: NIPS (2000)
Yoon, K., Kweon, I.: Adaptive support-weight approach for correspondence search. IEEE T-PAMI (2006)
Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. Pattern Recognition (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Hornáček, M., Besse, F., Kautz, J., Fitzgibbon, A., Rother, C. (2014). Highly Overparameterized Optical Flow Using PatchMatch Belief Propagation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8691. Springer, Cham. https://doi.org/10.1007/978-3-319-10578-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-10578-9_15
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10577-2
Online ISBN: 978-3-319-10578-9
eBook Packages: Computer ScienceComputer Science (R0)