Polarimetric Pose Prediction | SpringerLink
Skip to main content
Springer Nature Link
Log in

Polarimetric Pose Prediction

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13669))

Included in the following conference series:

Abstract

Light has many properties that vision sensors can passively measure. Colour-band separated wavelength and intensity are arguably the most commonly used for monocular 6D object pose estimation. This paper explores how complementary polarisation information, i.e. the orientation of light wave oscillations, influences the accuracy of pose predictions. A hybrid model that leverages physical priors jointly with a data-driven learning strategy is designed and carefully tested on objects with different levels of photometric complexity. Our design significantly improves the pose accuracy compared to state-of-the-art photometric approaches and enables object pose estimation for highly reflective and transparent objects. A new multi-modal instance-level 6D object pose dataset with highly accurate pose annotations for multiple objects with varying photometric complexity is introduced as a benchmark.

D. Gao, Y. Li, P. Ruhkamp, I. Skobleva and M. Wysocki—Equal contribution; Alphabetical order.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 12583
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 15729
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Latin for “let there be light”.

  2. 2.

    Dataset and code publicly available at: https://daoyig.github.io/PPPNet/.

References

  1. Atkinson, G.A.: Polarisation photometric stereo. Comput. Vis. Image Underst. 160, 158–167 (2017)

    Article  Google Scholar 

  2. Atkinson, G.A., Hancock, E.R.: Multi-view surface reconstruction using polarization. In: IEEE International Conference on Computer Vision (ICCV), pp. 309–316 (2005)

    Google Scholar 

  3. Atkinson, G.A., Hancock, E.R.: Recovery of surface orientation from diffuse polarization. Trans. Image Process. 15(6), 1653–1664 (2006)

    Article  Google Scholar 

  4. Ba, Y., Gilbert, A., Wang, F., Yang, J., Chen, R., Wang, Y., Yan, L., Shi, B., Kadambi, A.: Deep shape from polarization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12369, pp. 554–571. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_33

    Chapter  Google Scholar 

  5. Besl, P.J., McKay, N.D.: Method for registration of 3d shapes. In: Sensor Fusion IV: Control Paradigms and Data Structures, vol. 1611, pp. 586–606. International Society for Optics and Photonics (1992)

    Google Scholar 

  6. Birdal, T., Ilic, S.: Point pair features based object detection and pose estimation revisited. In: IEEE International Conference on 3D Vision (3DV), pp. 527–535 (2015)

    Google Scholar 

  7. Brachmann, E., Krull, A., Michel, F., Gumhold, S., Shotton, J., Rother, C.: Learning 6D object pose estimation using 3D object coordinates. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 536–551. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_35

    Chapter  Google Scholar 

  8. Brachmann, E., Michel, F., Krull, A., Yang, M.Y., Gumhold, S., et al.: Uncertainty-driven 6d pose estimation of objects and scenes from a single rgb image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3364–3372 (2016)

    Google Scholar 

  9. Busam, B., Jung, H.J., Navab, N.: I like to move it: 6d pose estimation as an action decision process. arXiv preprint arXiv:2009.12678 (2020)

  10. Cui, Z., Gu, J., Shi, B., Tan, P., Kautz, J.: Polarimetric multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1558–1567 (2017)

    Google Scholar 

  11. Cui, Z., Larsson, V., Pollefeys, M.: Polarimetric relative pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2671–2680 (2019)

    Google Scholar 

  12. Di, Y., Manhardt, F., Wang, G., Ji, X., Navab, N., Tombari, F.: So-pose: exploiting self-occlusion for direct 6d pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12396–12405 (2021)

    Google Scholar 

  13. Drost, B., Ulrich, M., Bergmann, P., Hartinger, P., Steger, C.: Introducing mvtec itodd-a dataset for 3d object recognition in industry. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 2200–2208 (2017)

    Google Scholar 

  14. Drost, B., Ulrich, M., Navab, N., Ilic, S.: Model globally, match locally: efficient and robust 3d object recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 998–1005 (2010)

    Google Scholar 

  15. Fließbach, T.: Elektrodynamik: Lehrbuch zur Theoretischen Physik II, vol. 2. Springer-Verlag (2012)

    Google Scholar 

  16. Garcia, N.M., De Erausquin, I., Edmiston, C., Gruev, V.: Surface normal reconstruction using circularly polarized light. Opt. Express 23(11), 14391–14406 (2015)

    Article  Google Scholar 

  17. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2961–2969 (2017)

    Google Scholar 

  18. He, Y., Huang, H., Fan, H., Chen, Q., Sun, J.: Ffb6d: a full flow bidirectional fusion network for 6d pose estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  19. He, Y., Sun, W., Huang, H., Liu, J., Fan, H., Sun, J.: Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  20. Heikkila, J., Silvén, O.: A four-step camera calibration procedure with implicit image correction. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1106–1112 (1997)

    Google Scholar 

  21. Hinterstoisser, S., et al.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In: Asian Conference on Computer Vision (ACCV), pp. 548–562 (2012)

    Google Scholar 

  22. Hodan, T., Barath, D., Matas, J.: Epos: estimating 6d pose of objects with symmetries. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11703–11712 (2020)

    Google Scholar 

  23. Hodan, T., Haluza, P., Obdržálek, Š., Matas, J., Lourakis, M., Zabulis, X.: T-less: An rgb-d dataset for 6d pose estimation of texture-less objects. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 880–888 (2017)

    Google Scholar 

  24. Hu, Y., Fua, P., Wang, W., Salzmann, M.: Single-stage 6d object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2930–2939 (2020)

    Google Scholar 

  25. Hu, Y., Hugonot, J., Fua, P., Salzmann, M.: Segmentation-driven 6d object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3385–3394 (2019)

    Google Scholar 

  26. Islam, M.N., Tahtali, M., Pickering, M.: Specular reflection detection and inpainting in transparent object through msplfi. Remote Sens 13(3), 455 (2021)

    Article  Google Scholar 

  27. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Depth sensing using geometrically constrained polarization normals. Int. J. Comput. Vis. (IJCV) 125(1–3), 34–51 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  28. Kalra, A., Taamazyan, V., Rao, S.K., Venkataraman, K., Raskar, R., Kadambi, A.: Deep polarization cues for transparent object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8602–8611 (2020)

    Google Scholar 

  29. Kaskman, R., Zakharov, S., Shugurov, I., Ilic, S.: Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects. In: International Conference on Computer Vision (ICCV) Workshops (2019)

    Google Scholar 

  30. Kehl, W., Manhardt, F., Tombari, F., Ilic, S., Navab, N.: Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 1521–1529 (2017)

    Google Scholar 

  31. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  32. Kundu, A., Li, Y., Rehg, J.M.: 3d-rcnn: instance-level 3d object reconstruction via render-and-compare. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3559–3568 (2018)

    Google Scholar 

  33. Labbé, Y., Carpentier, J., Aubry, M., Sivic, J.: CosyPose: consistent multi-view multi-object 6D pose estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12362, pp. 574–591. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58520-4_34

    Chapter  Google Scholar 

  34. Lei, C., Huang, X., Zhang, M., Yan, Q., Sun, W., Chen, Q.: Polarized reflection removal with perfect alignment in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1750–1758 (2020)

    Google Scholar 

  35. Li, Y., Wang, G., Ji, X., Xiang, Yu., Fox, D.: DeepIM: deep iterative matching for 6D pose estimation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 695–711. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_42

    Chapter  Google Scholar 

  36. Li, Z., Wang, G., Ji, X.: Cdpn: coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7678–7687 (2019)

    Google Scholar 

  37. Liu, X., Iwase, S., Kitani, K.M.: Stereobj-1m: large-scale stereo image dataset for 6d object pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10870–10879 (2021)

    Google Scholar 

  38. Liu, X., Jonschkowski, R., Angelova, A., Konolige, K.: Keypose: multi-view 3d labeling and keypoint estimation for transparent objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11602–11610 (2020)

    Google Scholar 

  39. Manhardt, F., et al.: Explaining the ambiguity of object detection and 6d pose from visual data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6841–6850 (2019)

    Google Scholar 

  40. Park, K., Patten, T., Vincze, M.: Pix2pose: pixel-wise coordinate regression of objects for 6d pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7668–7677 (2019)

    Google Scholar 

  41. Peng, S., Liu, Y., Huang, Q., Zhou, X., Bao, H.: Pvnet: pixel-wise voting network for 6dof pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4561–4570 (2019)

    Google Scholar 

  42. Phillips, C.J., Lecce, M., Daniilidis, K.: Seeing glassware: From edge detection to pose estimation and shape recovery. In: Robotics: Science and Systems, vol. 3 (2016)

    Google Scholar 

  43. Rad, M., Lepetit, V.: Bb8: a scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3828–3836 (2017)

    Google Scholar 

  44. Sajjan, S., et al.: Clear grasp: 3d shape estimation of transparent objects for manipulation. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3634–3642 (2020)

    Google Scholar 

  45. Saxena, A., Driemeyer, J., Ng, A.Y.: Robotic grasping of novel objects using vision. Int. J. Robot. Res. 27(2), 157–173 (2008)

    Article  Google Scholar 

  46. Smith, W.A., Ramamoorthi, R., Tozza, S.: Height-from-polarisation with unknown lighting or albedo. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 41(12), 2875–2888 (2018)

    Google Scholar 

  47. Song, C., Song, J., Huang, Q.: Hybridpose: 6d object pose estimation under hybrid representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 431–440 (2020)

    Google Scholar 

  48. Sundermeyer, M., Durner, M., Puang, E.Y., Marton, Z.C., Vaskevicius, N., Arras, K.O., Triebel, R.: Multi-path learning for object pose estimation across domains. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13916–13925 (2020)

    Google Scholar 

  49. Sundermeyer, M., Marton, Z.C., Durner, M., Brucker, M., Triebel, R.: Implicit 3d orientation learning for 6d object detection from rgb images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 699–715 (2018)

    Google Scholar 

  50. Verdie, Y., Song, J., Mas, B., Benjamin, B., Leonardis, A., McDonagh, S.: Cromo: cross-modal learning for monocular depth estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Google Scholar 

  51. Wang, C., et al.: Densefusion: 6d object pose estimation by iterative dense fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3343–3352 (2019)

    Google Scholar 

  52. Wang, G., Manhardt, F., Tombari, F., Ji, X.: Gdr-net: geometry-guided direct regression network for monocular 6d object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16611–16621 (2021)

    Google Scholar 

  53. Wang, P., et al.: Phocal: a multimodal dataset for category-level object pose estimation with photometrically challenging objects. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Google Scholar 

  54. Wohlhart, P., Lepetit, V.: Learning descriptors for object recognition and 3d pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3109–3118 (2015)

    Google Scholar 

  55. Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: Posecnn: a convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 (2017)

  56. Yu, Y., Zhu, D., Smith, W.A.: Shape-from-polarisation: a nonlinear least squares approach. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 2969–2976 (2017)

    Google Scholar 

  57. Zakharov, S., Shugurov, I., Ilic, S.: Dpod: 6d pose object detector and refiner. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1941–1950 (2019)

    Google Scholar 

  58. Zhang, Y., Morel, O., Blanchon, M., Seulin, R., Rastgoo, M., Sidibé, D.: Exploration of deep learning-based multimodal fusion for semantic road scene segmentation. In: VISIGRAPP (5: VISAPP), pp. 336–343 (2019)

    Google Scholar 

  59. Zhang, Z.: A flexible new technique for camera calibration. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 22(11), 1330–1334 (2000)

    Google Scholar 

  60. Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5745–5753 (2019)

    Google Scholar 

  61. Zhu, D., Smith, W.A.: Depth from a polarisation + rgb stereo pair. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7586–7595 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Technical University of Munich, Munich, Germany

    Daoyi Gao, Yitong Li, Patrick Ruhkamp, Iuliia Skobleva, Magdalena Wysocki, HyunJun Jung, Pengyuan Wang, Arturo Guridi & Benjamin Busam

Authors
  1. Daoyi Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yitong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patrick Ruhkamp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Iuliia Skobleva
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Magdalena Wysocki
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. HyunJun Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Pengyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Arturo Guridi
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Benjamin Busam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Ruhkamp .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4014 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gao, D. et al. (2022). Polarimetric Pose Prediction. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13669. Springer, Cham. https://doi.org/10.1007/978-3-031-20077-9_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20077-9_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20076-2

  • Online ISBN: 978-3-031-20077-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation