Learning Representations for Masked Facial Recovery | SpringerLink
Skip to main content
Springer Nature Link
Log in

Learning Representations for Masked Facial Recovery

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13598))

Included in the following conference series:

Abstract

The pandemic of these very recent years has led to a dramatic increase in people wearing protective masks in public venues. This poses obvious challenges to the pervasive use of face recognition technology that now is suffering a decline in performance. One way to address the problem is to revert to face recovery methods as a preprocessing step. Current approaches to face reconstruction and manipulation leverage the ability to model the face manifold, but tend to be generic. We introduce a method that is specific for the recovery of the face image from an image of the same individual wearing a mask. We do so by designing a specialized GAN inversion method, based on an appropriate set of losses for learning an unmasking encoder. With extensive experiments, we show that the approach is effective at unmasking face images. In addition, we also show that the identity information is preserved sufficiently well to improve face verification performance based on several face recognition benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9151
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11439
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: how to embed images into the styleGAN latent space? In: IEEE ICCV, pp. 4432–4441 (2019)

    Google Scholar 

  2. Abdal, R., Zhu, P., Mitra, N.J., Wonka, P.: Labels4Free: unsupervised segmentation using StyleGAN. In: IEEE ICCV, pp. 13970–13979 (2021)

    Google Scholar 

  3. Anwar, A., Raychowdhury, A.: Masked face recognition for secure authentication. arXiv preprint arXiv:2008.11104 (2020)

  4. Cabani, A., Hammoudi, K., Benhabiles, H., Melkemi, M.: MaskedFace-Net-a dataset of correctly/incorrectly masked face images in the context of COVID-19. Smart Health 19, 100144 (2021)

    Article  Google Scholar 

  5. Chen, Y.A., Chen, W.C., Wei, C.P., Wang, Y.C.F.: Occlusion-aware face inpainting via generative adversarial networks. In: IEEE ICIP, pp. 1202–1206. IEEE (2017)

    Google Scholar 

  6. Cheng, L., Wang, J., Gong, Y., Hou, Q.: Robust deep auto-encoder for occluded face recognition. In: ACM International Conference on Multimedia, pp. 1099–1102 (2015)

    Google Scholar 

  7. Daras, G., Dean, J., Jalal, A., Dimakis, A.G.: Intermediate layer optimization for inverse problems using deep generative models. arXiv preprint arXiv:2102.07364 (2021)

  8. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. In: IEEE CVPR, pp. 4690–4699 (2019)

    Google Scholar 

  9. Deng, Y., Dai, Q., Zhang, Z.: Graph Laplace for occluded face completion and recognition. IEEE TIP 20(8), 2329–2338 (2011)

    MathSciNet  MATH  Google Scholar 

  10. Din, N.U., Javed, K., Bae, S., Yi, J.: A novel GAN-based network for unmasking of masked face. IEEE Access 8, 44276–44287 (2020)

    Article  Google Scholar 

  11. Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial feature learning. arXiv preprint arXiv:1605.09782 (2016)

  12. Fidler, S., Skocaj, D., Leonardis, A.: Combining reconstructive and discriminative subspace methods for robust classification and regression by subsampling. IEEE PAMI 28(3), 337–350 (2006)

    Article  Google Scholar 

  13. Gao, R., Grauman, K.: From one-trick ponies to all-rounders: on-demand learning for image restoration. arXiv preprint arXiv:1612.01380 (2016)

  14. Ge, S., Li, C., Zhao, S., Zeng, D.: Occluded face recognition in the wild by identity-diversity inpainting. IEEE T-CSVT 30(10), 3387–3397 (2020)

    Google Scholar 

  15. Ge, S., Li, J., Ye, Q., Luo, Z.: Detecting masked faces in the wild with LLE-CNNs. In: IEEE CVPR, pp. 2682–2690 (2017)

    Google Scholar 

  16. Ghosh, P., Zietlow, D., Black, M.J., Davis, L.S., Hu, X.: InvGAN: invertable GANs. arXiv preprint arXiv:2112.04598 (2021)

  17. He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGan: facial attribute editing by only changing what you want. IEEE TIP 28(11), 5464–5478 (2019)

    MathSciNet  MATH  Google Scholar 

  18. Horé, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: ICPR, pp. 2366–2369 (2010)

    Google Scholar 

  19. Hu, B., Zheng, Z., Liu, P., Yang, W., Ren, M.: Unsupervised eyeglasses removal in the wild. IEEE Trans. Cybern. 51(9), 4373–4385 (2020)

    Article  Google Scholar 

  20. Huang, G.B., Mattar, M., Lee, H., Learned-Miller, E.: Learning to align from scratch. In: NIPS (2012)

    Google Scholar 

  21. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (ToG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  22. Iliadis, M., Wang, H., Molina, R., Katsaggelos, A.K.: Robust and low-rank representation for fast face identification with occlusions. IEEE TIP 26(5), 2203–2218 (2017)

    MathSciNet  MATH  Google Scholar 

  23. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE CVPR, pp. 4401–4410 (2019)

    Google Scholar 

  24. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: IEEE CVPR, pp. 8110–8119 (2020)

    Google Scholar 

  25. Leonardis, A., Bischof, H.: Robust recognition using eigenimages. Comput. Vis. Image Underst. 78(1), 99–118 (2000)

    Article  MATH  Google Scholar 

  26. Li, C., Ge, S., Zhang, D., Li, J.: Look through masks: towards masked face recognition with de-occlusion distillation. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 3016–3024 (2020)

    Google Scholar 

  27. Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: IEEE CVPR, pp. 3911–3919 (2017)

    Google Scholar 

  28. Li, Y., Feng, J.: Reconstruction based face occlusion elimination for recognition. Neurocomputing 101, 68–72 (2013)

    Article  Google Scholar 

  29. Li, Z., Hu, Y., He, R., Sun, Z.: Learning disentangling and fusing networks for face completion under structured occlusions. Pattern Recogn. 99, 107073 (2020)

    Article  Google Scholar 

  30. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: ECCV 2018, pp. 85–100 (2018)

    Google Scholar 

  31. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE ICCV, pp. 3730–3738 (2015)

    Google Scholar 

  32. Luan, X., Fang, B., Liu, L., Yang, W., Qian, J.: Extracting sparse error of robust PCA for face recognition in the presence of varying illumination and occlusion. Pattern Recogn. 47(2), 495–508 (2014)

    Article  Google Scholar 

  33. Ma, X., Zhou, X., Huang, H., Jia, G., Chai, Z., Wei, X.: Contrastive attention network with dense field estimation for face completion. Pattern Recogn. 124, 108465 (2022)

    Article  Google Scholar 

  34. Menon, S., Damian, A., Hu, S., Ravi, N., Rudin, C.: Pulse: self-supervised photo upsampling via latent space exploration of generative models. In: IEEE CVPR, pp. 2437–2445 (2020)

    Google Scholar 

  35. Mishra, S., Majumdar, P., Dosi, M., Vatsa, M., Singh, R.: Dual sensor Indian masked face dataset. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–8. IEEE (2021)

    Google Scholar 

  36. Ngan, M., Grother, P., Hanaoka, K.: Ongoing face recognition vendor test (FRVT) part 6a: face recognition accuracy with masks using pre-COVID-19 algorithms. Technical report, NIST (2020)

    Google Scholar 

  37. Nitzan, Y., Bermano, A., Li, Y., Cohen-Or, D.: Face identity disentanglement via latent space mapping. arXiv preprint arXiv:2005.07728 (2020)

  38. Park, J.S., Oh, Y.H., Ahn, S.C., Lee, S.W.: Glasses removal from facial image using recursive error compensation. IEEE PAMI 27(5), 805–811 (2005)

    Article  Google Scholar 

  39. Pernuš, M., Štruc, V., Dobrišek, S.: High resolution face editing with masked GAN latent code optimization. arXiv preprint arXiv:2103.11135 (2021)

  40. Pidhorskyi, S., Adjeroh, D.A., Doretto, G.: Adversarial latent autoencoders. In: IEEE CVPR, pp. 14104–14113 (2020)

    Google Scholar 

  41. Richardson, E., et al.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: IEEE CVPR, pp. 2287–2296 (2021)

    Google Scholar 

  42. Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744 (2021)

  43. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: IEEE CVPR, pp. 815–823 (2015)

    Google Scholar 

  44. Shukor, M., Yao, X., Damodaran, B.B., Hellier, P.: Semantic and geometric unfolding of StyleGAN latent space. arXiv preprint arXiv:2107.04481 (2021)

  45. Su, Y., Yang, Y., Guo, Z., Yang, W.: Face recognition with occlusion. In: IAPR Asian Conference on Pattern Recognition (ACPR), pp. 670–674. IEEE (2015)

    Google Scholar 

  46. Trigueros, D.S., Meng, L., Hartnett, M.: Enhancing convolutional neural networks for face recognition with occlusion maps and batch triplet loss. Image Vis. Comput. 79, 99–108 (2018)

    Article  Google Scholar 

  47. Wang, M., Hu, Z., Sun, Z., Zhao, S., Sun, M.: Varying face occlusion detection and iterative recovery for face recognition. J. Electron. Imaging 26(3), 033009 (2017)

    Article  Google Scholar 

  48. Wang, Z., et al.: Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020)

  49. Wei, T., et al.: A simple baseline for StyleGAN inversion. arXiv preprint arXiv:2104.07661 (2021)

  50. Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

    Google Scholar 

  51. Xinyi, Z., Runqing, J., Tianxiang, H., Hao, Y.: Identity preserving face completion with landmark based generative adversarial network (2021)

    Google Scholar 

  52. Xiong, W., et al.: Foreground-aware image inpainting. In: IEEE CVPR, pp. 5840–5848 (2019)

    Google Scholar 

  53. Zeng, D., Veldhuis, R., Spreeuwers, L.: A survey of face recognition techniques under occlusion. IET Biometrics 10(6), 581–606 (2021)

    Article  Google Scholar 

  54. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE CVPR, pp. 586–595 (2018)

    Google Scholar 

  55. Zhao, F., Feng, J., Zhao, J., Yang, W., Yan, S.: Robust LSTM-autoencoders for face de-occlusion in the wild. IEEE TIP 27, 778–790 (2016)

    MathSciNet  MATH  Google Scholar 

  56. Zhao, S., Hu, Z.P.: A modular weighted sparse representation based on fisher discriminant and sparse residual for face recognition with occlusion. Inf. Proc. Lett. 115(9), 677–683 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  57. Zhu, P., Abdal, R., Qin, Y., Femiani, J., Wonka, P.: Improved StyleGAN embedding: where are the good Latents? arXiv preprint arXiv:2012.09036 (2020)

Download references

Acknowledgements

This material is based upon work supported in part by the Center for Identification Technology Research and the National Science Foundation under Grants No. 1650474 and No. 1920920.

Author information

Authors and Affiliations

  1. West Virginia University, Morgantown, WV, 26506, USA

    Zaigham A. Randhawa, Shivang Patel, Donald A. Adjeroh & Gianfranco Doretto

Authors
  1. Zaigham A. Randhawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shivang Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donald A. Adjeroh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gianfranco Doretto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gianfranco Doretto .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. University of Illinois Urbana-Champaign, Urbana, IL, USA

    Bo Li

  3. National University of Singapore, Singapore, Singapore

    Angela Yao

  4. Microsoft Research Asia, Beijing, China

    Yang Liu

  5. University of Missouri, Columbia, MO, USA

    Ye Duan

  6. City University of Hong Kong, Kowloon, Hong Kong

    Manfred Lau

  7. Idaho National Laboratory, Idaho Falls, ID, USA

    Rajiv Khadka

  8. Salesforce, Seattle, WA, USA

    Ana Crisan

  9. Tufts University, Medford, MA, USA

    Remco Chang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Randhawa, Z.A., Patel, S., Adjeroh, D.A., Doretto, G. (2022). Learning Representations for Masked Facial Recovery. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13598. Springer, Cham. https://doi.org/10.1007/978-3-031-20713-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20713-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20712-9

  • Online ISBN: 978-3-031-20713-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics