Abstract
In this paper, we present a method for highlight removal in facial images. In contrast to previous works relying on physical models such as dichromatic reflection models, we adopt the structure of conditional generative adversarial network (CGAN) to generate highlight-free images. By taking the facial images with specular highlight as the condition, the network predicts the corresponding highlight-free images. Meanwhile, a novel mask loss is introduced through highlight detection, which aims to make the network focus on more on the highlight regions. With the help of multi-scale discriminators, our method generates highlight-free images with high-quality details and fewer artifacts. We also built a dataset containing both real and synthetic facial images, which is, to our best knowledge, the largest image dataset for facial highlight removal. By comparing with the state-of-the-arts, our method shows high effectiveness and strong robustness in different lighting environments.
The first author is a student.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Fu, G., Zhang, Q., Song, C., Lin, Q., Xiao, C.: Specular highlight removal for real-world images. Comput. Graph. Forum 38, 253–263 (2019)
Kim, H., Jin, H., Hadap, S., Kweon, I.: Specular reflection separation using dark channel prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1460–1467 (2013)
Li, C., Lin, S., Zhou, K., Ikeuchi, K.: Specular highlight removal in facial images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3107–3116 (2017)
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)
Meslouhi, O., Kardouchi, M., Allali, H., Gadi, T., Benkaddour, Y.: Automatic detection and inpainting of specular reflections for colposcopic images. Open Comput. Sci. 1(3), 341–354 (2011)
Klinker, G.J., Shafer, S.A., Kanade, T.: The measurement of highlights in color images. Int. J. Comput. Vision 2(1), 7–32 (1988)
Schluns, K., Teschner, M.: Analysis of 2D color spaces for highlight elimination in 3d shape reconstruction. Proc. ACCV. 2, 801–805 (1995)
Bajcsy, R., Lee, S.W., Leonardis, A.: Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation. Int. J. Comput. Vision 17(3), 241–272 (1996)
Suo, J., An, D., Ji, X., Wang, H., Dai, Q.: Fast and high quality highlight removal from a single image. IEEE Trans. Image Process. 25(11), 5441–5454 (2016)
Yang, Q., Wang, S., Ahuja, N.: Real-time specular highlight removal using bilateral filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 87–100. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_7
Liu, Y., Yuan, Z., Zheng, N., Wu, Y.: Saturation-preserving specular reflection separation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3725–3733 (2015)
Guo, J., Zhou, Z., Wang, L.: Single image highlight removal with a sparse and low-rank reflection model. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 282–298. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_17
Shen, H.L., Zhang, H.G., Shao, S.J., Xin, J.H.: Chromaticity-based separation of reflection components in a single image. Pattern Recogn. 41(8), 2461–2469 (2008)
Souza, C.S., Macedo, M.C.F., Nascimento, V.P., Oliveira, B.S.: Real-time high-quality specular highlight removal using efficient pixel clustering. In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images, pp. 56–63. IEEE (2018)
Yi, R., Zhu, C., Tan, P., Lin, S.: Faces as lighting probes via unsupervised deep highlight extraction. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11213, pp. 321–338. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01240-3_20
Tan, P., Lin, S., Quan, L., Shum, H.Y.: Highlight removal by illumination constrained inpainting. In: Proceedings Ninth IEEE International Conference on Computer Vision, pp. 164–169. IEEE (2003)
Tan, P., Quan, L., Lin, S.: Separation of highlight reflections on textured surfaces. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1855–1860. IEEE (2006)
Sun, T., et al.: Single image portrait relighting. ACM Trans. Graph. 38(4), 79:1–79:12 (2019)
Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W.: Deep Single-image portrait relighting. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7194–7202 (2019)
Meka, A., et al.: Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Trans. Graph. 38(4), 1–12 (2019)
Li, Y., Liu, M.-Y., Li, X., Yang, M.-H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 468–483. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_28
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990–4998 (2017)
Character Creator. https://www.reallusion.com/character-creator/
VISIA. https://www.canfieldsci.com/imaging-systems/visia-cr/
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Kingma, D.P., Adam, J.B.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, pp. 1–15 (2014)
Shen, H.L., Cai, Q.Y.: Simple and efficient method for specularity removal in an image. Appl. Opt. 48(14), 2711–2719 (2009)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, T., Xia, S., Bian, Z., Lu, C. (2020). Highlight Removal in Facial Images. In: Peng, Y., et al. Pattern Recognition and Computer Vision. PRCV 2020. Lecture Notes in Computer Science(), vol 12305. Springer, Cham. https://doi.org/10.1007/978-3-030-60633-6_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-60633-6_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60632-9
Online ISBN: 978-3-030-60633-6
eBook Packages: Computer ScienceComputer Science (R0)