Abstract
Pupil segmentation extracts the pupil area of human eyes to track the eye movement. It is the first step for most of the vision-based gaze tracking systems which have attracted significant interests from both academic and industrial communities. When incorporated in commercial VR/AR devices, the accuracy, robustness and efficiency of the pupil segmentation is the fundamental for the successful usage of gaze tracking in human computer interaction, foveated rendering and etc. In this paper, we propose, KD-Eye, a lightweight vision-based pupil segmentation approach to realize accurate and efficient estimation of pupil regions. We introduce a coarse-to-fine strategy to significantly reduce the computation of the algorithm and complexity of the network. Then knowledge distillation is applied to guide the training of a lightweight student network with a large teacher network trained by a public dataset. According to our evaluation on real-world dataset and implementation on a VR platform, the coarse-to-fine strategy and lightweight network can speed up the segmentation process by over 240 times and the knowledge distillation can improve the accuracy of the student network especially when the number of training samples are limited.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Clay, V., König, P., Koenig, S.: Eye tracking in virtual reality. J. Eye Movement Res. 12, April 2019. https://doi.org/10.16910/jemr.12.1.3
Lohr, D., Griffith, H., Komogortsev, O.: Eye know you: Metric learning for end-to-end biometric authentication using eye movements from a longitudinal dataset, April 2021
Oliveira, D., et al.: Consumers’ attention to functional food labels: Insights from eye-tracking and change detection in a case study with probiotic milk. LWT Food Sci. Technol. 68, 160–167 (2016)
Gwizdka, J., Hosseini, R., Cole, M., Wang, S.: Temporal dynamics of eye-tracking and eeg during reading and relevance decisions. Journal of the Association for Information Science and Technology 68(10), 2299–2312
Zaretskaya, N., Bause, J., Polimeni, J.R., Grassi, P.R., Scheffler, K., Bartels, A.: Eye-selective fmri activity in human primary visual cortex: Comparison between 3 t and 9.4 t, and effects across cortical depth. NeuroImage 220, 117078 (2020). https://www.sciencedirect.com/science/article/pii/S1053811920305644
Valliappan, N., Dai, N., Steinberg, E., He, J., Rogers, K., Ramachandran, V., Xu, P., Shojaeizadeh, M., Guo, L., Kohlhoff, K., et al.: Accelerating eye movement research via accurate and affordable smartphone eye tracking. Nat. Commun. 11(1), 1–12 (2020)
Cheng, Y., Wang, H., Bao, Y., Lu, F.: Appearance-based gaze estimation with deep learning: A review and benchmark. arXiv preprint arXiv:2104.12668 (2021)
Morimoto, C.H., Mimica, M.R.M.: Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst. 98(1), 4–24 (2005)
Wang, X., Zhang, J., Zhang, H., Zhao, S., Liu, H.: Vision-based gaze estimation: a review. IEEE Trans. Cogn. Dev. Syst. (2021)
Kim, J., et al.: Nvgaze: an anatomically-informed dataset for low-latency, near-eye gaze estimation. In: Proceedings of the 2019 CHI Conference. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300780
Cherif, Z., Nait-Ali, A., Motsch, J., Krebs, M.: An adaptive calibration of an infrared light device used for gaze tracking. In: Proceedings of the 19th IEEE Instrumentation and Measurement Technology Conference, vol. 2, pp. 1029–1033 (2002). https://doi.org/10.1109/IMTC.2002.1007096
Wang, K., Ji, Q.: Real time eye gaze tracking with 3d deformable eye-face model. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 1003–1011 (2017). https://doi.org/10.1109/ICCV.2017.114
Guestrin, E., Eizenman, M.: General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Trans. Biomed. Eng. 53(6), 1124–1133 (2006). https://doi.org/10.1109/TBME.2005.863952
Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4511–4520 (2015)
Kwok, T.C., Kiefer, P., Schinazi, V.R., Adams, B., Raubal, M.: Gaze-guided narratives: adapting audio guide content to gaze in virtual and real environments. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019)
Mariakakis, A., Goel, M., Aumi, M.T.I., Patel, S.N., Wobbrock, J.O.: SwitchBack: using focus and saccade tracking to guide users’ attention for mobile task resumption. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 2953–2962 (2015)
Lan, G., Heit, B., Scargill, T., Gorlatova, M.: GazeGraph: Graph-based few-shot cognitive context sensing from human visual behavior. In: Proceedings of the ACM Conference on Embedded Networked Sensor Systems (SenSys). pp. 422–435 (2020)
Srivastava, N., Newn, J., Velloso, E.: Combining low and mid-level gaze features for desktop activity recognition. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2(4), 1–27 (2018)
Kim, J., et al.: Foveated AR: dynamically-foveated augmented reality display. ACM Trans. Graph. 38(4), 1–15 (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Barea, R., Boquete, L., Mazo, M., Lopez, E.: System for assisted mobility using eye movements based on electrooculography. IEEE Trans. Neural Syst. Rehabil. Eng. 10(4), 209–218 (2002). https://doi.org/10.1109/TNSRE.2002.806829
Robinson, D.A.: A method of measuring eye movement using a scieral search coil in a magnetic field. IEEE Trans. Bio-med. Electron. 10(4), 137–145 (1963). https://doi.org/10.1109/TBMEL.1963.4322822
Zhao, G., et al.: Ev-eye: rethinking high-frequency eye tracking through the lenses of event cameras. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems, vol. 36, pp. 62169–62182. Curran Associates, Inc. (2023)
Fuhl, W., Kasneci, G., Kasneci, E.: Teyed: over 20 million real-world eye images with pupil, eyelid, and iris 2d and 3d segmentations, 2d and 3d landmarks, 3d eyeball, gaze vector, and eye movement types. In: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 367–375. IEEE (2021)
Zhang, X., Park, S., Beeler, T., Bradley, D., Tang, S., Hilliges, O.: Eth-xgaze: a large scale dataset for gaze estimation under extreme head pose and gaze variation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision - ECCV 2020, pp. 365–381. Springer, Cham (2020)
Wang, K., Su, H., Ji, Q.: Neuro-inspired eye tracking with eye movement dynamics. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9823–9832 (2019)
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 658–666 (2019)
Chaudhary, A.K., et al.: RITnet: real-time semantic segmentation of the eye for gaze tracking. In: Proceedings of IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) (2019)
Acknowledgment
This work is supported by Shandong Provincial Natural Science Foundation, China, Grant No. 2022HWYQ-040.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Data Availability Statement
The datasets used in this manuscript are all publicly available. The corresponding repositories are all properly cited in the manuscript.
Disclosure of Interests
The authors declare that they have no conflict of interest.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Y., Chen, N., Zhao, G., Shen, Y. (2025). KD-Eye: Lightweight Pupil Segmentation for Eye Tracking on VR Headsets via Knowledge Distillation. In: Cai, Z., Takabi, D., Guo, S., Zou, Y. (eds) Wireless Artificial Intelligent Computing Systems and Applications. WASA 2024. Lecture Notes in Computer Science, vol 14997. Springer, Cham. https://doi.org/10.1007/978-3-031-71464-1_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-71464-1_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71463-4
Online ISBN: 978-3-031-71464-1
eBook Packages: Computer ScienceComputer Science (R0)