Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner | SpringerLink
Skip to main content
Springer Nature Link
Log in

Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12372))

Included in the following conference series:

  • 5924 Accesses

Abstract

Remote heart rate estimation is the measurement of heart rate without any physical contact with the subject and is accomplished using remote photoplethysmography (rPPG) in this work. rPPG signals are usually collected using a video camera with a limitation of being sensitive to multiple contributing factors, e.g. variation in skin tone, lighting condition and facial structure. End-to-end supervised learning approach performs well when training data is abundant, covering a distribution that doesn’t deviate too much from the distribution of testing data or during deployment. To cope with the unforeseeable distributional changes during deployment, we propose a transductive meta-learner that takes unlabeled samples during testing (deployment) for a self-supervised weight adjustment (also known as transductive inference), providing fast adaptation to the distributional changes. Using this approach, we achieve state-of-the-art performance on MAHNOB-HCI and UBFC-rPPG.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437 (2013)

    Google Scholar 

  2. Bengio, Y., et al.: A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901.10912 (2019)

  3. Bobbia, S., Macwan, R., Benezeth, Y., Mansouri, A., Dubois, J.: Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recogn. Lett. 124, 82–90 (2019)

    Article  Google Scholar 

  4. Bousefsaf, F., Pruski, A., Maaoui, C.: 3D convolutional neural networks for remote pulse rate measurement and mapping from facial video. Appl. Sci. 9(20), 4364 (2019)

    Article  Google Scholar 

  5. Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools (2000)

    Google Scholar 

  6. Cao, W., Mirjalili, V., Raschka, S.: Rank-consistent ordinal regression for neural networks. arXiv preprint arXiv:1901.07884 (2019)

  7. Cennini, G., Arguel, J., Akşit, K., van Leest, A.: Heart rate monitoring via remote photoplethysmography with motion artifacts reduction. Opt. Express 18(5), 4867–4875 (2010)

    Article  Google Scholar 

  8. Chen, W., McDuff, D.: DeepPhys: video-based physiological measurement using convolutional attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 356–373. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_22

    Chapter  Google Scholar 

  9. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 886–893. IEEE (2005)

    Google Scholar 

  10. De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)

    Article  Google Scholar 

  11. Digiglio, P., Li, R., Wang, W., Pan, T.: Microflotronic arterial tonometry for continuous wearable non-invasive hemodynamic monitoring. Ann. Biomed. Eng. 42(11), 2278–2288 (2014)

    Article  Google Scholar 

  12. Dou, Q., de Castro, D.C., Kamnitsas, K., Glocker, B.: Domain generalization via model-agnostic learning of semantic features. In: Advances in Neural Information Processing Systems, pp. 6450–6461 (2019)

    Google Scholar 

  13. Doyle, O.M., et al.: Predicting progression of Alzheimer’s disease using ordinal regression. PloS One 9(8) (2014)

    Google Scholar 

  14. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1126–1135 (2017). JMLR.org

  15. Finn, C., Rajeswaran, A., Kakade, S., Levine, S.: Online meta-learning. arXiv preprint arXiv:1902.08438 (2019)

  16. Gidaris, S., Komodakis, N.: Dynamic few-shot visual learning without forgetting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367–4375 (2018)

    Google Scholar 

  17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  18. Hu, S.X., et al.: Empirical Bayes transductive meta-learning with synthetic gradients. In: International Conference on Learning Representations (ICLR) (2020). https://openreview.net/forum?id=Hkg-xgrYvH

  19. Jaderberg, M., et al.: Decoupled neural interfaces using synthetic gradients. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1627–1635 (2017). JMLR.org

  20. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  21. Koch, G., Zemel, R., Salakhutdinov, R.: Siamese neural networks for one-shot image recognition. In: ICML Deep Learning Workshop, vol. 2. Lille (2015)

    Google Scholar 

  22. Lee, E., Hsu, T.J., Lee, C.Y.: Centralized state sensing using sensor array on wearable device. In: 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2019)

    Google Scholar 

  23. Li, X., et al.: The OBF database: a large face video database for remote physiological signal measurement and atrial fibrillation detection. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 242–249. IEEE (2018)

    Google Scholar 

  24. Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)

    Google Scholar 

  25. Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 (2017)

  26. Liu, Y., et al.: Learning to propagate labels: transductive propagation network for few-shot learning. arXiv preprint arXiv:1805.10002 (2018)

  27. Maeda, Y., Sekine, M., Tamura, T.: The advantages of wearable green reflected photoplethysmography. J. Med. Syst. 35(5), 829–834 (2011)

    Article  Google Scholar 

  28. Menikdiwela, M., Nguyen, C., Li, H., Shaw, M.: CNN-based small object detection and visualization with feature activation mapping. In: 2017 International Conference on Image and Vision Computing New Zealand (IVCNZ), pp. 1–5. IEEE (2017)

    Google Scholar 

  29. Mishra, N., Rohaninejad, M., Chen, X., Abbeel, P.: A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141 (2017)

  30. Moço, A.V., Stuijk, S., de Haan, G.: Skin inhomogeneity as a source of error in remote PPG-imaging. Biomed. Opt. Express 7(11), 4718–4733 (2016)

    Article  Google Scholar 

  31. Munkhdalai, T., Yu, H.: Meta networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2554–2563 (2017). JMLR.org

  32. Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_29

    Chapter  Google Scholar 

  33. Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 (2018)

  34. Niu, X., Han, H., Shan, S., Chen, X.: SynRhythm: learning a deep heart rate estimator from general to specific. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585. IEEE (2018)

    Google Scholar 

  35. Niu, X., Shan, S., Han, H., Chen, X.: RhythmNet: end-to-end heart rate estimation from face via spatial-temporal representation. IEEE Trans. Image Process. (2019)

    Google Scholar 

  36. Niu, Z., Zhou, M., Wang, L., Gao, X., Hua, G.: Ordinal regression with multiple output CNN for age estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4920–4928 (2016)

    Google Scholar 

  37. Parra, D., Karatzoglou, A., Amatriain, X., Yavuz, I.: Implicit feedback recommendation via implicit-to-explicit ordinal logistic regression mapping. In: Proceedings of the CARS-2011, vol. 5 (2011)

    Google Scholar 

  38. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’ Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  39. Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2010)

    Article  Google Scholar 

  40. Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  41. Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)

    Google Scholar 

  42. Ren, J., et al.: Likelihood ratios for out-of-distribution detection. In: Advances in Neural Information Processing Systems, pp. 14680–14691 (2019)

    Google Scholar 

  43. Rettie, R., Grandcolas, U., Deakins, B.: Text message advertising: response rates and branding effects. J. Target. Meas. Anal. Mark. 13(4), 304–312 (2005)

    Article  Google Scholar 

  44. Rusu, A.A., et al.: Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 (2018)

  45. Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., Lillicrap, T.: Meta-learning with memory-augmented neural networks. In: International Conference on Machine Learning, pp. 1842–1850 (2016)

    Google Scholar 

  46. Sigrist, M.K., Taal, M.W., Bungay, P., McIntyre, C.W.: Progressive vascular calcification over 2 years is associated with arterial stiffening and increased mortality in patients with stages 4 and 5 chronic kidney disease. Clin. J. Am. Soc. Nephrol. 2(6), 1241–1248 (2007)

    Article  Google Scholar 

  47. Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, pp. 4077–4087 (2017)

    Google Scholar 

  48. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2011)

    Article  Google Scholar 

  49. Špetlík, R., Franc, V., Matas, J.: Visual heart rate estimation with convolutional neural network. In: Proceedings of the British Machine Vision Conference, Newcastle, UK, pp. 3–6 (2018)

    Google Scholar 

  50. Streifler, J.Y., Eliasziw, M., Benavente, O.R., Hachinski, V.C., Fox, A.J., Barnett, H.: Lack of relationship between leukoaraiosis and carotid artery disease. Arch. Neurol. 52(1), 21–24 (1995)

    Article  Google Scholar 

  51. Takano, C., Ohta, Y.: Heart rate measurement based on a time-lapse image. Med. Eng. Phys. 29(8), 853–857 (2007)

    Article  Google Scholar 

  52. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2396–2404 (2016)

    Google Scholar 

  53. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)

    Article  Google Scholar 

  54. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, pp. 3630–3638 (2016)

    Google Scholar 

  55. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2016)

    Article  Google Scholar 

  56. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Robust heart rate from fitness videos. Physiol. Meas. 38(6), 1023 (2017)

    Article  Google Scholar 

  57. Weersma, R.K., et al.: Molecular prediction of disease risk and severity in a large Dutch Crohn’s disease cohort. Gut 58(3), 388–395 (2009)

    Article  Google Scholar 

  58. Wu, Y., Rosca, M., Lillicrap, T.: Deep compressed sensing. arXiv preprint arXiv:1905.06723 (2019)

  59. Yu, H., et al.: Foal: fast online adaptive learning for cardiac motion estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4313–4323 (2020)

    Google Scholar 

  60. Yu, Z., Li, X., Zhao, G.: Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. In: Proceedings BMVC, pp. 1–12 (2019)

    Google Scholar 

  61. Yu, Z., Peng, W., Li, X., Hong, X., Zhao, G.: Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 151–160 (2019)

    Google Scholar 

  62. Zintgraf, L.M., Shiarlis, K., Kurin, V., Hofmann, K., Whiteson, S.: Fast context adaptation via meta-learning. arXiv preprint arXiv:1810.03642 (2018)

Download references

Acknowledgements

This work is supported by Ministry of Science and Technology (MOST) of Taiwan: 107-2221-E-009 -125 -MY3.

Author information

Authors and Affiliations

  1. Institute of Electronics, National Chiao Tung University, Hsinchu, Taiwan

    Eugene Lee, Evan Chen & Chen-Yi Lee

Authors
  1. Eugene Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Evan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chen-Yi Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eugene Lee .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 651 KB)

Supplementary material 2 (mp4 1753 KB)

Supplementary material 3 (mp4 1601 KB)

Supplementary material 4 (mp4 1715 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lee, E., Chen, E., Lee, CY. (2020). Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12372. Springer, Cham. https://doi.org/10.1007/978-3-030-58583-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58583-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58582-2

  • Online ISBN: 978-3-030-58583-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics