Video-based AI for beat-to-beat assessment of cardiac function | Nature
Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Video-based AI for beat-to-beat assessment of cardiac function

Nature volume 580, pages 252–256 (2020)Cite this article

Subjects

Abstract

Accurate assessment of cardiac function is crucial for the diagnosis of cardiovascular disease1, screening for cardiotoxicity2 and decisions regarding the clinical management of patients with a critical illness3. However, human assessment of cardiac function focuses on a limited sampling of cardiac cycles and has considerable inter-observer variability despite years of training4,5. Here, to overcome this challenge, we present a video-based deep learning algorithm—EchoNet-Dynamic—that surpasses the performance of human experts in the critical tasks of segmenting the left ventricle, estimating ejection fraction and assessing cardiomyopathy. Trained on echocardiogram videos, our model accurately segments the left ventricle with a Dice similarity coefficient of 0.92, predicts ejection fraction with a mean absolute error of 4.1% and reliably classifies heart failure with reduced ejection fraction (area under the curve of 0.97). In an external dataset from another healthcare system, EchoNet-Dynamic predicts the ejection fraction with a mean absolute error of 6.0% and classifies heart failure with reduced ejection fraction with an area under the curve of 0.96. Prospective evaluation with repeated human measurements confirms that the model has variance that is comparable to or less than that of human experts. By leveraging information across multiple cardiac cycles, our model can rapidly identify subtle changes in ejection fraction, is more reproducible than human evaluation and lays the foundation for precise diagnosis of cardiovascular disease in real time. As a resource to promote further innovation, we also make publicly available a large dataset of 10,030 annotated echocardiogram videos.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: EchoNet-Dynamic workflow.
Fig. 2: Model performance.
Fig. 3: Beat-to-beat evaluation of the ejection fraction.

Similar content being viewed by others

Data availability

This project introduces the EchoNet-Dynamic dataset, a publicly available dataset of de-identified echocardiogram videos, which are available at https://echonet.github.io/dynamic/.

Code availability

The code for EchoNet-Dynamic is available at https://github.com/echonet/dynamic.

References

  1. Ziaeian, B. & Fonarow, G. C. Epidemiology and aetiology of heart failure. Nat. Rev. Cardiol. 13, 368–378 (2016).

    Article  Google Scholar 

  2. Shakir, D. K. & Rasul, K. I. Chemotherapy induced cardiomyopathy: pathogenesis, monitoring and management. J. Clin. Med. Res. 1, 8–12 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  3. Dellinger, R. P. et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 39, 165–228 (2013).

    Article  CAS  Google Scholar 

  4. Farsalinos, K. E. et al. Head-to-head comparison of global longitudinal strain measurements among nine different vendors: the EACVI/ASE Inter-Vendor Comparison Study. J. Am. Soc. Echocardiogr. 28, 1171–1181 (2015).

    Article  Google Scholar 

  5. Lang, R. M. et al. Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. Eur. Heart J. Cardiovasc. Imaging 16, 233–271 (2015).

    Article  Google Scholar 

  6. McMurray, J. J. et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2012. Eur. J. Heart Fail. 14, 803–869 (2012).

    Article  CAS  Google Scholar 

  7. Loehr, L. R., Rosamond, W. D., Chang, P. P., Folsom, A. R. & Chambless, L. E. Heart failure incidence and survival (from the Atherosclerosis Risk in Communities study). Am. J. Cardiol. 101, 1016–1022 (2008).

    Article  Google Scholar 

  8. Bui, A. L., Horwich, T. B. & Fonarow, G. C. Epidemiology and risk profile of heart failure. Nat. Rev. Cardiol. 8, 30–41 (2011).

    Article  Google Scholar 

  9. Roizen, M. F. Forecasting the future of cardiovascular disease in the United States: a policy statement from the American Heart Association. Yearbook Anesthesiol. Pain Manage. 2012, 12–13 (2012).

    Article  Google Scholar 

  10. Yancy, C. W. et al. 2013 ACCF/AHA guideline for the management of heart failure. Circulation 128, e240–e327 (2013).

    PubMed  Google Scholar 

  11. Huang, H. et al. Accuracy of left ventricular ejection fraction by contemporary multiple gated acquisition scanning in patients with cancer: comparison with cardiovascular magnetic resonance. J. Cardiovasc. Magn. Reson. 19, 34 (2017).

    Article  Google Scholar 

  12. Pellikka, P. A. et al. Variability in ejection fraction measured by echocardiography, gated single-photon emission computed tomography, and cardiac magnetic resonance in patients with coronary artery disease and left ventricular dysfunction. JAMA Netw. Open 1, e181456 (2018).

    Article  Google Scholar 

  13. Malm, S., Frigstad, S., Sagberg, E., Larsson, H. & Skjaerpe, T. Accurate and reproducible measurement of left ventricular volume and ejection fraction by contrast echocardiography: a comparison with magnetic resonance imaging. J. Am. Coll. Cardiol. 44, 1030–1035 (2004).

    Article  Google Scholar 

  14. Cole, G. D. et al. Defining the real-world reproducibility of visual grading of left ventricular function and visual estimation of left ventricular ejection fraction: impact of image quality, experience and accreditation. Int. J. Cardiovasc. Imaging 31, 1303–1314 (2015).

    Article  Google Scholar 

  15. Koh, A. S. et al. A comprehensive population-based characterization of heart failure with mid-range ejection fraction. Eur. J. Heart Fail. 19, 1624–1634 (2017).

    Article  Google Scholar 

  16. Chioncel, O. et al. Epidemiology and one-year outcomes in patients with chronic heart failure and preserved, mid-range and reduced ejection fraction: an analysis of the ESC Heart Failure Long-Term Registry. Eur. J. Heart Fail. 19, 1574–1585 (2017).

    Article  CAS  Google Scholar 

  17. Shah, K. S. et al. Heart failure with preserved, borderline, and reduced ejection fraction: 5-year outcomes. J. Am. Coll. Cardiol. 70, 2476–2486 (2017).

    Article  Google Scholar 

  18. Papolos, A., Narula, J., Bavishi, C., Chaudhry, F. A. & Sengupta, P. P. U.S. hospital use of echocardiography: insights from the nationwide inpatient sample. J. Am. Coll. Cardiol. 67, 502–511 (2016).

    Article  Google Scholar 

  19. Douglas, P. S. et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 Appropriate use criteria for echocardiography. J. Am. Soc. Echocardiogr. 24, 229–267 (2011).

    Article  Google Scholar 

  20. Zhang, J. et al. Fully automated echocardiogram interpretation in clinical practice: feasibility and diagnostic accuracy. Circulation 138, 1623–1635 (2018).

    Article  Google Scholar 

  21. Madani, A., Arnaout, R., Mofrad, M. & Arnaout, R. Fast and accurate view classification of echocardiograms using deep learning. NPJ Digit. Med. 1, 6 (2018).

    Article  Google Scholar 

  22. Ghorbani, A. et al. Deep learning interpretation of echocardiograms. NPJ Digit. Med. 3, 10 (2020).

    Article  Google Scholar 

  23. Behnami, D. et al. in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support 65–73 (Springer, 2018).

  24. Ardila, D. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25, 954–961 (2019).

    Article  CAS  Google Scholar 

  25. Poplin, R. et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158–164 (2018).

    Article  Google Scholar 

  26. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  ADS  CAS  Google Scholar 

  27. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

    Article  CAS  Google Scholar 

  28. Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H. Rethinking atrous convolution for semantic image segmentation. Preprint at https://arxiv.org/abs/1706.05587 (2017).

  29. Tran, D. et al. A closer look at spatiotemporal convolutions for action recognition. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 6450–6459 (2018).

  30. Tran, D., Bourdev, L., Fergus, R., Torresani, L. & Paluri, M. Learning spatiotemporal features with 3D convolutional networks. In Proc. IEEE International Conference on Computer Vision 4489–4497 (2015).

  31. Kay, W. et al. The kinetics human action video dataset.Preprint at https://arxiv.org/abs/1705.06950 (2017).

Download references

Acknowledgements

This work is supported by a Stanford Translational Research and Applied Medicine pilot grant, a Stanford Cardiovascular Institute pilot grant and a Stanford Artificial Intelligence in Imaging and Medicine Center seed grant. D.O. is supported by the American College of Cardiology Foundation – Merck Research Fellowship and NIH F32HL149298. B.H. is supported by a NSF Graduate Research Fellowship. A.G. is supported by the Stanford Robert Bosch Graduate Fellowship in Science and Engineering. J.Y.Z. is supported by NSF CCF 1763191, NIH R21 MD012867-01, NIH P30AG059307 and by a Chan-Zuckerberg Biohub Fellowship.

Author information

Author notes

  1. These authors jointly supervised this work: Euan A. Ashley, James Y. Zou

Authors and Affiliations

  1. Department of Medicine, Stanford University, Stanford, CA, USA

    David Ouyang, Curtis P. Langlotz, Paul A. Heidenreich, Robert A. Harrington, David H. Liang & Euan A. Ashley

  2. Department of Computer Science, Stanford University, Stanford, CA, USA

    Bryan He & James Y. Zou

  3. Department of Electrical Engineering, Stanford University, Stanford, CA, USA

    Amirata Ghorbani, David H. Liang & James Y. Zou

  4. Smidt Heart Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA

    Neal Yuan & Joseph Ebinger

  5. Department of Radiology, Stanford University, Stanford, CA, USA

    Curtis P. Langlotz

  6. Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

    Euan A. Ashley & James Y. Zou

Authors
  1. David Ouyang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Bryan He
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Amirata Ghorbani
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Neal Yuan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Joseph Ebinger
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Curtis P. Langlotz
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Paul A. Heidenreich
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Robert A. Harrington
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. David H. Liang
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Euan A. Ashley
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. James Y. Zou
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

D.O. retrieved, preprocessed and quality-controlled Stanford videos and merged electronic medical record data. D.O., B.H., A.G. and J.Y.Z. developed and trained the deep learning algorithms, performed statistical tests and created all of the figures. D.O., C.P.L., P.A.H. and R.A.H. coordinated the public release of the de-identified echocardiogram dataset. D.O., P.A.H., D.H.L. and E.A.A. performed the clinical evaluation of model performance. N.Y. and J.E. retrieved, preprocessed and quality-controlled data from Cedar-Sinai Medical Center for model testing. D.O., B.H., E.A.A. and J.Y.Z. wrote the manuscript with feedback from all authors.

Corresponding authors

Correspondence to David Ouyang or James Y. Zou.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature thanks Giorgio Quer, Partho Sengupta and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Hyperparameter search for spatiotemporal convolutions on the video dataset to predict ejection fraction.

Model architecture (R2+1D, which is the architecture selected by EchoNet-Dynamic for ejection fraction prediction, R3D and MC3), initialization (solid line, Kinetics-400 pretrained weights; dotted line, random initial weights), clip length (1, 8, 16, 32, 64, 96 and all frames) and sampling period (1, 2, 4, 6 and 8) were considered. a, When varying clip lengths, performance is best at 64 frames (corresponding to 1.28 s) and starting from pretrained weights improves performance slightly across all models. b, Varying sampling period with a length to approximately correspond to 64 frames before subsampling. Performance is best with a sampling period of 2.

Extended Data Fig. 2 Individual beat assessment of ejection fraction for each clip in the test dataset.

Left, patients with low variance across beats (s.d. < 2.5, n = 3,353); right, patients with high variance across beats (s.d. > 2.5, n = 717). Each patient video is represented by multiple points that represent the estimate of each beat and a line that indicates 1.96 s.d. from the mean. A greater proportion of beats are within 5% of the ejection fraction estimate made by the human observer (the shaded regions) in videos with low variance compared with individual beat assessment of ejection fraction in high-variance patients.

Extended Data Fig. 3 Model performance during training.

a, b, Mean square error (MSE) loss for the prediction of left ventricular ejection fraction during training on the training (a) and validation (b) dataset. Pixel-level cross-entropy loss for semantic segmentation of the left ventricle during training on the training (c) and validation (d) dataset.

Extended Data Fig. 4 Relationship between clip length, and speed and memory.

Hyperparameter search for model architecture (R2+1D, which is used by EchoNet-Dynamic for ejection fraction prediction, R3D and MC3) and input video clip length (1, 8, 16, 32, 64 and 96 frames) and impact on model processing time and model memory usage.

Extended Data Fig. 5 Variation in echocardiogram video quality and relationship with EchoNet-Dynamic model performance.

a, b, Representative quintile video frames are shown with the respective mean pixel intensity (a) and the s.d. of the pixel intensity (b) compared with the mean absolute error of the ejection fraction prediction of EchoNet-Dynamic (EF MAE) and the Dice similarity coefficient for segmentation of the left ventricle (LV DSC). Box plots show the median as a thick line, the 25th and 75th percentiles as upper and lower bounds of the box, and whiskers extend to 1.5× the interquartile range from the median. n = 1,277.

Extended Data Fig. 6 Impact of degraded image quality on model performance.

Random pixels were removed and replaced with pure black pixels to simulate ultrasound dropout. Representative video frames with dropout are shown across a range of dropouts. The proportion of dropout was compared with model performance with respect to the R2 of the prediction of ejection fraction and the Dice similarity coefficient (DSC) compared with human segmentation of the left ventricle.

Extended Data Table 1 Summary statistics of patient and device characteristics in the Stanford dataset
Full size table
Extended Data Table 2 Performance of EchoNet-Dynamic compared with alternative deep learning architectures in assessing cardiac function
Full size table
Extended Data Table 3 Videos with the most discordance between model prediction and human label of ejection fraction
Full size table
Extended Data Table 4 Model parameters and computational cost
Full size table

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ouyang, D., He, B., Ghorbani, A. et al. Video-based AI for beat-to-beat assessment of cardiac function. Nature 580, 252–256 (2020). https://doi.org/10.1038/s41586-020-2145-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-020-2145-8

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics