Multi-source Multimodal Data and Deep Learning for Disaster Response: A Systematic Review | SN Computer Science Skip to main content
Springer Nature Link
Log in

Multi-source Multimodal Data and Deep Learning for Disaster Response: A Systematic Review

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Mechanisms for sharing information in a disaster situation have drastically changed due to new technological innovations throughout the world. The use of social media applications and collaborative technologies for information sharing have become increasingly popular. With these advancements, the amount of data collected increases daily in different modalities, such as text, audio, video, and images. However, to date, practical Disaster Response (DR) activities are mostly depended on textual information, such as situation reports and email content, and the benefit of other media is often not realised. Deep Learning (DL) algorithms have recently demonstrated promising results in extracting knowledge from multiple modalities of data, but the use of DL approaches for DR tasks has thus far mostly been pursued in an academic context. This paper conducts a systematic review of 83 articles to identify the successes, current and future challenges, and opportunities in using DL for DR tasks. Our analysis is centred around the components of learning, a set of aspects that govern the application of Machine learning (ML) for a given problem domain. A flowchart and guidance for future research are developed as an outcome of the analysis to ensure the benefits of DL for DR activities are utilized.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. Python Apriori algorithm implementation v1.1.2, https://pypi.org/project/apyori/.

  2. Keras, https://keras.io/.

  3. TensorFlow, https://www.tensorflow.org/.

  4. PyTorch, https://pytorch.org/.

  5. Google Earth, https://earth.google.com/web/.

  6. CrisisNLP datasets, https://crisisnlp.qcri.org/.

  7. CrisisLex datasets, https://crisislex.org/.

  8. MediaEval datasets, http://www.multimediaeval.org/.

  9. Figure Eight external annotation service, https://appen.com/.

  10. Places365 dataset, http://places2.csail.mit.edu/download.html.

  11. ImageNet dataset, https://image-net.org/.

References

  1. Abavisani M, Wu L, Hu S, Tetreault J, Jaimes A. Multimodal categorization of crisis events in social media. arXiv (2020).

  2. Acerbo FS, Rossi C. Filtering informative tweets during emergencies: a machine learning approach. In: I-TENDER 2017—Proceedings of the 2017 1st CoNEXT Workshop on ICT Tools for Emergency Networks and DisastEr Relief pp. 1–6 (2017). https://doi.org/10.1145/3152896.3152897.

  3. Agarwal M, Leekha M, Sawhney R, Shah RR. Crisis-DIAS: towards multimodal damage analysis—deployment, challenges and assessment. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020;34(01):346–53. https://doi.org/10.1609/aaai.v34i01.5369.

  4. Ahmad K, Pogorelov K, Riegler M, Ostroukhova O, Halvorsen P, Conci N, Dahyot R. Automatic detection of passable roads after floods in remote sensed and social media data. Signal Process Image Commun. 2019;74((December 2018)):110–8. https://doi.org/10.1016/j.image.2019.02.002.

    Article  Google Scholar 

  5. Aipe A, Ekbal A, S, MN, Kurohashi S. Linguistic feature assisted deep learning approach towards multi-label classification of crisis related tweets. In: Boersma K, Tomaszewski BM (eds.) Proceedings of the 15th International Conference on Information Systems for Crisis Response and Management, Rochester, NY, USA, May 20-23, 2018. ISCRAM Association (2018). http://idl.iscram.org/files/alanaipe/2018/1592_AlanAipe_etal2018.pdf.

  6. Akter S, Wamba SF. Big data and disaster management: a systematic review and agenda for future research. Ann Oper Res. 2017. https://doi.org/10.1007/s10479-017-2584-2.

    Article  Google Scholar 

  7. Alam F, Hassan Z, Ahmad K, Gul A, Reiglar M, Conci N, Al-Fuqaha A. Flood detection via twitter streams using textual and visual features. arXiv 2020; p. 4–6

  8. Alam F, Imran M, Ofli F. Image4Act: online social media image processing for disaster response. In: Proceedings of the 2017 IEEE/ACM International Conference on advances in social networks analysis and mining, ASONAM 2017 2017;601–604. https://doi.org/10.1145/3110025.3110164.

  9. Alam F, Joty S, Imran M. Graph based semi-supervised learning with convolution neural networks to classify crisis related tweets. In: Twelfth International AAAI Conference on Web and Social Media (2018). Accessed 10 May 2021.

  10. Alam F, Ofli F, Imran M. Crisismmd: multimodal twitter datasets from natural disasters. In: Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25–28, 2018, pp. 465–473. AAAI Press 2018. https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17816.

  11. Alexander DE. Social media in disaster risk reduction and crisis management. Sci Eng Ethics. 2014;20(3):717–33.

    Article  Google Scholar 

  12. Algiriyage N, Doyle EEH, Stock K, Johnston D. Data analysis details of the systematic literature review of dl for dr. https://github.com/mu-clab/DLforDR.

  13. Amit SNKB, Aoki Y. Disaster detection from aerial imagery with convolutional neural network. In: Proceedings—International Electronics Symposium on knowledge creation and intelligent computing, IES-KCIC 2017, 2017—January (July 2018), p. 239–245 (2017). https://doi.org/10.1109/KCIC.2017.8228593.

  14. Anbarasan M, Muthu B, Sivaparthipan C, Sundarasekar R, Kadry S, Krishnamoorthy S, Dasel AA, et al. Detection of flood disaster system based on iot, big data and convolutional deep neural network. Comput Commun. 2020;150:150–7. https://doi.org/10.1016/j.comcom.2019.11.022.

    Article  Google Scholar 

  15. Arif Amin MA, Ali AA, Rahman AK. Visual attention-based comparative study on disaster detection from social media images. Innov Syst Softw Eng. 2020;16(3–4):309–19. https://doi.org/10.1007/s11334-020-00368-1. Accessed 10 May 2021.

    Article  Google Scholar 

  16. Attari N, Ofli F, Awad M, Lucas J, Chawla S. Nazr-CNN: Fine-grained classification of UAV imagery for damage assessment. In: Proceedings—2017 International Conference on data science and advanced analytics, DSAA 2017, 2018-January, p. 50–59 (2017). https://doi.org/10.1109/DSAA.2017.72.

  17. Baltrušaitis T, Ahuja C, Morency LP. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell. 2019;41(2):423–43. Accessed 10 May 2021.

    Article  Google Scholar 

  18. Bansal S. A Mutli-task mutlimodal framework for tweet classification based on CNN (Grand Challenge). In: Proceedings—2020 IEEE 6th International Conference on multimedia big data, BigMM 2020; pp. 456–460 (2020). https://doi.org/10.1109/BigMM50055.2020.00075.

  19. Basu M, Shandilya A, Khosla P, Ghosh K, Ghosh S. Extracting resource needs and availabilities from microblogs for aiding post-disaster relief operations. IEEE Trans Comput Soc Syst. 2019;6(3):604–18. https://doi.org/10.1109/TCSS.2019.2914179.

    Article  Google Scholar 

  20. Bejiga MB, Zeggada A, Nouffidj A, Melgani F. A convolutional neural network approach for assisting avalanche search and rescue operations with UAV imagery. Remote Sens. 2017. https://doi.org/10.3390/rs9020100.

    Article  Google Scholar 

  21. Burel G, Alani H. Crisis event extraction service (CREES)—automatic detection and classification of crisis-related content on social media. In: Proceedings of the International ISCRAM Conference 2018; 2018-May, pp. 597–608.

  22. Burel G, Saif H, Alani H. Semantic wide and deep learning for detecting crisis-information categories on social media. In: International Semantic Web Conference, 2017;138–155. Springer. https://doi.org/10.1007/978-3-319-68288-4_9.

  23. Castillo C. Big crisis data: social media in disasters and time-critical situations. Cambridge: Cambridge University Press; 2016.

    Book  Google Scholar 

  24. Cha YJ, Choi W, Büyüköztürk O. Deep learning-based crack damage detection using convolutional neural networks. Comput-Aided Civ Infrastruct Eng. 2017;32(5):361–78. https://doi.org/10.1111/mice.12263.

    Article  Google Scholar 

  25. Chaudhuri N, Bose I. Exploring the role of deep neural networks for post-disaster decision support. Decis Support Syst. 2020;130((July 2019)):130. https://doi.org/10.1016/j.dss.2019.113234.

    Article  Google Scholar 

  26. Chen F, Yu B. Earthquake-induced building damage mapping based on multi-task deep learning framework. IEEE Access. 2019;7:181396–404. https://doi.org/10.1109/ACCESS.2019.2958983.

    Article  Google Scholar 

  27. Cheng CS, Behzadan AH, Noshadravan A. Deep learning for post-hurricane aerial damage assessment of buildings. Comput-Aided Civ Infrastruct Eng. 2021;1:16. https://doi.org/10.1111/mice.12658.

    Article  Google Scholar 

  28. Climate change: how do we know? https://climate.nasa.gov/evidence/. Accessed 29 Oct 2021.

  29. Das S, Datta S, Chaudhuri BB. Handling data irregularities in classification: Foundations, trends, and future challenges. Pattern Recognit. 2018;81:674–93. https://doi.org/10.1016/j.patcog.2018.03.008.

    Article  Google Scholar 

  30. Devaraj A, Murthy D, Dontula A. Machine-learning methods for identifying social media-based requests for urgent help during hurricanes. Int J Disaster Risk Reduct. 2020. https://doi.org/10.1016/j.ijdrr.2020.101757.

    Article  Google Scholar 

  31. Duarte D, Nex F, Kerle N, Vosselman G. Satellite image classification of building damages using airborne and satellite image samples in a deep learning approach. ISPRS Ann Photogramm Remote Sens Spatial Inf Sci. 2018. https://doi.org/10.5194/isprs-annals-IV-2-89-2018.

    Article  Google Scholar 

  32. Emergency response and recovery. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/253488/Emergency_Response_and_Recovery_5th_edition_October_2013.pdf (2013). Accessed 30 Apr 2021.

  33. Fayyad UM, Piatetsky-Shapiro G, Smyth P. From data mining to knowledge discovery in databases. AI Mag. 1996;17(3):37–54. https://doi.org/10.1609/aimag.v17i3.1230.

    Article  Google Scholar 

  34. Gao W, Li L, Zhu X, Wang Y. Detecting disaster-related tweets via multimodal adversarial neural network. IEEE Multimed. 2020;27(4):28–37. https://doi.org/10.1109/MMUL.2020.3012675.

    Article  Google Scholar 

  35. Ghaffarian S, Kerle N, Pasolli E, Arsanjani JJ. Post-disaster building database updating using automated deep learning: an integration of pre-disaster OpenStreetMap and multi-temporal satellite data. Remote Sens. 2019;11(20):1–20. https://doi.org/10.3390/rs11202427.

    Article  Google Scholar 

  36. Ghosh Mondal T, Jahanshahi MR, Wu RT, Wu ZY. Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct Control Health Monit. 2020;27(4):1–15. https://doi.org/10.1002/stc.2507.

    Article  Google Scholar 

  37. Gomez C, Purdie H. Uav-based photogrammetry and geocomputing for hazards and disaster risk monitoring-a review. Geoenviron Disasters. 2016;3(1):23. https://doi.org/10.1186/s40677-016-0060-y.

    Article  Google Scholar 

  38. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x.

    Article  Google Scholar 

  39. Hao H, Wang Y. Leveraging multimodal social media data for rapid disaster damage assessment. Int J Disaster Risk Reduct. 2020. https://doi.org/10.1016/j.ijdrr.2020.101760.

    Article  Google Scholar 

  40. Hao H, Wang Y. Leveraging multimodal social media data for rapid disaster damage assessment. Int J Disaster Risk Reduct. 2020. https://doi.org/10.1016/j.ijdrr.2020.101760.

    Article  Google Scholar 

  41. Hartawan DR, Purboyo TW, Setianingsih C. Disaster victims detection system using convolutional neural network (CNN) method. In: Proceedings—2019 IEEE International Conference on Industry 4.0, artificial intelligence, and communications technology, IAICT 2019 2019;105–111. https://doi.org/10.1109/ICIAICT.2019.8784782.

  42. Hawkins DM. The problem of overfitting. J Chem Inf Comput Sci. 2004;44(1):1–12.

    Article  MathSciNet  Google Scholar 

  43. Hernandez-Suarez A, Sanchez-Perez G, Toscano-Medina K, Perez-Meana H, Portillo-Portillo J, Sanchez V, Villalba LJG. Using twitter data to monitor natural disaster social dynamics: a recurrent neural network approach with word embeddings and kernel density estimation. Sensors (Switzerland). 2019. https://doi.org/10.3390/s19071746.

    Article  Google Scholar 

  44. Hezaveh MM, Kanan C, Salvaggio C. Roof damage assessment using deep learning. In: Proceedings—Applied Imagery Pattern Recognition Workshop 2018; 2017-October, p. 6403–6408. https://doi.org/10.1109/AIPR.2017.8457946.

  45. Huang X, Li Z, Wang C, Ning H. Identifying disaster related social media for rapid response: a visual-textual fused CNN architecture. Int J Digit Earth. 2020;13(9):1017–39. https://doi.org/10.1080/17538947.2019.1633425.

    Article  Google Scholar 

  46. Huang X, Wang C, Li Z, Ning H. A visual-textual fused approach to automated tagging of flood-related tweets during a flood event. Int J Digit Earth. 2019;12(11):1248–64. https://doi.org/10.1080/17538947.2018.1523956.

    Article  Google Scholar 

  47. Imran M, Mitra P, Srivastava J. Cross-language domain adaptation for classifying crisis-related short messages. In: Tapia AH, Antunes P, Bañuls VA, Moore KA, de Albuquerque JP (eds.) 13th Proceedings of the International Conference on information systems for crisis response and management, Rio de Janeiro, Brasil, May 22–25, 2016. ISCRAM Association 2016. http://idl.iscram.org/files/muhammadimran/2016/1396_MuhammadImran_etal2016.pdf.

  48. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on machine learning, 2015; p. 448–456. PMLR.

  49. Jabbar H, Khan RZ. Methods to avoid over-fitting and under-fitting in supervised machine learning (comparative study). Commun Instrum Devices Comput Sci. 2015; p. 163–72.

  50. Japkowicz N. The class imbalance problem: significance and strategies. In: Proceedings of the International Conference on Artificial Intelligence, vol. 56. Citeseer 2000. Accessed 10 May 2021.

  51. Jones S, Saniie J. Using deep learning and satellite imagery to assess the damage to civil structures after natural disasters. In: IEEE International Conference on electro information technology. 2019;189–93. https://doi.org/10.1109/EIT.2019.8833724.

  52. Kabir MY, Madria S. A deep learning approach for tweet classification and rescue scheduling for effective disaster management. Proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, SIGSPATIAL. 2019;269–78. https://doi.org/10.1145/3347146.3359097.

  53. Khan SH, He X, Porikli F, Bennamoun M. Forest change detection in incomplete satellite images with deep neural networks. IEEE Trans Geosci Remote Sens. 2017;55(9):5407–23. https://doi.org/10.1109/TGRS.2017.2707528.

    Article  Google Scholar 

  54. Kim J, Jeong J, Shin J. M2m: Imbalanced classification via major-to-minor translation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13–19, 2020, pp. 13893–13902. Computer Vision Foundation / IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01391, https://openaccess.thecvf.com/content_CVPR_2020/html/Kim_M2m_Imbalanced_Classification_via_Major-to-Minor_Translation_CVPR_2020_paper.html.

  55. Kruspe A, Kersten J, Klan F. Detection of informative tweets in crisis events. In: Natural Hazards and Earth System Sciences (NHESS) 2021.

  56. Kumar A, Singh JP. Location reference identification from tweets during emergencies: a deep learning approach. Int J Disaster Risk Reduct. 2019;33:365–75. https://doi.org/10.1016/j.ijdrr.2018.10.021.

    Article  Google Scholar 

  57. Kumar A, Singh JP. Disaster severity prediction from Twitter images. Adv Intell Syst Comput. 2021;1279(December 2020):65–73. https://doi.org/10.1007/978-981-15-9290-4_7.

    Article  Google Scholar 

  58. Kumar A, Singh JP, Dwivedi YK, Rana NP. A deep multi-modal neural network for informative Twitter content classification during emergencies. 0123456789. Ann Oper Res. 2020. https://doi.org/10.1007/s10479-020-03514-x.

  59. Kumar P, Ofli F, Imran M, Castillo C. Detection of disaster-affected cultural heritage sites from social media images using deep learning techniques. J Comput Cult Herit. 2020;13:3. https://doi.org/10.1145/3383314.

    Article  Google Scholar 

  60. Kundu S, Srijith PK, Desarkar MS. Classification of short-texts generated during disasters: a deep neural network based approach. In: Proceedings of the 2018 IEEE/ACM International Conference on advances in social networks analysis and mining, ASONAM 2018 2018; pp. 790–793. https://doi.org/10.1109/ASONAM.2018.8508695.

  61. Laghrabli S, Benabbou L, Berrado A. A new methodology for literature review analysis using association rules mining. In: 10th International Conference on intelligent systems: theories and applications, SITA 2015, Rabat, Morocco, October 20–21, 2015, 2015; pp. 1–6. IEEE . https://doi.org/10.1109/SITA.2015.7358394.

  62. Li X, Caragea C, Caragea D, Imran M, Ofli F. Identifying disaster damage images using a domain adaptation approach. Proceedings of the International ISCRAM Conference 2019; 2019-May (May 2019), pp. 633–645.

  63. Li X, Caragea D. Improving Disaster-related Tweet Classification with a Multimodal Approach. Social Media for Disaster Response and Resilience Proceedings ofthe 17th ISCRAM Conference (May), 2020;893–902

  64. Li X, Caragea D, Zhang H, Imran M. Localizing and quantifying damage in social media images. In: Proceedings of the 2018 IEEE/ACM International Conference on advances in social networks analysis and mining, ASONAM 2018 2018; pp. 194–201. https://doi.org/10.1109/ASONAM.2018.8508298.

  65. Li Y, Caragea C, Park S, Caragea D, Tapia A. Sympathy detection in disaster Twitter data. In: Proceedings of the International ISCRAM Conference. 2019;788–798. http://idl.iscram.org/files/yingjieli/2019/1899\_YingjieLi\_etal2019.pdf.

  66. Li Y, Ye S, Bartoli I. Semisupervised classification of hurricane damage from postevent aerial imagery using deep learning. J Appl Remote Sens. 2018;12(4):1–13.  https://doi.org/10.1117/1.JRS.12.045008.

  67. Li Y, Ye S, Bartoli I. Semisupervised classification of hurricane damage from postevent aerial imagery using deep learning. J Appl Remote Sens. 2018;12(04):1. https://doi.org/10.1117/1.jrs.12.045008.

    Article  Google Scholar 

  68. Liang X. Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization. Comput-Aided Civ Infrastruct Eng. 2019;34(5):415–30. https://doi.org/10.1111/mice.12425.

    Article  Google Scholar 

  69. Liu Y, Wu L. Geological disaster recognition on optical remote sensing images using deep learning. Proc Comput Sci. 2016;91(Itqm):566–75. https://doi.org/10.1016/j.procs.2016.07.144.

    Article  Google Scholar 

  70. Liu Y, Wu L. High performance geological disaster recognition using deep learning. Proc Comput Sci. 2018;139:529–36. https://doi.org/10.1016/j.procs.2018.10.237.

    Article  Google Scholar 

  71. Lohumi K, Roy S. Automatic detection of flood severity level from flood videos using deep learning models. In: 2018 5th International Conference on information and communication technologies for disaster management, ICT-DM 2018 2019; pp. 1–7. https://doi.org/10.1109/ICT-DM.2018.8636373.

  72. Luna S, Pennock MJ. Social media applications and emergency management: a literature review and research agenda. Int J Disaster Risk Reduct. 2018;28:565–77. https://doi.org/10.1016/j.ijdrr.2018.01.006.

    Article  Google Scholar 

  73. Madichetty S, Muthukumarasamy S. Detection of situational information from Twitter during disaster using deep learning models. Sadhana Acad Proc Eng Sci. 2020;45(1):1–13. https://doi.org/10.1007/s12046-020-01504-0.

    Article  Google Scholar 

  74. Madichetty S, Muthukumarasamy S, Jayadev P. Multi-modal classification of Twitter data during disasters for humanitarian response. J Ambient Intell Humaniz Comput. 2021. https://doi.org/10.1007/s12652-020-02791-5.

    Article  Google Scholar 

  75. Madichetty S, Sridevi M. Detecting informative tweets during disaster using deep neural networks. In: 2019 11th International Conference on communication systems and networks, COMSNETS 2019. 2019;2061:709–13. https://doi.org/10.1109/COMSNETS.2019.8711095.

  76. Madichetty S, Sridevi M. Classifying informative and non-informative tweets from the twitter by adapting image features during disaster. Multimed Tools Appl. 2020;79(39–40):28901–23. https://doi.org/10.1007/s11042-020-09343-1.

    Article  Google Scholar 

  77. Madichetty S, Sridevi M. A stacked convolutional neural network for detecting the resource tweets during a disaster. Multimed Tools Appl. 2021;80(3):3927–49. https://doi.org/10.1007/s11042-020-09873-8.

    Article  Google Scholar 

  78. Mangalathu S, Burton HV. Deep learning-based classification of earthquake-impacted buildings using textual damage descriptions. Int J Disaster Risk Reduct. 2019. https://doi.org/10.1016/j.ijdrr.2019.101111.

    Article  Google Scholar 

  79. Miura H, Aridome T, Matsuoka M. Deep learning-based identification of collapsed, non-collapsed and blue tarp-covered buildings from post-disaster aerial images. Remote Sens. 2020. https://doi.org/10.3390/rs12121924.

    Article  Google Scholar 

  80. Moechammad S, Cahya R, Berkah ANA. Detecting body parts from natural disaster victims using You Only Look Once (YOLO). IOP Conf Ser Mater Sci Eng. 2021. https://doi.org/10.1088/1757-899x/1073/1/012062.

    Article  Google Scholar 

  81. Mohanty SD, Biggers B, Sayedahmed S, Pourebrahim N, Goldstein EB, Bunch R, Chi G, Sadri F, McCoy TP, Cosby A. A multi-modal approach towards mining social media data during natural disasters - A case study of Hurricane Irma. Int J Disaster Risk Reduct. 2021;54(July 2020):102032. https://doi.org/10.1016/j.ijdrr.2020.102032.

    Article  Google Scholar 

  82. Mouzannar H, Rizk Y, Awad M. Damage identification in social media posts using multimodal deep learning. In: Proceedings of the International ISCRAM Conference 2018; 2018-May(May), pp. 529–543.

  83. Muhammad K, Ahmad J, Baik SW. Early fire detection using convolutional neural networks during surveillance for effective disaster management. Neurocomputing. 2018;288:30–42. https://doi.org/10.1016/j.neucom.2017.04.083.

    Article  Google Scholar 

  84. Mullick SS, Datta S, Das S. Generative adversarial minority oversampling. In: 2019 IEEE/CVF International Conference on computer vision, ICCV 2019, Seoul, Korea (South), October 27–November 2, 2019, pp. 1695–1704. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00178.

  85. Naga Pavan Srivathsav C, Anitha K, Anvitha K, Maneesha B, Sagar Imambi S. Detection of disaster affected regions based on change detection using deep architecture. Int J Innov Technol Explor Eng. 2019;8(5):124–8.

    Google Scholar 

  86. Neppalli VK, Caragea C, Caragea D. Deep neural networks versus Naive Bayes classifiers for identifying informative tweets during disasters. In: Boersma K, Tomaszewski BM (eds.) Proceedings of the 15th International Conference on Information Systems for Crisis Response and Management, Rochester, NY, USA, May 20–23, 2018. ISCRAM Association 2018. http://idl.iscram.org/files/venkatakishoreneppalli/2018/1589_VenkataKishoreNeppalli_etal2018.pdf.

  87. Neppalli VK, Caragea C, Caragea D. Deep neural networks versus Naïve Bayes classifiers for identifying informative tweets during disasters. In: Proceedings of the International ISCRAM Conference 2018; 2018-May (May), pp. 677–686.

  88. Nex F, Duarte D, Tonolo FG, Kerle N. Structural building damage detection with deep learning: assessment of a state-of-the-art CNN in operational conditions. Remote Sens. 2019. https://doi.org/10.3390/rs11232765.

    Article  Google Scholar 

  89. Nguyen DT, Ofli F, Imran M, Mitra P. Damage assessment from social media imagery data during disasters. In: Proceedings of the 2017 IEEE/ACM International Conference on advances in social networks analysis and mining, ASONAM 2017 2017;569–76. https://doi.org/10.1145/3110025.3110109. Accessed 10 May 2021.

  90. Nguyen VQ, Anh TN, Yang HJ. Real-time event detection using recurrent neural network in social sensors. Int J Distrib Sens Netw. 2019;(15)6. https://doi.org/10.1177/1550147719856492.

  91. Nunavath V, Goodwin M. The role of artificial intelligence in social media big data analytics for disaster management-initial results of a systematic literature review. In: 2018 5th International Conference on information and communication technologies for disaster management (ICT-DM), 2018;1–4. IEEE. https://doi.org/10.1109/ICT-DM.2018.8636388.

  92. Ofli F, Alam F, Imran M. Analysis of social media data using multimodal deep learning for disaster response 2020;1(May 2020). arXiv: 2004.11838.

  93. Padhee S, Saha TK, Tetreault J, Jaimes A. Clustering of social media messages for humanitarian aid response during crisis. arXiv 2020.

  94. Parilla-Ferrer BE, Fernandez PL, Ballena JT. Automatic classification of disaster-related tweets. In: Proc. International Conference on innovative engineering technologies (ICIET), 2014; vol. 62.

  95. Pi Y, Nath ND, Behzadan AH. Convolutional neural networks for object detection in aerial imagery for disaster response and recovery. Adv Eng Inf. 2020. https://doi.org/10.1016/j.aei.2019.101009.

    Article  Google Scholar 

  96. Pi Y, Nath ND, Behzadan AH. Disaster impact information retrieval using deep learning object detection in crowdsourced drone footage. In: EG-ICE 2020 Workshop on Intelligent Computing in Engineering, Proceedings 2020; pp. 134–143.

  97. Pogrebnyakov N, Maldonado EA. Identifying emergency stages in facebook posts of police departments with convolutional and recurrent neural networks and support vector machines. In: 2017 IEEE International Conference on Big Data (IEEE BigData),2017;4343–52. IEEE Computer Society (2017). https://doi.org/10.1109/BigData.2017.8258464.

  98. Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Reyes MEP, Shyu M, Chen S, Iyengar SS. A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surv. 2019;51(5):92:1-92:36. https://doi.org/10.1145/3234150.

    Article  Google Scholar 

  99. Pouyanfar S, Tao Y, Tian H, Chen SC, Shyu ML. Multimodal deep learning based on multiple correspondence analysis for disaster management. World Wide Web. 2018. https://doi.org/10.1007/s11280-018-0636-4.

    Article  Google Scholar 

  100. Pranesh RR, Shekhar A, Kumar A. Exploring multimodal features and fusion strategies for analyzing disaster tweets.

  101. Prasanna R, Huggins TJ. Factors affecting the acceptance of information systems supporting emergency operations centres. Comput Hum Behav. 2016;57:168–81. https://doi.org/10.1016/j.chb.2015.12.013.

    Article  Google Scholar 

  102. Prasanna R, Yang L, King M. Guidance for developing human-computer interfaces for supporting fire emergency response. Risk Manag. 2013;15(3):155–79. https://doi.org/10.1057/rm.2013.3.

    Article  Google Scholar 

  103. Priya S, Bhanu M, Dandapat SK, Ghosh K, Chandra J. TAQE: tweet retrieval-based infrastructure damage assessment during disasters. IEEE Trans Comput Soc Syst. 2020;7(2):389–403. https://doi.org/10.1109/TCSS.2019.2957208.

    Article  Google Scholar 

  104. Public feed api. https://developers.facebook.com/docs/public_feed/. Accessed 10 May 2021.

  105. Qadir J, Ali A, ur Rasool R, Zwitter A, Sathiaseelan A, Crowcroft J. Crisis analytics: big data-driven crisis response. J Int Hum Action. 2016;1(1):12. https://doi.org/10.1186/s41018-016-0013-9.

    Article  Google Scholar 

  106. Rahnemoonfar M, Murphy R, Miquel MV, Dobbs D, Adams A. Flooded area detection from UAV images based on densely connected recurrent neural networks. In: International Geoscience and Remote Sensing Symposium (IGARSS) 2018; 2018-July, 1788–1791. https://doi.org/10.1109/IGARSS.2018.8517946.

  107. Rice L, Wong E, Kolter JZ. Overfitting in adversarially robust deep learning. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, vol. 119, pp. 8093–8104. PMLR 2020. http://proceedings.mlr.press/v119/rice20a.html.

  108. Rizk Y, Awad M, Jomaa HS, Castillo C. A computationally efficient multi-modal classification approach of disaster-related Twitter images. In: Proceedings of the ACM Symposium on applied computing Part. 2019;F1477(January):2050–9. https://doi.org/10.1145/3297280.3297481.

  109. Robertson BW, Johnson M, Murthy D, Smith WR, Stephens KK. Using a combination of human insights and ‘deep learning’ for real-time disaster communication. Progress Disaster Sci. 2019;2:100030. https://doi.org/10.1016/j.pdisas.2019.100030.

    Article  Google Scholar 

  110. Sadiq AM, Ahn H, Choi YB. Human sentiment and activity recognition in disaster situations using social media images based on deep learning. Sensors. 2020;20(24):7115. https://doi.org/10.3390/s20247115.

    Article  Google Scholar 

  111. Sahoh B, Choksuriwong A. Smart emergency management based on social big data analytics: Research trends and future directions. In: Proceedings of the 2017 International Conference on Information Technology, 2017;1–6. https://doi.org/10.1145/3176653.3176657.

  112. Seydi ST, Rastiveis H. A deep learning framework for roads network damage assessment using post-earthquake lidar data. Int Arch Photogramm Remote Sens Spatial Inf Sci ISPRS Arch. 2019;42(4/W18):955–61. https://doi.org/10.5194/isprs-archives-XLII-4-W18-955-2019.

    Article  Google Scholar 

  113. Shams S, Goswami S, Lee K. Deep learning-based spatial analytics for disaster-related tweets: an experimental study. In: Proceedings—IEEE International Conference on mobile data management 2019-June (Mdm), 2019;337–342. https://doi.org/10.1109/MDM.2019.00-40.

  114. Shin H, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura DJ, Summers RM. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. CoRR abs/1602.03409 (2016). arxiv:1602.03409.

  115. Sit MA, Koylu C, Demir I. Identifying disaster-related tweets and their semantic, spatial and temporal context using deep learning, natural language processing and spatial analysis: a case study of Hurricane Irma. Int J Digit Earth. 2019;12(11):1205–29. https://doi.org/10.1080/17538947.2018.1563219.

    Article  Google Scholar 

  116. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15(1):1929–58. http://dl.acm.org/citation.cfm?id=2670313.

  117. Sublime J, Kalinicheva E. Automatic post-disaster damage mapping using deep-learning techniques for change detection: case study of the Tohoku tsunami. Remote Sens. 2019. https://doi.org/10.3390/rs11091123.

    Article  Google Scholar 

  118. Sun W, Bocchini P, Davison BD. Applications of artificial intelligence for disaster management. Dordrecht: Springer Netherlands; 2020. https://doi.org/10.1007/s11069-020-04124-3 (0123456789). Accessed 10 May 2021.

    Book  Google Scholar 

  119. Tatulli E, Hueber T. Feature extraction using multimodal convolutional neural networks for visual speech recognition. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2971–2975. IEEE (2017). https://doi.org/10.1109/ICASSP.2017.7952701.

  120. Tian H, Zheng HC, Chen S. Sequential deep learning for disaster-related video classification. IEEE 1st Conference on multimedia information processing and retrieval (MIPR). 2018;106–111. IEEE 2018.https://doi.org/10.1109/MIPR.2018.00026.

  121. The learning problem. http://work.caltech.edu/slides/slides01.pdf. Accessed 10 May 2021.

  122. Vetrivel A, Gerke M, Kerle N, Nex F, Vosselman G. Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J Photogramm Remote Sens. 2018;140:45–59. https://doi.org/10.1016/j.isprsjprs.2017.03.001.

    Article  Google Scholar 

  123. Wang T, Tao Y, Chen SC, Shyu ML. Multi-Task Multimodal Learning for Disaster Situation Assessment. Proceedings - 3rd International Conference on Multimedia Information Processing and Retrieval, MIPR 2020 pp. 209–212 2020. https://doi.org/10.1109/MIPR49039.2020.00050.

  124. Wang Z, Ye X. Social media analytics for natural disaster management. Int J Geogr Inf Sci. 2018;32(1):49–72. https://doi.org/10.1080/13658816.2017.1367003.

    Article  Google Scholar 

  125. Watson C, Cooper N, Palacio DN, Moran KP, Poshyvanyk D. A systematic literature review on the use of deep learning in Software Engineering Research. arXiv (2020).

  126. Westerlund M. The emergence of deepfake technology: a review. Technol Innov Manag Rev. 2019;9(11).

  127. Yang T, Xie J, Li G, Mou N, Li Z, Tian C, Zhao J. Social media big data mining and spatiotemporal analysis on public emotions for disaster mitigation. ISPRS Int. J. Geo Inf. 2019;(8)1:29. https://doi.org/10.3390/ijgi8010029.

  128. Yigitcanlar T, Kamruzzaman M, Foth M, Sabatini J, da Costa E, Ioppolo G. Can cities become smart without being sustainable? a systematic review of the literature. Sustain Cities Soc. 2018. https://doi.org/10.1016/j.scs.2018.11.033.

    Article  Google Scholar 

  129. Zhang QX, Lin GH, Zhang YM, Xu G, Wang JJ. Wildland forest fire smoke detection based on faster R-CNN using synthetic smoke images. Proc Eng. 2018;211:441–6. https://doi.org/10.1016/j.proeng.2017.12.034.

    Article  Google Scholar 

  130. Zhao F, Zhang C. Building damage evaluation from satellite imagery using deep learning. In: Proceedings–2020 IEEE 21st International Conference on information reuse and integration for data science, IRI 2020 pp. 82–89 (2020). https://doi.org/10.1109/IRI49571.2020.00020.

Download references

Author information

Authors and Affiliations

  1. Joint Centre for Disaster Research, Massey University, Wellington, New Zealand

    Nilani Algiriyage, Raj Prasanna, Emma E. H. Doyle & David Johnston

  2. Massey Geoinformatics Collaboratory, Massey University, Auckland, New Zealand

    Kristin Stock

Authors
  1. Nilani Algiriyage
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Raj Prasanna
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Kristin Stock
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Emma E. H. Doyle
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. David Johnston
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nilani Algiriyage.

Ethics declarations

Conflicts interest

The authors declare no conflicts interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Glossary of Terms

Table 7 shows the expansions of abbreviated terms used in the paper.

Table 7 Glossary of Terms
Full size table

Appendix B: Publication Venues

Table 8 provides article publication venues that are also listed in our online appendix [12].

Table 8 Article Publication Venues
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Algiriyage, N., Prasanna, R., Stock, K. et al. Multi-source Multimodal Data and Deep Learning for Disaster Response: A Systematic Review. SN COMPUT. SCI. 3, 92 (2022). https://doi.org/10.1007/s42979-021-00971-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00971-4

Keywords