Multi-object trajectory extraction based on YOLOv3-DeepSort for pedestrian-vehicle interaction behavior analysis at non-signalized intersections | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

Multi-object trajectory extraction based on YOLOv3-DeepSort for pedestrian-vehicle interaction behavior analysis at non-signalized intersections

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Pedestrian-vehicle interaction behavior analysis at non-signalized intersections has become the important content of traffic safety research. In this paper, we focus on presenting a processing framework for the analysis of pedestrian-vehicle interaction behavior based on YOLOv3-DeepSort. Comparative experiments are done for verifying the performance of the presented framework. The presented YOLOv3-DeepSort can achieve the results that the ML value is 14.10% and the IDs value is 382 on the MOT16 dataset. And YOLOv3–416 can achieve the efficiency of 29 ms. Results show that the presented framework has excellent performance for trajectory extraction. Furthermore, the methodology is confirmed by the presented case. And we found that the studies on exit interactions can provide theoretical evidences for presenting some protective measures, to ensure the safety of pedestrians and vehicles at non-signalized intersections.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Listing 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The experiment data used to support the findings of this study are available from the corresponding author upon request.

References

  1. Ackermann C, Beggiato M, Bluhm LF, Low A, Krems JF (2019) Deceleration parameters and their applicability as informal communication signal between pedestrians and automated vehicles. Trans Res Part F-Traffic Psychol Behav 62:757–768

    Article  Google Scholar 

  2. Ahmad T, Ma Y, Yahya M, Ahmad B, Nazir S, ul Haq A(2020) Object detection through modified YOLO neural network Scientific Program

  3. Alsaleh R, Sayed T, Zaki MH (2018) Assessing the effect of Pedestrians' use of cell phones on their walking behavior: a study based on automated video analysis. Transp Res Rec 2672(35):46–57

    Article  Google Scholar 

  4. Bennett MK, Van Houten R (2016) Variables influencing efficacy of gateway in-street sign configuration on yielding at crosswalks. Trans Res Record (2586):100–105

  5. Bewley A, Ge Z, Ott L, Ramov F, Upcroft B, Ieee (2016) Simple onlne and realtime tracking. IEEE Int Conf Image Process ICIP:3464–3468

  6. Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A, Di W (2018) Recommendation system for immunization coverage and monitoring. Human Vaccines Immunotherapeutics 14(1):165–171 2018

  7. Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enter Inform Syst 13(3):329–351

    Article  Google Scholar 

  8. Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z, Yuan L (2022) Assessing the change of ambient air quality patterns in Jiangsu Province of China pre-to post-COVID-19. Chemosphere 288

  9. Bhatti UA, Yu Z, Chanussot J, Zeeshan Z, Yuan L, Luo W, Nawaz SA, Bhatti MA (2022) Local similarity-based spatial-spectral fusion hyperspectral image classification with deep CNN and Gabor filtering. IEEE Trans Geosci Remote Sens 60:2022

    Article  Google Scholar 

  10. Bhatti UA, Yu Z, Hasnain A, Nawaz SA, Yuan L, Wen L, Bhatti MA (2022) Evaluating the impact of roads on the diversity pattern and density of trees to improve the conservation of species. Environ Sci Pollut Res 29(10):14780–14790

    Article  Google Scholar 

  11. Bloemendal P (2013) Stichting Wetenschappelijk Onderzoek Verkeersveiligheid SWOV

  12. Cai Y, Dai L, Wang H, Chen L, Li Y, Sotelo MA, Li Z (2022) Pedestrian motion trajectory prediction in intelligent driving from far shot first-person perspective video. IEEE Trans Intell Transp Syst 23(6):5298–5313

    Article  Google Scholar 

  13. Chen C, Liu B, Wan S, Qiao P, Pei Q (2021) An edge traffic flow detection scheme based on deep learning in an intelligent transportation system. IEEE Trans Intell Transp Syst 22(3):1840–1852

    Article  Google Scholar 

  14. Cheng R, Wang K, Yang K, Long N, Hu W, Chen H, Bai J, Liu D (2017) Crosswalk navigation for people with visual impairments on a wearable device. J Electronic Imaging 26(5)

  15. Chu Q, Ouyang WL, Li HS, Wang XG, Liu B, Yu NH, Ieee (2017) Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism. 2017 Ieee Int Conf Comput Vision, IEEE Int Conf Comput Vision. 4846–4855

  16. Coifman B, Beymer D, McLauchlan P, Malik J (1998) A real-time computer vision system for vehicle tracking and traffic surveillance. Trans Res Part C-Emerg Technol 6(4):271–288

    Article  Google Scholar 

  17. Dai J, Li Y, He K, Sun J (2016) R-FCN: object detection via region-based fully convolutional networks. Adv Neural Inform Process Syst

  18. Dewan MC, Rattani A, Gupta S, Baticulon RE, Hung Y-C, Punchak M, Agrawal A, Adeleye AO, Shrime MG, Rubiano AM, Rosenfeld JV, Park KB (2019) Estimating the global incidence of traumatic brain injury. J Neurosurg 130(4):1080–1097

    Article  Google Scholar 

  19. Dey D, Martens M, Eggen B, Terken J (2019) Pedestrian road-crossing willingness as a function of vehicle automation, external appearance, and driving behaviour. Trans Res Part F-Traffic Psychol Behav 65:191–205

    Article  Google Scholar 

  20. Dingus TA, Guo F, Lee S, Antin JF, Perez M, Buchanan-King M, Hankey J (2016) Driver crash risk factors and prevalence evaluation using naturalistic driving data. Proceed Nat Acad Sci USA 113(10):2636–2641

    Article  Google Scholar 

  21. Dinno A (2015) Nonparametric pairwise multiple comparisons in independent groups using Dunn's test. Stata J 15(1):292–300

    Article  Google Scholar 

  22. Eluru N, Bhat CR, Hensher DA (2008) A mixed generalized ordered response model for examining pedestrian and bicyclist injury severity level in traffic crashes. Accid Anal Prev 40(3):1033–1054

    Article  Google Scholar 

  23. Fang K, Xiang Y, Li X, Savarese S, Ieee (2018) Recurrent autoregressive networks for online multi-object tracking. IEEE Winter Conf Appl Comp Vision:466–475

  24. Fang Y, Liu J, Li J, Cheng J, Hu J, Yi D, Xiao X, Bhatti UA (2022) Robust zero-watermarking algorithm for medical images based on SIFT and Bandelet-DCT. Multimed Tools Appl 81(12):16863–16879

    Article  Google Scholar 

  25. Fu T, Miranda-Moreno L, Saunier N (2016) Pedestrian crosswalk safety at nonsignalized crossings during nighttime use of thermal video data and surrogate safety measures. Transp Res Record (2586):90–99

  26. Fu CY, Liu W, Ranga A, Tyagi A, Berg AC (2017) DSSD : Deconvolutional Single Shot Detector

  27. Fu T, Miranda-Moreno L, Saunier N (2018) A novel framework to evaluate pedestrian safety at non-signalized locations. Accid Anal Prev 111:23–33

    Article  Google Scholar 

  28. Fu T, Hu W, Miranda-Moreno L, Saunier N (2019) Investigating secondary pedestrian-vehicle interactions at non-signalized intersections using vision-based trajectory data. Transp Res Part C Emerging Technol 105C(AUG):222–240

    Article  Google Scholar 

  29. Gorrini A, Crociani L, Vizzari G, Bandini S (2018) Observation results on pedestrian-vehicle interactions at non-signalized intersections towards simulation. Trans Res Part F-Traffic Psychol Behav 59:269–285

    Article  Google Scholar 

  30. Guo Y, Li Z, Wu Y, Xu C (2018) Exploring unobserved heterogeneity in bicyclists' red-light running behaviors at different crossing facilities. Accid Anal Prev 115:118–127

    Article  Google Scholar 

  31. Hacohen S, Shvalb N, Shoval S (2018) Dynamic model for pedestrian crossing in congested traffic based on probabilistic navigation function. Trans Res Part C-Emerging Technol 86:78–96

    Article  Google Scholar 

  32. He KM, Zhang XY, Ren SQ, Sun J, Ieee (2016) Deep residual learning for image recognition. 2016 Ieee conf comput vision patt recogn, IEEE conf comput vision patt Recogn:770–778

  33. He K, Gkioxari G, Dollar P, Girshick R, Ieee (2017) Mask R-CNN. IEEE Int Conf Comput Vision:2980–2988

  34. Iasmin H, Kojima A, Kubota H (2016) Safety effectiveness of pavement design treatment at intersections: left turning vehicles and pedestrians on crosswalks. IATSS Res

  35. Jeon K, Hoon LS, Eom J, Cho S, Cho HK Smart crosswalk traffic safety system useful for visualizing pedestrians crosswalk use state information comprises camera that is installed adjacent to crosswalk for photographing crosswalk and surrounding areas of crosswalk. KR2021065219-A,to Wisedream Co Ltd; Toicos Inc; Gumi Electronics & Information Technolog

  36. Kalman RE (1960) A new approach to linear filtering and prediction problems. J Basic Eng 82D:35–45

    Article  MathSciNet  Google Scholar 

  37. Khasnabis S, Zegeer CV, Cynecki MJ (1982) Effects of pedestrian signals on safety, operations, and pedestrian behavior--literature review Transp Res Rec

  38. Kim H (2022) Multiple vehicle tracking and classification system with a convolutional neural network. J Ambient Intell Humaniz Comput 13(3):1603–1614

    Article  Google Scholar 

  39. Kirchner A, Nishinari K, Schadschneider A (2003) Friction effects and clogging in a cellular automaton model for pedestrian dynamics. Phys Rev E 67(5)

  40. Kuhn HW (2005) The Hungarian method for the assignment problem. Nav Res Logist 52(1):7–21

    Article  MathSciNet  Google Scholar 

  41. Laureshyn A, Svensson A, Hyden C (2010) Evaluation of traffic safety, based on micro-level behavioural data: theoretical framework and first implementation. Accid Anal Prev 42(6):1637–1646

    Article  Google Scholar 

  42. Li G, Yang Y, Qu X (2020) Deep learning approaches on pedestrian detection in hazy weather. IEEE Trans Ind Electron 67(10):8889–8899

    Article  Google Scholar 

  43. Li T, Li J, Liu J, Huang M, Chen Y-W, Bhatti UA (2022) Robust watermarking algorithm for medical images based on log-polar transform. Eurasip J Wireless Commun Networking 2022, 25(1)

  44. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar P, Zitnick CL (2014) Microsoft COCO: Common Objects in Context. Comput Vision - Eccv 2014, Pt V, Lecture Notes in Computer Science D. Fleet, T. Pajdla, B. Schiele and T. Tuytelaars, eds. 740–755

  45. Lin T-Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S, Ieee (2017) Feature pyramid networks for object detection. IEEE Conf Comput Vision Patt Recogn. 936–944

  46. Lin T-Y, Goyal P, Girshick R, He K, Dollar P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327

    Article  Google Scholar 

  47. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) SSD: single shot MultiBox detector. Lect Notes Comput Sci:21–37

  48. Loukaitou-Sideris A, Liggett R, Sung H-G (2007) Death on the crosswalk - a study of pedestrian-automobile collisions in Los Angeles. J Plan Educ Res 26(3):338–351

    Article  Google Scholar 

  49. Madigan R, Nordhoff S, Fox C, Amini RE, Louw T, Wilbrink M, Schieben A, Merat N (2019) Understanding interactions between automated road transport systems and other road users: a video analysis. Trans Res Part F-Traffic Psychol Behav 66:196–213

    Article  Google Scholar 

  50. Mahmoudi N, Ahadi SM, Rahmati M (2019) Multi-target tracking using CNN-based features: CNNMTT. Multimed Tools Appl 78(6):7077–7096

    Article  Google Scholar 

  51. McCall JC, Trivedi MM (2006) Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation. IEEE Trans Intell Transp Syst 7(1):20–37

    Article  Google Scholar 

  52. Nantulya VM, Reich MR (2002) The neglected epidemic: road traffic injuries in developing countries. Br Med J 324(7346):1139–1141

    Article  Google Scholar 

  53. Nawaz SA, Li J, Bhatti UA, Bazai SU, Zafar A, Bhatti MA, Mehmood A, Ul Ain Q, Shoukat MU (2021) A hybrid approach to forecast the COVID-19 epidemic trend. PLoS One 16(10):2021

    Article  Google Scholar 

  54. Olszewski P, Dabkowski P, Szagala P, Czajewski W, Buttler I (2020) Surrogate safety indicator for unsignalised pedestrian crossings. Trans Res Part F-Traffic Psychol Behaviour 70:25–36

    Article  Google Scholar 

  55. Organization WH (2013) Global status report on road safety 2013:supporting a decade of action. World Health Organization

  56. Osama A, Sayed T (2017) Macro-spatial approach for evaluating the impact of socio-economics, land use, built environment, and road facility on pedestrian safety. Can J Civ Eng 44(12):1036–1044

    Article  Google Scholar 

  57. Pang B, Li Y, Zhang Y, Li M, Lu C, Ieee (2020) TubeTK: adopting tubes to track multi-object in a one-step training model. IEEE Conf Computer Vision Patt Recogn:6307–6317

  58. Peng J, Wang C, Wan F, Wu Y, Fu Y (2020) Chained-tracker: chaining paired attentive regression results for. End-to-End Joint Multiple-Object Detection Track

  59. Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement. arXiv e-prints

  60. Redmon J, Divvala S, Girshick R, Farhadi A, Ieee, (2016) You only look once: unified, real-time object detection. IEEE Conf Comput Vision Patt Recogn. pp. 779–788

  61. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. Comput Vision Patt Recogn

  62. Ren SQ, He KM, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networksAdvances in Neural Information Processing Systems

  63. Roudsari B, Kaufman R, Koepsell T (2006) Turning at intersections and pedestrian injuries. Traffic Injury Prevent 7(3):283–289

    Article  Google Scholar 

  64. Sanchez-Matilla R, Poiesi F, Cavallaro A (2016) Online multi-target tracking with strong and weak detections. Lect Notes Comput Sci:84–99

  65. Sedgwick P (2014) Cross sectional studies: advantages and disadvantages. Philip Sedgwick Reader Med Stat Med Educ:2

  66. Sheykhfard A, Haghighi F (2020) Assessment pedestrian crossing safety using vehicle-pedestrian interaction data through two different approaches: fixed videography (FV) vs in-motion videography (IMV). Accid Anal Prev 144

  67. Takala J, Hamalainen P, Saarela KL, Yun LY, Manickam K, Jin TW, Heng P, Tjong C, Kheng LG, Lim S, Lin GS (2014) Global estimates of the burden of injury and illness at work in 2012. J Occup Environ Hygiene 11(5):326–337

    Article  Google Scholar 

  68. Tang SY, AndresB, Andriluka M, Schiele B (2016) Multi-person Tracking by Multicut and Deep Matching," Computer Vision - Eccv 2016 Workshops, Pt Ii, Lecture Notes in Computer Science G. Hua and H. Jegou, eds.. 100–111

  69. Tian S, Zheng M, Zou W, Li X, Zhang L, “Dynamic crosswalk scene understanding for the visually impaired,” Ieee Trans Neural Syst Rehab Eng. vol. 29, pp. 1478–1486, 2021, 2021.

  70. Trivedi MM, Gandhi T, McCall J (2007) Looking-in and looking-out of a vehicle: computer-vision-based enhanced vehicle safety. IEEE Trans Intell Transp Syst 8(1):108–120

    Article  Google Scholar 

  71. van Haperen W, Riaz MS, Daniels S, Saunier N, Brijs T, Wets G (2019) Observing the observation of (vulnerable) road user behaviour and traffic safety: a scoping review. Accid Anal Prev 123:211–221

    Article  Google Scholar 

  72. Veeramani B, Raymond JW, Chanda P (2018) DeepSort: deep convolutional networks for sorting haploid maize seeds. Bmc Bioinform 19:Aug

  73. Wan X, Zhao Q, Wang J, Deng S, Kong Z, Ieee (2018) Multi-object tracking using online metric learning with long short-term memory. IEEE Int Conf Image Process ICIP:788–792

  74. Wang Z, Zheng L, Liu Y, Wang S (2019) Towards real-time multi-object tracking

  75. Wang Y, Su Q, Wang C, Prato CG (2021) Investigating yielding behavior of heterogeneous vehicles at a semi-controlled crosswalk. Accid Anal Prev 161:106381–106381

    Article  Google Scholar 

  76. Wanvik PO (2009) Effects of road lighting: an analysis based on Dutch accident statistics 1987-2006. Accid Anal Prev 41(1):123–128

    Article  Google Scholar 

  77. Wojke N, Bewley A, Paulus D, Ieee (2017) Simple online and realtime tracking with a DEEP association metric. IEEE Int Conf Image Process ICIP:3645–3649

  78. Xing Y, Lv C, Wang H, Cao D, Velenis E, Wang F.-Y, “Driver activity recognition for intelligent vehicles: a deep learning approach,” IEEE Trans Veh Technol, vol. 68, no. 6, pp. 5379–5390, Jun, 2019.

  79. Yan F, Li B, Zhang W, Hu G (2016) Red-light running rates at five intersections by road user in Changsha, China: an observational study. Accid Anal Prev 95:381–386

    Article  Google Scholar 

  80. Yu F, Li W, Li Q, Liu Y, Shi X, Yan J (2016) POI: multiple object tracking with high performance detection and appearance feature. Lect Notes Comput Sci:36–42

  81. Yu K, Lin L, Alazab M, Tan L, Gu B (2021) Deep learning-based traffic safety solution for a mixture of autonomous and manual vehicles in a 5G-enabled intelligent transportation system. IEEE Trans Intell Transp Syst 22(7):4337–4347

    Article  Google Scholar 

  82. Zhang G, Yau KKW, Chen G (2013) Risk factors associated with traffic violations and accident severity in China. Accid Anal Prev 59:18–25

    Article  Google Scholar 

  83. Zhang J, Huang M, Jin X, Li X (2017) A real-time Chinese traffic sign detection algorithm based on modified YOLOv2. Algorithms 10(4)

  84. Zhang X, Hao X, Liu S, Wang J, Xu J, Hu J (2019) Multi-target tracking of surveillance video with differential YOLO and DeepSort. Proc SPIE

  85. Zhang Y, Wang C, Wang X, Zeng W, Liu W (2021) FairMOT: on the fairness of detection and re-identification in multiple object tracking. Int J Comput Vis 129(11):3069–3087

    Article  Google Scholar 

  86. Zhou Z, Xing J, Zhang M, Hu W, Ieee (2018) Online multi-target tracking with tensor-based high-order graph matching. Int Conf Patt Recogn:1809–1814

Download references

Acknowledgments

The author would like to sincerely thank the editor and anonymous reviewers for their thoughtful and valuable comments which have significantly improved the quality of this paper.

Author information

Authors and Affiliations

  1. School of Transportation, Southeast University, Southeast University Road #2, Nanjing, 211189, China

    Qiang Zhang

Authors
  1. Qiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiang Zhang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Q. Multi-object trajectory extraction based on YOLOv3-DeepSort for pedestrian-vehicle interaction behavior analysis at non-signalized intersections. Multimed Tools Appl 82, 15223–15245 (2023). https://doi.org/10.1007/s11042-022-13805-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13805-z

Keywords

Search

Navigation