Tracking and Simulating Pedestrian Movements at Intersections Using Unmanned Aerial Vehicles

Abstract

For a city to be livable and walkable is the ultimate goal of future cities. However, conflicts among pedestrians, vehicles, and cyclists at traffic intersections are becoming severe in high-density urban transportation areas, especially in China. Correspondingly, the transit time at intersections is becoming prolonged, and pedestrian safety is becoming endangered. Simulating pedestrian movements at complex traffic intersections is necessary to optimize the traffic organization. We propose an unmanned aerial vehicle (UAV)-based method for tracking and simulating pedestrian movements at intersections. Specifically, high-resolution videos acquired by a UAV are used to recognize and position moving targets, including pedestrians, cyclists, and vehicles, using the convolutional neural network. An improved social force-based motion model is proposed, considering the conflicts among pedestrians, cyclists, and vehicles. In addition, maximum likelihood estimation is performed to calibrate an improved social force model. UAV videos of intersections in Shenzhen are analyzed to demonstrate the performance of the presented approach. The results demonstrate that the proposed social force-based motion model can effectively simulate the movement of pedestrians and cyclists at road intersections. The presented approach provides an alternative method to track and simulate pedestrian movements, thus benefitting the organization of pedestrian flow and traffic signals controlling the intersections.

1. Introduction

With rapid economic development, the use of automobiles has greatly increased in developing countries, especially in China, India, and Vietnam, where vehicles are replacing bicycles as the dominant transportation mode [1,2]. Facing this great change, the space allocated to automobiles has been expanded, thus alleviating traffic congestion, which encroaches on the space for cyclists and pedestrians and constrains bicycling and walking. Consequently, potential conflicts of vehicles, cycles, and pedestrians not only exacerbate travel delays but also increase the randomness of pedestrian movements, substantially threatening pedestrian safety. In a recent traffic safety report released by the World Health Organization (WHO), road collisions are the world’s leading cause of preventable death; over 1.25 million people die annually on the roads (especially at intersections) because of traffic collisions [3]. In some ways, active transportation users, such as cyclists and pedestrians, are more vulnerable to injuries than other road users due to their labile speed and direction [4]. Therefore, a walkable city for people is regarded as one ultimate goals of future cities [5]. Urban planners have taken several actions to encourage walking, such as configuring special walkable lanes, designing good walking interfaces, building friendly walkable infrastructures, etc. These actions also highlight the necessity of pedestrian tracking and simulation.

Traditionally, pedestrian safety inspection largely relies on historical collision records. However, due to the lack of detailed and precise historical data and the infrequent occurrence of collisions, the task of inspection often cannot be fully accomplished. Recently, the use of pedestrian conflicts as an alternative for collisions to analyze road safety has attracted significant interest [6,7,8]. Pedestrian conflicts can provide detailed information concerning road dynamics at intersections, allowing the detection of the series of events that lead to collisions [9,10]. Pedestrian conflict analysis can be conducted by detecting and tracking moving traffic objects or flawed design elements that may be causing safety issues [10]. The introduction of computer vision algorithms has greatly strengthened pedestrian conflict and violation analysis by automating the extraction of accurate movements of traffic objects, overcoming many shortcomings of manual pedestrian analysis techniques [11]. Furthermore, considering the interactions of pedestrians, cyclists, and vehicles, this study aims to track and simulate pedestrian movements at complex traffic intersections using advanced computer vision algorithms. These algorithms will provide microscopic simulations of traffic intersections to optimize traffic organization and promote pedestrian safety.

Recently, many methods have been developed to track pedestrians and vehicles, including the microwave-based method [12], the global positioning system (GPS)-based method [13,14], the induction coil-based method, the video-based method [15,16], and the laser radar-based method [17,18]. Favored for its easy data acquisition, the video-based method, which typically uses videos of traffic intersections to monitor objects, has been extensively applied. Currently, video-based target recognition and tracking can simultaneously target multiple objects of interest (e.g., pedestrians, vehicles, and animals) [19,20]. A set of methods involved in this process, i.e., mean shift, particle filter, and Kalman filter, are typically generative approaches and achieve satisfactory target recognition results [21]. Tracked pedestrians or vehicles are treated as foreground targets. Detectors with online learning or offline training (including random forest and support vector machine) are utilized to distinguish foreground targets from the background, and therefore obtain the positions of the foreground targets. On the other hand, because of their good performance in target detection and localization, deep learning methods have provided new approaches for target recognition and positioning [22,23,24], including the recurrent-convolutional neural network (R-CNN) [25], faster R-CNN [26], single shot multi-box detector (SSD) [27], you only look once v3 (YOLOv3) [28], cascade R-CNN [29], and especially the parallel feature pyramid network (PFPNet) [30], which obtained remarkable performance on many public datasets while maintaining a relatively high execution speed. Therefore, this study improves PFPNet and accurately extracts the trajectories of pedestrians, cyclists, and vehicles from high-resolution unmanned aerial vehicle (UAV) videos to accurately simulate pedestrian movement at traffic intersections.

Microscopic traffic simulation has been extensively employed as an effective tool to analyze the movements of vehicles and pedestrians. Widely employed microscopic traffic simulation models include the cellular automata, lattice gas, and social force models [31,32]. The social force model (SFM) transforms the internal motivation of individual movement into a corresponding force and describes the force that pedestrians experience using a force vector. By establishing a traditional SFM, Helbing and Molnar [32] and Helbing et al. [33,34,35,36,37,38] successfully simulated the self-organizing phenomenon in one- and two-way pedestrian flows under normal conditions. Alternative SFMs were subsequently developed by combining actual situations of pedestrian movements and modifying or adding new elements. For example, Hou et al. [39] observed that pedestrians are more inclined to follow leading pedestrians during an evacuation in emergencies, further investigated the impacts of leading pedestrians during an evacuation process, and established a social force evacuation model. Zeng et al. [40,41] analyzed the peculiarities of pedestrian movements at signalized traffic intersections and proposed the control of traffic signals for pedestrian movements on crosswalks. They improved the structure and parameters of the social force model and established a microscopic pedestrian model for traffic intersections. Liu et al. [42] explored various interactions of pedestrians on crosswalks by considering the collision avoidance behaviors of pedestrians moving backward and the follow-up behaviors of leading pedestrians, and established a microscopic model to incorporate the interactions between pedestrians and surrounding pedestrians. These studies provided valuable insights into microscopic pedestrian traffic simulation. However, the impacts of cyclists and vehicles on pedestrians have not been well investigated and integrated into microscopic traffic simulation. Additionally, high-resolution trajectories of pedestrians and other objects, especially cyclists and vehicles, at complex intersections is lacking.

A UAV is an effective tool to monitor geographical contexts with simple deployment and a low cost [43,44,45,46]. This study employs a UAV to automatically identify and track moving objects, including pedestrians, cyclists, and vehicles at complex traffic intersections, and to simulate the pedestrian movements. Especially, we consider interactions between pedestrians and the surrounding environment and simulate pedestrian movements at intersections affected by cyclists and vehicles. The traditional social force model is modified to integrate the boundary effects of zebra crossings and mutual interactions between pedestrians, cyclists, and vehicles. In addition, the model parameters are calibrated based on the maximum likelihood estimation (MLE) method. An experiment in Shenzhen City was conducted to evaluate the performance of the presented approach. The results demonstrate that the presented approach can accurately simulate pedestrian movements at traffic intersections. The improved social force model describes complex interactions within a complex intersection environment and outperforms the traditional social force model.

The main contributions of this study are summarized as follows: (1) This study successfully and accurately extracts high-resolution movements of pedestrians, cyclists, and vehicles at intersections using a UAV and a convolutional neural network. (2) The classic SFM is improved to integrate the interactions among cyclists, vehicles, and pedestrians, and the inherent law of pedestrians is verified and revealed. (3) The MLE method is introduced to calibrate the model parameters and quantify the range and extent of the impacts of surrounding pedestrians, cyclists, right-turning vehicles, and boundaries, which provide a useful reference for subsequent research on further calibration.

The remainder of this article is organized as follows: Section 2 introduces the study area and the presented methodology. Section 3 describes the experiment and analyzes the results. Section 4 concludes the results and outlooks on future research.

2. Study Area and Methodology

The study was conducted in Shenzhen City, the first special economic zone of China, covering 1996 km². Since the foundation in 1979, Shenzhen has experienced fast urban growth. The population has increased from 0.6 million to 18 million in 2018. Shenzhen has become one of the highest density cities in China. During rush hour, the pedestrian density at some complex traffic intersections may be up to 2–5 persons/m². Therefore, there are highly potential conflicts for pedestrians at these intersections, which highlights the importance of pedestrian monitoring. Here, we present a UAV-based approach to monitor pedestrians, cyclists, and vehicles at complex traffic intersections and simulate pedestrian movements with an improved social force model. The workflow of the presented approach is displayed in Figure 1. First, UAVs are used to capture pedestrians, cyclists, and vehicles. Objects are recognized and localized using state-of-the-art PFPNet. High-resolution trajectories are produced for further microscopic traffic simulation. Considering cyclists and vehicles, an improved social force-based motion model is developed to simulate pedestrian movements at intersections.

2.1. Pedestrian, Cyclist, and Vehicle Detection Using a UAV

To obtain high-quality data and reduce noisy information contained in traffic videos, we use a UAV to capture pedestrians, cyclists, and vehicles at road intersections. Compared with a traditional camera installed along the roadside or at intersections [45], the UAV surveillance approach has the following advantages: the hovering location and flying height of the UAV can be conveniently set and changed; the camera scope is substantially greater than that of traditional cameras; and high-resolution UAV videos can simultaneously capture richer information about pedestrians, cyclists, and vehicles.

Using high-resolution UAV videos, we extract pedestrian and cyclist trajectories. The PFPNet [30] is used to detect pedestrian and cyclist locations. Compared with current object detection methods, PFPNet constructs a feature pyramid by widening the network instead of increasing the depth, which aims to predict the locations of “hard-to-detect” objects, such as small (e.g., pedestrians are considerably smaller than vehicles), occluded (e.g., pedestrians can be in close proximity to each other), and blurred (e.g., the camera can shake) objects. Therefore, PFPNet is suitable for pedestrian, cyclist, and vehicle detection.

The architecture of PFPNet is illustrated in Figure 2. First, the base network produces a $W\times H$ output feature map with $C$ channels. Second, spatial pyramid pooling (SPP) [23] is employed to generate a wide feature pyramid (FP) pool with feature maps of various sizes. An additional feature abstraction strategy is applied to these feature maps in a parallel manner to balance the semantic abstraction levels. The multiscale context aggregation (MSCA) module rescales the feature maps to a uniform size and aggregates their contents to produce different levels of the final FP. Each MSCA module is followed by a prediction subnet, which is used to classify and localize objects, such as pedestrians and cyclists.

The base network is important for object detection and localization. We employ the prevalent VGGNet-16 [47] as the base network. The fully-connected layers are replaced with newly designed convolutional layers with downsampling. The modified VGGNet is pre-trained on the ILSVRC dataset [47]. A set of bottleneck layers [48] are employed in PFPNet for the feature transformation. In the bottleneck layer, a $1\times 1$ convolution is used to reduce the channel number to half of the original count. Batch normalization [49] without shift and the rectified linear unit (ReLU) [50] are used for normalization and activation.

The workflow to detect pedestrians, cyclists, and vehicles is displayed in Figure 2. Given a UAV image (a), the modified VGGNet-16 is employed as the base network to generate the input feature map. The high-dimensional FP pool (b) is formed via the SPP module, and the low-dimensional FP pool (c) is obtained by feature transformation with the bottleneck layer. Using these feature maps, these MSCA modules produce the final FP for multiscale object detection. The FP is fed into the prediction subnets to obtain the detected objects (e). Non-maximum suppression [51] is used to guarantee that each prediction corresponds to a single object.

After detection, we track the movements of pedestrians, cyclists, and vehicles at the crosswalks in videos. With the PFPNet results, the kernelized correlation filter (KCF) [52] is used to track a single object. The objective of a KCF tracker is to teach a classifier to distinguish the objects from their surrounding environment. Unlike other trackers that focus on the objects of interest, the KCF tracker develops circulant matrices to obtain additional environment samples (e.g., locations and scales of image patches) to train the classifier and produce the diagonal matrix with a discrete Fourier transformation, thus reducing the computational complexity. We select a single target from the PFPNet results, send its bounding box to the KCF, track pedestrian, cyclist, and vehicle movements, and collect its location in the UAV video to generate a two-dimensional point set. When the objects of interest are no longer on the crosswalk, the tracking is finished. These generated trajectories are reported in the final frame for further pedestrian movement simulation.

2.2. Pedestrian Movement Modeling

Using acquired trajectories of pedestrians, cyclists, and vehicles, the SFM is improved to simulate pedestrian movements at traffic intersections. Regarding pedestrians as particles satisfying the laws of mechanics, the classic SFM models the movements of a pedestrian $\alpha$ as being derived from the combined effects of the self-driving force (${\overrightarrow{f}}_i^0$), boundary (B) force (${\overrightarrow{f}}_B$), and repulsive force from the surrounding pedestrians $\beta$ (${\overrightarrow{f}}_{\alpha \beta }$). Given the strong influences of cyclists and vehicles on the movement of pedestrians at intersections, the improved SFM simulates two additional forces: the repulsive force of a cyclist $\gamma$ (${\overrightarrow{f}}_{\alpha \gamma }$) and the disturbing force of a right-turning vehicle $\omega$ on $\alpha ({\overrightarrow{f}}_{\alpha \omega })$. Hence, the joint force on a pedestrian, $\overrightarrow{f}$, is defined as Equation (1).

$$\overrightarrow{f}={\overrightarrow{f}}_i^0+{\overrightarrow{f}}_B+{\displaystyle \sum }_{\beta (\ne \alpha )}{\overrightarrow{f}}_{\alpha \beta }+{\displaystyle \sum }_{\gamma }{\overrightarrow{f}}_{\alpha \gamma }+{\displaystyle \sum }_{\omega }{\overrightarrow{f}}_{\alpha \omega }+\xi$$

(1)

where $\xi$ is the random fluctuation term of the joint force, which indicates the movement of a pedestrian that accidentally deviates from the normal movement. Figure 3 gives an example of the impacts on pedestrian movements considered.

2.2.1. Self-Driving Force

When pedestrians are moving toward their destinations at an expected speed, they are inevitably influenced by their surrounding environments. Therefore, a deviation emerges between the actual velocity and the expected velocity. In this context, following Helbing et al. [38], the self-driving force ${\overrightarrow{f}}_i^0$ tends to restore the actual velocity ${\overrightarrow{v}}_i$ to the expected velocity ${\overrightarrow{v}}_i^0$. Assuming that the time needed to restore the current speed to the expected speed value is ${\tau }_i$, the self-driving force of a pedestrian can be expressed as Equation (2), where M is the weight of the pedestrian.

$${\overrightarrow{f}}_i^0=\frac{M}{{\tau }_i}({\overrightarrow{v}}_i^0-{\overrightarrow{v}}_i)$$

(2)

2.2.2. Boundary Force

In general, pedestrians always walk within a crosswalk boundary. When an outward object exists at the crosswalk boundary, the boundary will exert a repulsive force to maintain a certain safe distance between the pedestrian and the boundary (Figure 4a). However, when there are high-density pedestrians, for example, 2–5 persons/m² at some intersections in Shenzhen, pedestrians may walk out of the crosswalk to avoid serious conflict with other pedestrians. Conversely, the boundary force becomes attractive rather than repulsive when a pedestrian steps out of the crosswalk (Figure 4b) to attract the pedestrian to return to the crosswalk [40]. Therefore, the force exerted by the boundary on pedestrians can be expressed as an exponentially decreasing function of the distance, as follows:

$${\overrightarrow{f}}_B=A_b{\overrightarrow{n}}_{\alpha }\exp (\frac{\Vert {\overrightarrow{P}}_{\alpha }-{\overrightarrow{P}}_B\Vert }{B_b})$$

(3)

where ${\overrightarrow{P}}_{\alpha }$ is current position of a pedestrian $\alpha$, ${\overrightarrow{P}}_B$ is the closest position at the crosswalk boundary to pedestrian $\alpha$, $\Vert {\overrightarrow{P}}_{\alpha }-{\overrightarrow{P}}_B\Vert$ is the Euclidean distance between them, $A_b$ is the boundary force strength, $B_b$ is the boundary force extent, and ${\overrightarrow{n}}_{\alpha }$ is the unit vector between the pedestrian and the boundary; ${\overrightarrow{n}}_{\alpha }$ becomes ${\overrightarrow{n}}_{B\alpha }$ with a direction from the boundary B to the pedestrian $\alpha$ when the pedestrian is within the boundary. Conversely, ${\overrightarrow{n}}_{\alpha }$ becomes ${\overrightarrow{n}}_{\alpha B}$ with a direction from pedestrian $\alpha$ to boundary B when the pedestrian steps out of the boundary.

2.2.3. Repulsive Force Exerted by Other Pedestrians

Pedestrians at traffic intersections tend to repel each other to create a comfortable walking space. The elliptical potential field of human interaction is generally employed to describe the interaction among pedestrians in the traditional SFM, which disregards crowding and bumping. Specifically, following Johansson et al. [38], the repulsive force between pedestrians $\alpha \mathrm{and}\beta$ can be expressed as follows:

$${\overrightarrow{f}}_{\alpha \beta }=-{\overrightarrow{\nabla }}_{{\overrightarrow{d}}_{\alpha \beta }}{V}_{\alpha \beta }(b_{\alpha \beta })$$

(4)

where $\nabla$ denotes the gradient operator, ${\overrightarrow{d}}_{\alpha \beta }$ is the vector from pedestrian $\beta$ to pedestrian $\alpha$, $V_{\alpha \beta }(b_{\alpha \beta })$ is the potential field.

Assuming that elliptical equipotential lines exist in this potential field I, an exponentially decreasing function should exist as Equation (6), depending on the short semi-axis of the ellipse ($b_{\alpha \beta }$), where $A_{\alpha \beta }$ is the strength of the repulsive force between two pedestrians $\alpha$ and $\beta$, $B_{\alpha \beta }$ is the extent of the repulsive force between two pedestrians.

$$V_{\alpha \beta }(b_{\alpha \beta })=A_{\alpha \beta }{B}_{\alpha \beta }\exp (-\frac{b_{\alpha \beta }}{B_{\alpha \beta }})$$

(5)

where $b_{\alpha \beta }$ is the short semi-axis of the elliptical potential field defined as:

$$b_{\alpha \beta }=\frac{\sqrt{{(\Vert {\overrightarrow{d}}_{\alpha \beta }\Vert +\Vert {\overrightarrow{d}}_{\alpha \beta }-{\overrightarrow{v}}_{\beta }∆t\Vert )}^2-{({\overrightarrow{v}}_{\beta }∆t)}^2}}{2}$$

(6)

where ${\overrightarrow{v}}_{\beta }$ is the walking velocity of pedestrian $\beta ,$ $∆t$ is the simulation time step.

According to the relationship between the potential field and the force, we obtain:

$${\overrightarrow{f}}_{\alpha \beta }(\overrightarrow{d})=-{\overrightarrow{\nabla }}_{{\overrightarrow{d}}_{\alpha \beta }}{V}_{\alpha \beta }(b_{\alpha \beta })=-\frac{d{V}_{\alpha \beta }(b_{\alpha \beta })}{d{b}_{\alpha \beta }}{\overrightarrow{\nabla }}_{{\overrightarrow{d}}_{\alpha \beta }}{b}_{\alpha \beta }(\overrightarrow{d})$$

(7)

Combining $\Vert \overrightarrow{z}\Vert =\sqrt{{(\overrightarrow{z})}^2}$ and ${\nabla }_z\Vert \overrightarrow{z}\Vert =\overrightarrow{z}/\sqrt{{(\overrightarrow{z})}^2}$ = $\widehat{\overrightarrow{z}}$ to simplify the operation (7), thus, we obtain:

$${\overrightarrow{f}}_{\alpha \beta }(\overrightarrow{d})=A_{\alpha \beta }\exp (-\frac{b_{\alpha \beta }}{B_{\alpha \beta }})·\frac{\Vert \overrightarrow{d}\Vert +\Vert \overrightarrow{d}-{\overrightarrow{v}}_{\beta }∆t\Vert }{2b_{\alpha \beta }}·\frac{1}{2}\left(\frac{\overrightarrow{d}}{\Vert \overrightarrow{d}\Vert }+\frac{\overrightarrow{d}-{\overrightarrow{v}}_{\beta }∆t}{\Vert \overrightarrow{d}-{\overrightarrow{v}}_{\beta }∆t\Vert }\right)$$

(8)

2.2.4. Repulsive Force Exerted by Cyclists on Pedestrians

According to China transportation regulations, cyclists should walk in the crosswalk at traffic intersections, following traffic lights. Cyclists have an equal road right to the pedestrian. Consequently, cyclists have an important impact on pedestrian movements at complex traffic intersections. Given the potential conflict between pedestrian movements and cyclists, we improve the classic SFM and assume another potential field II for the repulsive force of a cyclist $\gamma$, ${\overrightarrow{f}}_{\alpha \gamma }$, as follows:

$${\overrightarrow{f}}_{\alpha \gamma }=-{\overrightarrow{\nabla }}_{{\overrightarrow{d}}_{\alpha \gamma }}{V}_{\alpha \gamma }(b_{\alpha \gamma })$$

(9)

where ${\overrightarrow{d}}_{\alpha \gamma }$ is the vector from the cyclist $\gamma$ to pedestrian $\alpha$, $V_{\alpha \gamma }(b_{\alpha \gamma })$ is the potential field following the exponentially decreasing function as follows:

$$V_{\alpha \gamma }(b_{\alpha \gamma })=A_{\alpha \gamma }{B}_{\alpha \gamma }\exp (-\frac{b_{\alpha \gamma }}{B_{\alpha \gamma }})$$

(10)

Considering the difference in speed between cyclists and pedestrians, the short semi-axis of the elliptical potential field II ($b_{\alpha \gamma }$) is assumed to be:

$$b_{\alpha \gamma }=\frac{\sqrt{{(\Vert {\overrightarrow{d}}_{\alpha \gamma }\Vert +\Vert {\overrightarrow{d}}_{\alpha \gamma }-({\overrightarrow{v}}_{\gamma }-{\overrightarrow{v}}_{\alpha })∆t\Vert )}^2-{\left[({\overrightarrow{v}}_{\gamma }-{\overrightarrow{v}}_{\alpha })∆t\right]}^2}}{2}$$

(11)

where ${\overrightarrow{d}}_{\alpha \gamma }$ is the vector from cyclist $\gamma$ to pedestrian $\alpha$, ${\overrightarrow{v}}_{\gamma }$ is the velocity of the cyclist, and ${\overrightarrow{v}}_{\alpha }$ is the walking velocity of the pedestrian.

To verify the superiority of elliptical potential field II over the elliptical potential field I in describing the cyclist impact mode, two scenarios are simulated to compare the two potential fields. Assuming that the cyclist is stationary, ${\overrightarrow{v}}_{\gamma }=0$, the short semi-axis $b_{\alpha \gamma }$ is obtained according to Equation (11) in elliptical potential field II.

$$b_{\alpha \gamma }=\frac{\sqrt{{(\Vert {\overrightarrow{d}}_{\alpha \gamma }\Vert +\Vert {\overrightarrow{d}}_{\alpha \gamma }+{\overrightarrow{v}}_{\alpha }∆t\Vert )}^2-{(-{\overrightarrow{v}}_{\alpha }∆t)}^2}}{2}$$

(12)

According to Equation (6) in the elliptical potential field I, we can obtain:

$$b_{\alpha \gamma }=\Vert {\overrightarrow{d}}_{\alpha \gamma }\Vert$$

(13)

(1) Scenario 1

When the pedestrian $\alpha$ moves in the same or opposite direction as a cyclist $\gamma$ with a common speed of ${\overrightarrow{v}}_{\alpha }$, they share the same values of $\Vert {\overrightarrow{d}}_{\alpha \gamma }\Vert$ and ${\overrightarrow{v}}_{\alpha }∆t$ but have slightly different values of $r_i=\Vert {\overrightarrow{d}}_{\alpha \gamma }+{\overrightarrow{v}}_{\alpha }∆t\Vert$. As shown in Figure 5a, $r_1$ is smaller than $r_2$. As the repulsive force monotonously decreases with the short semi-axis $b_{\alpha \gamma }$, the repulsive force exerted by the cyclist on the pedestrian when they are moving in the opposite directions ($f_1$) is greater than the force when they are moving in the same direction ($f_2$), which is consistent with the real-world experience of the pedestrian. In elliptical potential field I, $b_{\alpha \gamma }=\Vert {\overrightarrow{d}}_{\alpha \gamma }\Vert$ is a constant value, which indicates equal disturbing forces in both scenarios and no influence of the relative movement direction on the disturbing force exerted by the cyclist.

(2) Scenario 2

When pedestrian $\mathsf{\alpha }$ moves toward cyclists with different speeds of $v_{\alpha }^1$ and $v_{\alpha }^2$ ($v_{\alpha }^1<v_{\alpha }^2$) (Figure 5b), the pedestrian will respond to the larger repulsive force exerted by the cyclist moving at a higher speed, which is also consistent with real-world experiences. Conversely, the elliptical potential field I is not sensitive to speed, and the repulsive force exerted by a cyclist is not related to the speed of pedestrian $\alpha$ and only related to the distance between the cyclist and the pedestrian.

2.2.5. Vehicle Force

According to Chinese traffic law, vehicles in China are allowed to turn right at a traffic signal even if it is red. However, drivers do not always yield to pedestrians. Instead, they tend to take advantage of short pedestrian clearance intervals to pass through intersections; therefore, they exert repulsive forces on pedestrians and force pedestrians to decelerate and avoid the vehicles. Given the different speeds of vehicles and pedestrians, similarly, the force of a turning vehicle $\omega$ on a pedestrian $\alpha$ in elliptical potential field II can be expressed as follows:

$${\overrightarrow{f}}_{\alpha \omega }=-{\overrightarrow{\nabla }}_{{\overrightarrow{d}}_{\alpha \omega }}{V}_{\alpha \omega }(b_{\alpha \omega })$$

(14)

where ${\overrightarrow{d}}_{\alpha \omega }$ is the vector from the vehicle $\omega$ to pedestrian $\alpha$,$V_{\alpha \omega }(b_{\alpha \omega })$ is the potential field following the exponentially decreasing function as below:

$$V_{\alpha \omega }(b_{\alpha \omega })=A_{\alpha \omega }{B}_{\alpha \omega }\exp (-\frac{b_{\alpha \omega }}{B_{\alpha \omega }})$$

(15)

where $A_{\alpha \omega }$ is the strength of the repulsive of a vehicle $\omega$ on pedestrian $\alpha$, and $B_{\alpha \omega }$ is the extent of the vehicle force.

The short semi-axis $b_{\alpha \omega }$ of the ellipse of ${\overrightarrow{f}}_{\alpha \omega }$ can be expressed as follows:

$$b_{\alpha \omega }=\frac{\sqrt{{(\Vert {\overrightarrow{d}}_{\alpha \omega }\Vert +\Vert {\overrightarrow{d}}_{\alpha \omega }-({\overrightarrow{v}}_{\omega }-{\overrightarrow{v}}_{\alpha })∆t\Vert )}^2-{\left[({\overrightarrow{v}}_{\omega }-{\overrightarrow{v}}_{\alpha })∆t\right]}^2}}{2}$$

(16)

where ${\overrightarrow{v}}_{\omega }$ is the velocity of the vehicle, and ${\overrightarrow{v}}_{\alpha }$ is the walking velocity of the pedestrian.

As shown in Figure 6, right-turning vehicles are assumed to be stationary at different positions. When the pedestrian moves toward the opposite exit at the speed ${\overrightarrow{v}}_{\alpha }$, a shorter distance between the pedestrian and the vehicle corresponds to a smaller elliptical short semi-axis $b_{\alpha \omega }$ of the force potential field and to a larger force experienced by the pedestrian. The repulsive force exerted by the right-turning vehicle is inversely proportional to the distance between the vehicle and the pedestrian.

2.3. Simulation of Pedestrian Movements at Complex Traffic Intersections

According to the aforementioned influential factors of pedestrian movements, the classic SFM model is improved to generate an acceleration vector of pedestrians to simulate the movements at complex traffic intersections. To evaluate the simulation results, the Verlet algorithm is employed to estimate the pedestrian trajectory. The purpose of the Verlet algorithm is to update the position of a pedestrian $x$($t+∆h$) at time $t+∆h$ using the position $x$($t$) and the acceleration a($t$) of the pedestrian at time $t$ and the sample interval $∆h$. First, we perform the Taylor expansion on $x(t+∆h)$ and $x(t-∆h)$:

$$x(t+∆h)=x(t)+v(t)∆h+\frac{a(t)∆h^2}{2}+\frac{b(t)∆h^3}{3!}$$

(17)

$$x(t-∆h)=x(t)-v(t)∆h+\frac{a(t)∆h^2}{2}-\frac{b(t)∆h^3}{3!}$$

(18)

By adding these two expressions, we can obtain the following positional expression:

$$x(t+∆h)=2x(t)-x(t-∆h)+a(t)∆h^2$$

(19)

By differentiating these two expressions, we obtain the speed and acceleration:

$$v(t+∆h)=v(t)+a(t)∆h+\frac{b(t)∆h^2}{2}$$

(20)

$$a(t+∆h)=a(t)+b(t)∆h$$

(21)

By substituting b(t) in Equation (22) into Equation (20), we can obtain updated equations for pedestrian speed and position:

$$v(t+∆h)=v(t)+1/2\left[a(t)+a(t+∆h)\right]∆h$$

(22)

$$x(t+∆h)=x(t)+1/2\left[v(t)+v(t+∆h)\right]∆h$$

(23)

where $∆h$ is a fixed time interval and accelerations $a(t)$ and $a(t+∆h)$ are calculated by the improved SFM.

2.4. Calibration of the Pedestrian-Cyclist Conflict Model

The improved SFM involves a variety of parameters, including the free speed of a pedestrian and the strength and range of forces exerted by the boundary, cyclists, other pedestrians, and vehicles (see

Table 1. Count results from the test UAV video

Type	TP	FP	FN	Correctness	Completeness	Quality
Pedestrian	132	5	3	0.964	0.978	0.943
Cyclist	29	2	2	0.935	0.935	0.879
Vehicle	37	0	0	1.0	1.0	1.0
Overall	198	7	5	0.966	0.975	0.943

). The MLE method, a widely used parameter calibration method based on statistical principles, is employed, using the extracted trajectories of pedestrians, cyclists, and vehicles. The position of a pedestrian in the next simulation time step, ${\overrightarrow{P}}_{\alpha }(t_{k+1})$, is assumed to be predicted by the model parameter $\theta$. The moving distance from points ${\overrightarrow{P}}_{\alpha }(t_k)$ and ${\overrightarrow{P}}_{\alpha }(t_{k+1})$ obeys the normal distribution with a mean $\mu$ and a standard deviation $\sigma$. According to the observed trajectories, the mean $\mu$ and a standard deviation $\sigma$ of the single step distance $∆d_{\alpha }(\theta )$ are estimated. The likelihood function concerning $\theta$ is obtained:

$$\begin{array}{ll}\hfill L(\theta )& =\prod \frac{1}{\sigma \sqrt{2\pi }}e{(\frac{-{(∆d_{\alpha }(\theta )-\mu )}^2}{2{\sigma }^2})}^{}\\ & =\frac{{(2\pi )}^{-\frac{n}{2}}}{{({\sigma }^2)}^{-\frac{n}{2}}}\exp (-\frac{\sum {(\Vert {\overrightarrow{d}}_{\alpha }^{obs}(t_k+1)-{\overrightarrow{d}}_{\alpha }^{obs}(t_k)\Vert -\mu )}^2}{2{\sigma }^2})\end{array}$$

(24)

For simplicity, both sides of Equation (24) are converted by logarithmic functions. Therefore, the value of $\theta$ that corresponds to the maximum of $L(\theta )$ can be obtained:

$$\ln L=-\frac{1}{2}n\ln (2\pi )-\frac{1}{2}n\ln ({\sigma }^2)-\frac{\sum {(\Vert {\overrightarrow{d}}_{\alpha }^{obs}(t_k+1)-{\overrightarrow{d}}_{\alpha }^{obs}(t_k)\Vert -\mu )}^2}{2{\sigma }^2}$$

(25)

3. Experiment and Result Analysis

3.1. Experimental Configuration

To evaluate the performance of the proposed approach, this study used the DJI Inspire 1 Pro to conduct experiments in the high-tech development zone at Nanshan district, Shenzhen. The used DJI Inspire 1 Pro is equipped with a GPS receiver and a built-in inertial measurement unit (IMU), which incorporates both a 6-axis gyroscope and an accelerometer for movement compensation. The camera mounted by the used DJI Inspire 1 Pro is capable of stably recording road traffic at 4K (3840 X 2187) resolution (30fps). The experimental area is located in the center of the Science and Technology Park, which is a key intersectional area in terms of the massive traffic around the commercial and industrial parks. Data were acquired by experienced drone pilots during peak hours (on- and off-duty hours and lunch breaks) when the pedestrian flow is large to ensure sufficient numbers of pedestrians, cyclists, and vehicles. The hovering height of the UAV was set to 50 meters above the ground. Videos of five road intersections were captured and processed. Flying permission was guaranteed by the local transportation administration. Extreme weather events (i.e., rain, winds, etc.) were avoided to ensure the safety.

3.2. Pedestrian and Cyclist Detection and Localization

To build the training set, we subsampled the raw UAV videos every 60 frames to generate the image set and manually annotated categories and locations of ground objects, such as pedestrians, cyclists, and vehicles. We trained PFPNet using these annotated images. Then, testing images were fed into the trained detector to produce sets of predicted boxes with class confidence scores, which were used to generate final trajectories.

To evaluate the performance of the proposed detection and tracking algorithm, we counted the number of pedestrians, cyclists, and vehicles in a sample of the obtained UAV videos. To be more specific, we used PFPNet to detect the ground objects (203 in total) and employed the KCF algorithm to keep track of them. We used the bounding box coordinates to mark the type and the location of the tracked objects. Hence, counting was done by simply tallying the number of bounding boxes. We quantitatively evaluated the counting result via the correctness (Cor), completeness (Com), and quality (Qua), which are defined in [45] as:

$$\mathrm{Correctness}=\frac{TP}{TP+FP}$$

(26)

$$\mathrm{Completeness}=\frac{TP}{TP+FN}$$

(27)

$$\mathrm{Quality}=\frac{TP}{TP + FP + FN}$$

(28)

The true positives value (TP) denotes the number of correctly detected ground objects, the false positives value (FP) represents the number of invalid detections, and the false negatives value (FN) denotes the number of missed objects. Among the three evaluation criteria, quality is most important, since it considers both the correctness and completeness of detection algorithms.

We report the count results for all types of ground objects in Table 1. The detection and tracking algorithm works very well, and nearly all types of ground objects are accurately detected and tracked, especially vehicles, which achieve 100% quality. We noted that a few pedestrians and cyclists cannot be correctly recognized as the quality of the results for pedestrian and cyclist are 94.3% and 87.9%, respectively. These results occurred because some bicycles were largely occluded by their cyclists, making them look very similar to pedestrians from the bird’s eye view of the high-resolution UAV videos.

After processing the UAV videos, we extracted 2134 trajectories of pedestrians, cyclists, and vehicles. A total of 203 trajectories were of pedestrians and were impacted by cyclists, vehicles, or both. Figure 7 provides an example of an extracted pedestrian trajectory at one intersection.

3.3. Performance of the Improved Social Force Model

To assess the performance of the improved SFM, 80% of the pedestrian trajectories were selected to calibrate the SFM. The remaining 20% of the pedestrian trajectories were employed to simulate pedestrian movements at complex traffic intersections to evaluate the SFM’s accuracy.

The improved SFM contains a set of parameters, including the free speed of a pedestrian and the strength and range of forces exerted by the boundary, cyclists, other pedestrians, and vehicles. The parameters that are measurable but difficult to derive from the observed dataset were set by referring to related studies. Other parameters that do not have concrete physical meanings but can be indirectly derived from the pedestrian trajectories were calibrated by MLE in the MATLAB program. According to related studies, the free passage speed can be 1.5 m/s and the time needed for a pedestrian to recover from their actual speed to the expected speed is 0.5 s. The p-value of the strength and range of each force is less than 0.05 at the 95% confidence level; all the parameters in the improved SFM are significant.

Both the classic and improved SFM were used to simulate pedestrian positions at complex traffic intersections, assuming a pedestrian mass ranging from 45 to 75 kilograms and a simulation time step ∆t of 0.2 s. The simulation results were compared with the corresponding pedestrian trajectories acquired by the UAV, in terms of absolute positioning accuracy and mean average percent error (MAPE). The obtained results are reported in

Table 2. Accuracy of simulation results using the social force model

	Classical SFM	Improved SFM
Positioning accuracy (meters)	0.33	0.25
MAPE	12.43%	9.04%

. The results demonstrate that the classical SFM achieves a positioning accuracy of 0.33 meters, with a MAPE of 12.43%. By considering additional influences of cyclists and vehicles, the improved SFM provides better performance, with a positioning accuracy of 0.25 meters and a MAPE of 9.04%.

Figure 8 provides an example to evaluate the performance of the improved SFM. Figure 8a shows the recognition results of the pedestrian movements influenced by the boundary and cyclists. Figure 8b illustrates the force that the pedestrian experiences in the improved SFM. The boundary force always exists when the pedestrian is crossing an intersection. The boundary force behaves as an attractive force during the first 17 s when the pedestrian is within the crosswalk, and the force strength is proportional to the distance between the pedestrian and the boundary. As the pedestrian enters the crosswalk, the boundary force becomes a repulsive force with a strength that is inversely proportional to the distance between the pedestrian and the boundary. The initial repulsive force exerted by cyclists is 0. During the time interval between 19 and 32 s, conflicts among the pedestrian and cyclists ${\gamma }_1$ and ${\gamma }_2$ emerge. Because the distance between them is small, the repulsive forces suddenly increase. Once the pedestrian passes a cyclist, the force exerted by the cyclist gradually reduces to zero. Figure 8c shows the estimated position and trajectory of pedestrian ${\alpha }_1$ in different models. To visually analyze the model performance, the MAPEs of estimated trajectories in the x and y directions are calculated. The maximum error of the improved SFM is 0.21 m, less than 0.47 m for classic SFM. The difference between the two models is primarily observed in the conflict area between pedestrians and cyclists. The improved model can estimate the pedestrian position and trajectory better than the traditional model.

Figure 9 shows another pedestrian’s movement affected by the boundary, cyclists, and nearby pedestrians. Figure 9a illustrates the recognized pedestrians and cyclists. Figure 9b shows that the cyclist force has the largest influence on the pedestrian’s movement, which is determined by their speeds and the distance between the cyclist and the pedestrian. The force exerted by surrounding pedestrians has the second largest impact on the pedestrian’s movement, which primarily occurs during the first 40 s when conflicts among pedestrians emerge; its strength is inversely proportional to the distance between two pedestrians. The boundary force always exists as the pedestrian remains within the crosswalk. Figure 9c displays the extracted and simulated trajectories. The maximum error of the improved SFM is 0.58 m, less than 0.88 m for classic SFM, which demonstrates the better performance of the improved SFM in describing the pedestrian movement as impacted by complex disturbances. In addition, the MAPE, which reflects the average error of the estimated position in each step of the simulation, is evenly distributed in the conflict area throughout the simulation process, which causes an indistinct difference in the model simulation accuracy.

4. Conclusions

Simulation of intersections is important to track and simulate pedestrian movements at these intersections. Correspondingly, a UAV-based method is proposed to track and simulate pedestrian movements at complex traffic intersections. High-resolution UAV videos of the intersections are employed to extract high-resolution movements of pedestrians, cyclists, and vehicles. Given the potential conflicts among pedestrians, cyclists, and vehicles, an improved social force model that considers the surrounding pedestrians, boundaries, cyclists, and right-turning vehicles is proposed and calibrated for pedestrian movement simulation. Videos acquired of intersections in Shenzhen City are utilized for high-precision pedestrian movement simulation. The results demonstrate that

In this study, a UAV is employed as a pedestrian monitoring platform to provide high-precision pedestrian trajectories at complex traffic intersections for the improved social force model, which significantly improves the simulation accuracy. The following aspects require additional attention in future research: (1) the UAV-based method is subject to slight drift during flight, which reduces the accuracy of trajectory data acquisition. Ground control points can assist in UAV image correction and improve absolute positioning accuracy. On the other hand, additional factors associated with pedestrians movements should be included in the presented SFM, such as pedestrian’s psychology and pedestrian density. (2) Following the connected UAV approach [53,54], the experiment at one traffic intersection will be extended to monitoring and simulating pedestrian movements at a set of intersections, simultaneously using connected UAVs [55] and cloud services [43].