Towards Collecting Big Data for Remote Photoplethysmography

Abstract

Remote photoplethysmography (rPPG) is a technique for non-contact estimation of human vital signs in video. It enriches knowledge about human state and makes interpretations of actions in human-computer interaction more accurate. An approach to distributed collection of rPPG dataset is proposed along with a central hub where data is accumulated as links to local storages hosted by participating organizations. An instrument for rPPG data collection is developed and described. It is an Android application, which captures dual camera video from front and rear cameras simultaneously. The front camera captures facial video while the rear camera with flash turned on captures a contact finger video. Facial videos constitute a dataset, while ground truth blood volume pulse (BVP) characteristics can be obtained by the analysis of correspondent finger videos. Such approach allows overcoming organizational and technical limitations of biometric data collection and hosting.

Appendices

Appendix

1. Non-public rPPG Datasets

Table 2 lists rPPG-suitable datasets declared as public but have not been accessed in part or in full due to various reasons. The estimated total video duration in a dataset is based on its description in the corresponding paper.

In addition to the above, there are dozens of private datasets mentioned in one or several papers on rPPG: HNU [64], HR-D [44], BSIPL-RPPG [47], and untitled ones [2, 6, 9, 10, 12,13,14,15, 48, 51, 52, 55,56,57, 59,60,61]. The both training and testing datasets of the RePSS competition [29] are also considered private because they have been publicly available only during the competition.

2. Dual-Camera Collector

Dual-Camera Collector or DCC-Client is an open source (https://github.com/Assargadon/dcc-client) Android mobile application that records video from both cameras of a mobile device simultaneously (Fig. 2). It has, though, several key differences from a generic video recorder, even if it is able to capture the video from two cameras simultaneously.

1.1 App Features

First, data records are designed to be anonymous, keeping track of a metadata related to experiment conditions. Users have the ability to provide a desired metadata. Other metadata is determined automatically.

Table 2. Datasets on rPPG that were declared publicly available but have not been fully accessed$^\text {a}$.

Second, there is no need to have a high-resolution image for a camera capturing a finger, but a flash constantly turned on would be of help. Despite of rear camera usually offers higher capture resolution than a front one, it was considered to use front camera to capture face video and rear camera to capture finger due to the following reasons:

• Usage of a face-oriented (front) camera to record a face video makes the user able to operate the UI of the application during the recording. Therefore, a user is able to capture a record of themselves without an assistant, on their own.

• A front camera usually has enough resolution to capture the user face at a regular distance from the device—and this is its purpose.

• A rear camera almost always has a flashlight, which is useful to record contact BVP signals. Front cameras sometimes have flash too—but less often.

Third, the application provides two video data streams obtained simultaneously from front and rear cameras. While heart rate value is only defined in a sliding window and there is no strict requirement for per-frame synchronization, some methods and studies could be very sensitive to a time offset, for example:

• Training of deep learning models to extract BVP signal in facial video based on asynchronous ground truth contact BVP signal obtained by rear camera.

• Studies based on measurement of the phase shift between BVP signals from finger and face.

• Methods of Heart Rate Variability estimation.

Fourth, the application collects the standardized data to the permanent dataset storage.

1.2 User Experience

The application determines if a device has the capability to capture videos from two cameras simultaneously—unfortunately, not all Android devices are able to do it. If no, an alert is presented to the user, which means that the device cannot be used to collect the dataset.

Fig. 4.

Screenshots of the DCC-Client UI: User Metadata form (a); Process of Video Capture (b).

The application has a simple form containing fields of “User ID”, “Year of Birth” and “Gender” (Fig. 4(a)). “User ID” refers to a surrogate identifier that has the purpose of tracking the same persons, while keeping the anonymity of their personalities. For example, it may be “1” for the first subject, “2” for the second subject and so on. If another record of the first subject is performed (maybe even on a different device), User ID “1” should be used again.

After the user enters the required information, a live video preview is shown in Fig. 4(b). User can start recording by tapping the respective button in the UI. When a recording is to be finished, the user taps the button again.

The dataset entry is represented in a form of three files that are named with a unique prefix <User ID>-<Start timestamp>. This ensures that the file group of the single recording is easily recognized. Two video files contain the streams for the subject’s face and finger. Third one, “.json”, contains metadata about both subject (gender and year of birth, as explained above), device (model, anonymous device unique identifier), and record itself (start and final timestamps of the recording, resolutions of face and finger videos).

After performing one or several recordings, one can use a data cable or another file transfer mechanism to extract recordings from the device and put it to the permanent dataset storage. There is no need to avoid naming collisions because file names are generated to be unique.