1. Introduction
Epilepsy is a disorder of the central nervous system where intermittent, and in general, unpredictable transitions to paroxysmal states, called seizures, can occur. These states elicit synchronous activity of the brain; they can recruit different areas of brain tissue and have a variety of clinical manifestations. In some cases, the epileptic condition is benign but in more severe forms of the disorder, patients have a significant risk of injuries, and even death, as a result of the seizures [
1,
2]. Especially vulnerable are the sufferers that experience convulsive fits or seizures. During those events, timely assistance can be critical for the health and wellbeing of the patient. Several products on the market provide alerts for convulsive seizures [
3,
4]. Some use wearable sensors typically attached to the patient’s arm, others register vibrations of the bed and alert for nocturnal fits. In all these products, direct or indirect contact between the patient and the sensors is required for the proper operating of the system. Such systems are therefore sensitive to accidental or deliberate misplacement of the primary sensor. In the case of wearable sensors, regular charging or battery change is also necessary. All these requirements make the use of contact sensors in some cases difficult, and may lead to additional workload from the caregivers. In many care facilities, continuous video monitoring is used for the safety of the patients. This involves constant time-consuming alertness of the personnel and may compromise the privacy of the patients.
To address the above issues in our care facility, we have developed a system for remote sensing and alerting for convulsive epileptic seizures. The system uses a continuous stream of images from a video camera providing a relatively constant rate of frame acquisition. During dark periods, the camera automatically switches to active infrared enhanced acquisition. Proprietary algorithms process, in real time, the image sequence and upon certain conditions may issue an alert. A standard nursing alert system receives the signal of an epileptic event and dispatches it to the caregiver’s observation post, and, if required, to mobile devices as well. We explain in detail the overall concept in the next section. The processing algorithms were tested earlier on pre-recorded video sequences screened by clinical experts [
5]. Results from night observations in a care facility [
6] and in children at home [
7] have proven that video-based seizure detection can provide sufficient effectiveness and security. As it is a non-contact, non-obstructive approach, it can also operate in combination with other detection modalities [
8], depending on the particular situation.
In short, our method uses optical flow type of reconstructive analysis that infers the rates of changes of a limited number of group transformation parameters, such as translations, rotations, dilatations, and shear transformations, from the video sequence. Subsequently, the obtained signals undergo time-frequency filtering using wavelet decomposition. Finally, we analyze the filtered signal that represents the likelihood of a convulsive seizure and upon pre-defined conditions produce an alert signal that deploys directly via a standard “dry contact” interface to an existing nurse alerting system.
The essential innovation proposed in this contribution is the addition of an adaptive or machine learning (ML) functionality to the system. We achieved this on two levels, by tuning the decision parameters for raising an alert, and on a deeper level, by dynamic selection of the wavelet filter parameters. We have realized an on-going system adaptation procedure that operates in parallel with the uninterrupted functioning of the alerting process. Unlike other ML paradigms, we do not require a training set of data to be available prior to the initial operational state. Our system acquires its own set of events and uses them to improve its performance. We introduce two possible scenarios of application. The first option relies on a supervised validation of the detected events where a qualified observer identifies the true detections from the false alerts. This involves time-consuming visual inspection on the recorded video fragments associated with the alarms. Alternatively, an automated off-line classification is proposed. In this case, the system works completely autonomously. When required, the operator’s inspection only provides an assessment of the system performance. We achieved the automated classification by estimating the average distance to similar events surrounding each detected event. Assuming that epileptic movements are compulsive and therefore more stereotyped than the “normal” movements of the body, the average distance provides a simple but effective clustering criterion that separates true from false detections. The distance metric uses the total optical flow information collected during the system operation. To summarize, the novelties proposed here are: (1) introduction and implementation of an algorithm for continuous adaptation of the seizure detection parameters; and (2) introduction and implementation of an autonomous procedure for retrospective validation of the detected events. In addition, this work implements a real-time version of a previously published original algorithm for optical flow group velocities reconstruction.
The organization of the rest of the paper is as follows. In
Section 2,
Section 2.1,
Section 2.2 and
Section 2.3 we present the clinical case considered as well as the overall layout of the seizure detection system and brief technical details related to the hardware as implemented.
Section 2.4,
Section 2.5 and
Section 2.6 describe the “rigid” version of the algorithm comprising optical flow reconstruction of group parameters, wavelet filtering and the alert decision procedure.
Section 2.7 introduces the novel approach of frequency interval selection as part of our adaptive algorithm.
Section 2.8 presents our decision parameter optimization approach.
Section 2.9 contains the formulation of our novel procedure for unsupervised retrospective validation of detected events based on clustering technique.
Section 3 presents our results from the trial. First in
Section 3.1 we show the detection statistics of the default “rigid” algorithm.
Section 3.2 contains the results obtained after applying supervised parameter optimization. In
Section 3.3, we introduce results from the autonomous, unsupervised adaptive algorithm. In addition to the detection statistics, we provide assessment of the automated event classification from two separate sequences of events during the trial.
Section 4 offers a discussion of our overall strategy, open problems, limitation and possible future directions of our quest for a reliable seizure alerting device.. Finally, we summarize our conclusions in
Section 5.
2. Materials and Methods
2.1. Patient Data
In this technical communication, we present results from a case study of single patient data as a proof-of-concept. The subject, a 46-year-old male, suffered from pharmacologically intractable epilepsy with a spectrum of motor disturbances. In addition to the tonic–clonic seizures of an average frequency of 1.5/day, the patient elicited successions of short convulsive movements at irregular periods during sleep. He also experienced impaired arm movement due to the frequent tremors. The above conditions make the selective detection of only the major motor TC seizures a significant challenge. Most detectors cannot distinguish easily between the various motor paroxysms.
2.2. Processing Flow
The overall processing flow and system layout shown on
Figure 1 represents both the supervised (panel a) and unsupervised (b) operational modes.
Deployable application written on Matlab®, version 2022b (The Mathworks Inc., Natick, MA, USA) incorporates all software modules and routines. The application runs on a HP Pavilion Desktop 595 personal computer equipped with 3GHz Intel I7 quad-core processor, 16Gb internal memory. The operational system is Windows 10 Home, version 22H2. Parallel processing for matrix calculus uses a NVIDEA GeForce GTX 1050 graphic card. For the connectivity to the nurse alerting system (De Heer Medicom®) we use DLP-IOR4 (DLP Design Inc., McKinney, TX, USA) USB based latching relay module operating in Normal-Open protocol.
2.3. Video Acquisition
In the “field trial” setting, we used a USB camera (720P USB2.0 OmniVision OV9712 Color CMOS Sensor USB Camera, AILIPU TECHNOLOGY CO., LTD., Shenzhen, China) that provides fixed frame rate and day-night infrared LED capabilities. The frame rate was constant at 24 frames per minute; resolution was 640 by 480 pixels, subsequently down sampled to 320 by 240. To this end, we applied a 2 × 2-pixel block averaging to reduce the image resolution. The post-acquisition software processes image sequences of given length in sequential cycles (in this work we used 36 frames per cycle corresponding to 1.5 s). For each processing cycle, an optical flow reconstruction algorithm explained in brief below applies to each consecutive pair of frames. The acquisition cycles of image sequences follow each other continuously but their processing is mutually independent.
2.4. Optical Flow Group Velocities Reconstruction
The theory and proprietary algorithm used to derive motion rates from image sequences are introduced in [
9,
10]. Here we recall the main idea.
Optical flow is an image-processing algorithm that aims at reconstructing the velocities of moving objects from the time-changes in the sequences of video images taken from those objects. If we are interested in only certain overall global movement rates, in our case, those of the two translations, rotation, dilatation and the two shear transformations, we can apply a direct reconstruction algorithm that saves a lot of computational power and complexity. Compared to other optical flow techniques, we avoid the reconstruction of local velocities vector field in all image locations (pixels). As a result, applying the algorithm on the image sequence produces six time series representing the rates of changes (group velocities) of the six two-dimensional linear inhomogeneous transformations.
Our algorithm uses all the available spectral components of the video sequence (for our case just red, green, and blue components), in contrast to other available algorithms that process only the intensity values in the images.
2.5. Wavelet Decomposition and Filtering. Seizure Biomarker
We use a set of Gabor wavelets with exponentially increasing wavelengths
For the exact definitions and normalizations, we refer to earlier publications [
3] and here we note that the wavelet spectrum in (2) is a time average along each image sequence denoted with q.
Next, we define the “epileptic content” as the fraction of the wavelet energy contained in the frequency range defined here as
.
In the “rigid” application, as well as an initial setting for the adaptive scheme, we use
Hz. To compensate for different frequency ranges that may be used, we calculated also the same quantity in (3) but for a signal with “flat” spectrum representing random noisy input. Then we rescale the epileptic marker as
Here is the relative wavelet spectral power of a white noise.
Note that in (3) the quantity q is a discrete index representing the frame sequence number and corresponds, as stated earlier, to a time window of approximately 1.5 s.
2.6. Event Detection
We use three parameters [N,n,T] to detect an event (seizure alert) in real time. At each time instance, we take the seizure marker (4) in the N preceding windows. If from those, N, or at least n have values > T, an event is generated and eventually (if within the time selected for alerts) sent as an alert. The default values are [7 6 0.4]. This corresponds to a criterion that detects if from the past 10.5 s at least 9 s contain an epileptic “charge” (4) higher than 0.4. These are the values for the rigid mode as well as for the initial setting in the adaptive mode.
Figure 2 is an illustration of the event detection algorithm.
2.7. Adaptive Frequency Range Selection
If we have a validated set {S} of seizure events, we can estimate the relevant frequency range used in Equations (3) and (4) as follows.
First, we average the frequency mean-subtracted spectrum over a number of sequences q around the seizure events
Here we define
In Equation (5) subscript s is the seizure event label. We also introduce a “penalty” for high variability around the average values. Conveniently, we take 10 windows before and 15 windows after each event for the time averaging over q.
Next, we determine the optimal frequency interval for seizure event detection as
2.8. Selection of Optimal Parameters for Event Detection
Once we have the biomarker for epileptic movements from Equation (4) which is calculated from either the default or the “trained” frequency range (6), we can optimize the decision triple of parameters {N,n,Tt} by introducing the following cost-function
In the second equation, the value of the parameter n, the filling number, is derived for each pair of (T,N) values as the minimal number of threshold-exceeding epochs n over all detected and validated seizure events. This way we can guarantee that, for any choice of the pair (T,N), all the previously detected true seizure events will be preserved. Therefore, we can postulate the optimized choice of parameters as:
2.9. Unsupervised Validation of Event Detection
Here we recall the original optical flow reconstruction representation (1) for each video frame sequence q.
Our main assumption is that epileptic events are conformant to each other and therefore a suitable distance measure can cluster them apart from the non-epileptic false positive detections. Fundamental models of the epileptic neurological condition suggesting that seizures may be states representing limit cycle dynamics [
11] back this working hypothesis.
First, we define the channel-average (we remind that channels are the group parameter rates) deviation or energy of the optical flow for each
q-sequence as:
Next, we take for each detected event
those frame sequences that are 20 windows (frame sequences) before and after the event. For the so obtained set of signals
collected around each detected event, we define the distance between each pair as:
According to (10) all the distances are bounded between 0 and 1. The metric is therefore non-quadratic and allows for an easier parametrization of the clustering criterion.
Foe each event the mean distance to all the other recorded events is:
The clustering of events defines seizure event category:
Here is a threshold parameter. We use here the convenient value for the threshold ε = 0.5. This value corresponds to the average distance between each event and the rest in case of randomly selected sequences. Another, more conservative choice is ε = 0.45 prevents classifying some false-positive detections as true ones.
4. Discussion
4.1. Features of the Proposed Method
Our concept is different from the “standard” machine-learning paradigm in that it accumulates its training set during the normal, ongoing operation of the system. There is no separation between the learning phase and the performance phase. When the system collects a sufficient number of events (we have selected a minimum of 10 in the current trial), the adaptation algorithm activates autonomously or by operator′s intervention. Our scheme thus avoids the need for large, validated data sets in order to train the detection algorithm in advance. Collecting such data, especially in cases of infrequent epileptic seizures, may require large time intervals before the system is ready to operate in real time. Another advantage is that in non-stationary situations, the adaptation can be perpetual and not restricted to an initial training. The autonomous classification can also restrict the number of previous events considered for the adaptation procedure. Events that served as a training set earlier may not be relevant later if the conditions (medication, environment) have changed.
In our unsupervised validation approach, we use a multivariate description of the data as opposed to the feature-based detection algorithm employed for the real-time detection. In this way, the total amount of accumulated data drives the adaptive process. The more events in the training set; the better is the classification accuracy. Although the clustering technique is dependent of the events generated by the detection algorithm, the clustering quantifier described in
Section 2.9, the mean distance to the rest of the events, is not related or dependent in any way on the processing parameters. Neither the frequency range nor the detection threshold can directly influence the result of the clustering. Therefore, it is possible to apply the automated labeling of the events on any subset of events regardless of the parameters used to detect them.
4.2. Comments on the Results
The specificity of the detection algorithm, the ration between the true detections and the total number of detections, improves with the introduction of parameter adaptation. In the “rigid” mode, this feature was 70% while after introducing the on-going learning procedure it went above 83%. To compare with previous results in [
3,
5] where the detection was of a “rigid” type with only fixed parameters, we note that the reported false positive rates of 0.7–1.0 per night are higher than in our trial. In [
5] the specificity is according to the sensitivity level, for 100% sensitivity the averaged specificity is at 80%. It is difficult however to compare the approaches as we have very limited amount of data. Another factor is the source of false alarms. In most other works they are reportedly due to behavioral events while in our case majority false positives are actually due to intermittent motor paroxysms during sleep that are not tonic-clonic seizures.
Concerning the results from
Table 1, the automated, unsupervised retrospective validation, we stress that the misclassifications of false negative type are not harmful for the adaptive concept proposed here. Unlike the missed detections during the online operation that may compromise patient′s safety, the missed true seizure events during the validation can only decrease the amount of events used for optimization of the detection parameters but in general will not alter essentially the outcome of the adaptive algorithm. On the contrary, FP type of misclassification may not be benign as it may introduce “alien” events into the training set and potentially steer the adaptive procedure away from optimal set of parameters. For the same reason we would favor test events being classified as false positives rather than as true TC seizures. We also decided to present that the outcomes of our second trial separately. Firstly, because it was the first trial that ran in fully automated adaptive mode, secondly and as stated in the previous sub-section, our scheme accounts for potential non-stationary situations. We limit the size of the accumulated training sets to a selectable number of previous events.
4.3. Limitations
Using “on the move” data to adjust detection parameters provides a dynamic reinforced learning scheme. However, it also contains a dependence on the initial state of the parameters. This creates the questions of convergence and stability, on the one hand, and missing the optimal settings on the other. So far, we have observed convergence and performance improvement, but in the future we will pursue more general evidence.
Another limitation of the proposed scheme is the issue of false negatives, or missed true seizure events. Our current methodology relies only on detected events, thereby leaving possible undetected events out of the validation process. This is a generic challenge, as its solution may require scanning continuous video data by qualified observers. One of the obstacles in achieving a complete scan of the events is the necessity to continuously record large amounts of data. It is, however, clear that no method will ever give a 100% sensitivity. Even thorough visual inspection of long-time continuous video records by qualified observers cannot guarantee the absence of missed seizure events.
For the automated classification, particularly dubious are those events where care personnel deliberately imitate convulsive seizures in order to test one or another alerting device. The automated classification algorithm will classify these “faked” seizures as either real ones or false positives depending on how realistic the imitation has been. In this respect, the visual operator-based validation will always be more accurate and therefore unequivocally serves as “ground truth”.
Our approach essentially relies on the assumption that only the seizure events form a cluster. If other habitual movements are frequently present in the optical flow data, this may alter the clustering outcome. A more advanced multi-cluster analysis can address those cases.
We have addressed here the major motor events that can have adverse effects for the patient. More subtle clinical events may go beyond the sensitivity range of pure video observation and require other sensors. This issue may certainly be relevant for diagnostic purposes. It remains, however, an open question whether all sorts of epileptic seizures can and should be detected and alerted for care purposes.
4.4. Future Research
One possibility for future research is to study the dependence of the adaptive algorithm on the initial state using data from multiple patients to derive an optimal set of parameters for the group. In such a scenario, group optimization will provide the initial setting of the personalization process and will improve with the inclusion of new patients. A secondary adaptive process will then take place on a personal level but starting from an already optimized initial state. We are now expanding the testing of the detection concept and we will be able to apply these ideas. To acquire more data from multiple subjects, we will use our pre-surgical observation facility where patients who are possible candidates for surgical treatment undergo video-EEG diagnostics while on controlled decrease of medication levels. The typical admission duration is between seven and 14 days, which is short for applying any adaptive algorithms. It is, however, possible to derive a group-optimal parameter set for later use in the residential facility as an initial state of the alerting system.
We note also that the concept of remote detection of adverse events can go beyond the application of convulsive epileptic seizures. We have shown earlier that certain dynamic features of the seizure process can be predictive for post-ictal suppression of brain activity [
12]. Falls and non-obstructive respiratory apnea can also be detected from video sequence analysis [
13,
14]. Upon proper testing and validation, the extended detection modules can be attached to the existing software and provide an enhanced automated “situation awareness” tool.
Addressing the challenge of missed epileptic events, the automated off-line clustering approach is applicable to larger sets of retrospective data. This will produce a list of suspected events that the on-line detection cycle may have missed. We are currently investigating this technique and designing a recovery module that, if proven reliable, will be integrated into the existing system.
Our approach is a “hybrid” one that combines model-based feature extraction from optical flow data with machine learning adaptive techniques. This gives several advantages when compared to pure machine learning concepts where large data sets must be prepared and scored in advance [
15]. Extracting interpretable features does not only reduce the computational complexity of the classification task but also may provide valuable data for the diagnostics of the disease.
5. Conclusions
In this short communication, we presented a novel methodology for automated, remote sensor detection of convulsive epileptic seizures. The base algorithm uses optical flow reconstruction followed by wavelet frequency filtering and detection criterion. The system works in real time, a USB day/night camera feeds the data, and after processing the alerts are sent to a nursing alerting system vie USB-controlled dry contacts.
In addition to the rigid video sequence processing, an adaptive parameter optimization algorithm improves the performance of the detector. The procedure collects data during operation and does not require pre-recorded training set. This part of the system can use either event classification by visual inspection of the video records, or an automated classifier based on retrospective clustering analysis. In either case, a list with true alerts provides data for the machine learning modules.
Besides real motor seizures and false positive detections, test events (typically deliberate seizure imitations by the personnel aimed at testing one or another detection device) were present in the trial as well. Operator-based validation excluded them from the training set. In the automated off-line detection, they can fall into either false or true epileptic events, depending on how close the testing procedure can imitate a real motor seizure.
Our final conclusion concerns the significance of the fully automated adaptive learning functionality proposed here. On one hand, it relieves the personnel of time-consuming validation procedures. In extra-mural applications where a qualified observer is not available, this may even be the only possible scenario for personalizing the system performance. Perhaps an even more important argument in favor of fully automated operation is that human-based visual inspection of video records may compromise a patient’s privacy. The automated classification introduced in this communication avoids this problem, as it does not require a recording of the raw video. It uses only the traces of the reconstructed group-velocities optical flow, which cannot reveal a patient’s identity.