Speech Intelligibility Prediction using Spectro-Temporal Modulation Analysis

A. Spectro-Temporal Modulation Analysis

To establish a foundation for the proposed algorithm, we briefly discuss an STMA scheme proposed in [35], [36] that has shown promising results in ASR [35], [36] and SIP [23]. Figure 1 shows the STMA scheme introduced in [36] which comprises two stages of processing: auditory-spectral analysis, followed by modulation analysis. The two stages are discussed in the following subsections.

B. Spectro-Temporal Modulation Index (STMI)

STMI is an intrusive SIP algorithm introduced in [29], [30] based on STMA of the clean and degraded/processed input speech samples. STMI uses a biologically inspired STMA scheme [30], [38], [39] to produce multi-resolution spectro-temporal decompositions ${\tilde{X}}_M$ and ${\tilde{Y}}_M$ for the clean and degraded signals, respectively. While conceptually similar to the STMA scheme reviewed in Section II-A, the implementation of the spectro-temporal modulation filters used in STMI comprises more sophisticated biologically-inspired processing steps (e.g., using a lateral inhibitory network before modulation filtering). We emphasize this difference by denoting the resulting modulation filtered spectrograms as ${\tilde{X}}_M[f,n;s,r]$ and ${\tilde{Y}}_M[f,n;s,r]$. For a detailed description of STMI’s STMA scheme, we refer the reader to [29], [30]. We emphasize that both STMA schemes produce a 4-D spectro-temporal decomposition of the input speech signal.

Next, the absolute values of the filtered spectrograms are summed along the frequency axis in order to capture the temporal variation of energy in each spectro-temporal modulation channel:

STMI then uses NCC to compare $\overline{X}$ and $\overline{Y}$ within each modulation channel. An intermediate intelligibility measure is defined as:

where the inner product and the induced norm are defined as:

and ${\mu }_{\overline{X}}$ is defined as:

where K is the total number of time frames. Note that the channel indexes s and r are dropped in Equations 7 to 9 for simplicity. Finally, an overall speech intelligibility estimate is defined as the average over all intermediate intelligibility measures:

where Z denotes the total number of spectro-temporal modulation channels.

A. Intermediate Intelligibility Measure

Here, we propose to modify the intermediate intelligibility measure in Eq. (6). This modification is motivated by the fact that at all levels of the human auditory system, the speech signal is represented tonotopically, i.e., distinct acoustic frequencies are analyzed separately [42]. Hence, for each filtered spectrogram and each acoustic frequency bin, the reference and degraded signals are compared using NCC:

Here, in contrast to Eq. (6), the acoustic frequency bins are compared separately. Next, a new intermediate intelligibility measure ρ[s_i, r_j] is defined as:

where F_M is the total number of Mel frequency bins.

Comparing speech spectrograms using NCC was beneficial for SIP in (e)STOI [17], [21]. However, (e)STOI compares the clean and degraded/processed speech spectrograms over short-time segments. In particular, it was shown [21] [17] that lengthening the segment in (e)STOI to above 384 ms decreases SIP performance, especially in the presence of non-stationary maskers. In contrast to (e)STOI, in the proposed approach, speech signals are not segmented into short-time windows, but, instead, integration is performed across the full duration of the speech signals in question (Eq. (11)). This “full duration” analysis was completed in the present experiment as a first investigation of the utility of the proposed algorithm. Future work will need to explore if a segmental implementation provides advantages to SIP beyond that implemented here, and under what conditions.

B. Intelligibility Estimation

To enable unequal contribution of distinct modulation frequencies to speech perception we propose to combine the set of intermediate intelligibility measures ρ[s_i, r_j] using regression. In this study, a linear model is used to estimate the intelligibility of the degraded signal as a linear combination of the intermediate intelligibility measures:

where $w\left[s_i,r_j\right]\in ℝ$ denotes the weights and $b\in ℝ$ is the intercept. Below, we show how the weights can be computed using least squares with L₁ regularization. Using a linear model offers some advantages over more complex models. Besides model simplicity, the number of parameters can be controlled via sparsification in order to reduce the risk of over-fitting. Sparse linear models are easier to interpret and do not require a massive amount of data for training.

Eq. (13) can be written in a matrix form as:

where $W\in {ℝ}^{SR}$ and $\mathrm{\Phi }\in {ℝ}^{SR}$ are constructed by stacking the weights w[s_i, r_j] and intermediate measures ρ[s_i, r_j] into a column vector, respectively.

A. Spectro-Temporal Modulation Analysis

Tables I and II summarize the STMA parameters used in this study. To ensure that a sufficient range of high spectral modulation frequencies are covered, the calculation of the Mel-spectrogram consists of F_M = 130 frequency channels. The values of spectral and temporal modulation center frequencies are selected following the suggestions in [35] and [36]. Since a Mel-scaled frequency axis is used, the spectral modulation frequencies are specified in cycles per kMel (cyc/kMel). The selection of spectral and temporal modulation center frequencies ensures that adjacent filters exhibit a constant overlap in the modulation frequency domain. For the DC modulation filters, we adopt the suggestions in [36] and set $b_{max}=b_s^m=3\times F_M=390$ channels for s_i = 0 and $b_{max}=b_t^m=40$ time-frames for r_j = 0. For a more detailed description of the modulation filter bank, we refer the reader to [35], [36].

B. Parameter Optimization

We optimize W and b in Eq. (14) by minimizing the root mean square error (RMSE) between the wSTMI scores and the subjective intelligibility scores of a training data set. The minimization is augmented with a Lasso [43] sparsification constraint.

Consider a training dataset comprising L clean and degraded speech signal pairs and a subjective intelligibility score for each degraded signal. Let ${\mathrm{\Phi }}_i\in {ℝ}^{SR}$ and 0 ≤ I_i ≤ 1, 1 ≤ i ≤ L denote the intermediate intelligibility vector and the subjective intelligibility score associated with the i-th signal, respectively. The optimization problem can be expressed as:

where λ ≥ 0 is the regularization parameter, and ∥W∥₁ is the L₁ norm of W [43]. Increasing λ increases our preference for a sparse model with fewer non-zero weights. Sparse linear models are easier to interpret and can generalize more accurately to unseen data [43]. Sparsification also reduces the number of selected modulation channels and hence the computational complexity of the SIP algorithm. Here, an algorithm based on cyclical coordinate descent is used to optimize W and b. For a detailed description of the algorithm, we refer the reader to [44]. We emphasize that the computational complexity of the optimization is of little concern, since it is performed once, and W and b are fixed afterward.

To optimize W and b, speech stimuli and the subjective intelligibility scores of the ITFS-Kjems and NELE-Taal datasets (described in Section V) are used for training and validation, respectively. The generalizability of the trained model is investigated by evaluating its performance over unseen data in Section VI. Figure 4 shows the RMSE and Pearson correlation between the model predictions and the subjective scores of the training and validation datasets as a function of λ. It is interesting to note that the RMSE curves for the training and validation datasets are almost flat for a wide range of λ, indicating that reducing the number of channels used has little impact on RMSE. For large values of λ, the performance over the validation set is better than the performance over the training set. We attribute this phenomenon to the fact that, unlike the common practice of partitioning one data set into a training and a validation set, two different datasets were used with one for training and the other for validation, resulting in drawing the training and validation samples from two different distributions.

We select the regularization parameter λ* = 0.065 to minimize the RMSE over the validation dataset. Figure 5 shows the selected set of weights W associated with λ*. The horizontal and vertical axes indicate the temporal and spectral modulation center frequencies, respectively. The weights associated with the selected spectro-temporal modulation channels are also shown. Figure 5 shows that only 8 out of the given 55 spectro-temporal modulation channels are selected for SIP. The optimized weights identify the importance of spectral modulation frequencies between 0.72 and 4.54 cyc/kMel and temporal modulation frequencies below 10 Hz.

A. ITFS-Kjems

This dataset consists of speech subjected to ideal time-frequency segregation (ITFS) [45] processing. 150 Sentences from the Dantale II corpus [46] were degraded by four types of noise: unmodulated SSN, cafeteria noise, car interior noise, and noise from a bottling factory [47]. Noisy sentences were processed using an ideal binary mask (IBM) or a target binary mask (TBM). Noises were presented at three different SNRs: at 20% speech reception threshold (SRT), 50% SRT, and −60 dB SNR. Each binary mask was created using eight different relative-power criteria (RC). This resulted in a total of 7 (mask and noise type combinations) x 3 (SNRs) x 8 (RCs) = 168 conditions. N=15 subjects participated in the listening test. This dataset was used as the training set to optimize the parameters of wSTMI.

B. ModN-Jensen

This dataset consists of speech degraded by modulated noise [17]. Dantale II sentences were degraded by ten types of modulated noise, each presented at six SNRs selected to cover the full range of performance across the different noise types. Four of the maskers were selected from the ICRA noise corpus [48]: unmodulated SSN and 1/2/6-person babble. The ICRA signals are synthetic signals with spectral and temporal modulation properties similar to speech. Machine gun noise and destroyer operation room noise were selected from the Noisex corpus [49]. The rest of the maskers were sinusoidally amplitude-modulated SSN at 2, 4, 8, and 16 Hz. This resulted in a total of 10 (noise types) x 6 (SNRs) = 60 conditions. N=12 subjects participated in the experiment.

C. ModN-Fogerty

This dataset consists of speech degraded by modulated noise. In [50], IEEE sentences [51] spoken by a male talker [52] were degraded by six modulated maskers: unmodulated SSN and single-talker modulated SSN that was either time-compressed or expanded. Pitch-synchronous overlap-add time compression/expansion was employed to modify the modulation spectrum of the masker to run at 25%, 50%, 100%, 200%, and 400% of the duration; thereby, increasing or decreasing the rate of noise modulation. Maskers were presented at −7 dB SNR. This resulted in a total of 1 (unmodulated SSN) + 5 (modulated SSN) = 6 conditions. N=15 normal hearing subjects participated in the study.

D. ModN-Gibbs

In [53], IEEE sentences were corrupted by four types of noise: unmodulated SSN and three types of time-compressed/expanded speech-modulated noise. Unmodulated SSN and single-talker speech-modulated SSNs were created similar to those in Section V-C, and the noise modulation was time-compressed/expanded using pitch-synchronous overlap-add to run at 25%, 100%, or 400% of the original duration. Noises were presented at three SNR levels: −8, −4, and 0 dB. This resulted in a total of 4 (noise types) x 3 (SNRs) = 12 degradation conditions. Stimuli were presented to 5 listeners.

E. ModFN-Fogerty

This dataset [54] consists of temporally filtered speech, degraded by unmodulated SSN or speech-modulated noise. IEEE sentences, spoken by a male talker, and speech-modulated SSN were temporally filtered to retain modulations in either the low-pass (0–8 Hz), high-pass (8–16 Hz), or combined range. Moreover, the noise envelope was compressed (to 50% depth), left alone (100% depth), or expanded (to 200% depth) using an exponential function of the instantaneous envelope amplitude (as in [55]), and noises were presented at two different SNRs (0 dB and −2 dB). This resulted in a total of 16 (speech + noise configurations)×2 (SNRs)=32 conditions. N=20 young normal hearing subjects participated in the listening test.

F. NR-Jensen

This dataset consists of speech processed by non-linear single-microphone noise reduction (NR) algorithms. In [56], the Dutch version of the noisy Hagerman sentences [57], [58] were degraded by unmodulated SSN at −8, −6, −4, −2, and 0 dB SNR. The degraded speech was processed by three NR algorithms aimed at finding binary or soft minimum MSE estimates of the short-time spectral amplitude. This resulted in a total of 4 (3 NR algorithms + 1 unprocessed) x 5 (SNRs) = 20 conditions. N=13 subjects participated in the listening test.

G. NR-Hu

In [52], IEEE sentences and isolated consonants were degraded by four types of noise: babble, car, street, and train at 0 and 5 dB SNRs. Degraded signals were then processed by eight non-linear NR algorithms, including spectral subtraction, sub-space, statistical model-based, and Wiener-type algorithms. This resulted in a total of 4 (noise types) x 9 (1 unprocessed + 8 algorithms) x 2 (SNRs) = 72 conditions. N=40 subjects participated in the listening test. This is the only dataset in this study that provides sentence instead of word recognition scores.

H. Reverberation datasets

In [59], HINT [60] and IEEE sentences were convolved with room impulse responses simulated using the image method. The sentences were processed subject to four reverberation times: T60 = 0.9, 1.2, 1.5, and 2.1 seconds, and three direct-to-reverberant ratios (DRR): 0, −10, −20 dB. This resulted in a total of 4 (T60) x 3 (DRR) = 12 reverberation conditions. The stimuli were presented to N=15 normal-hearing subjects. This is the only dataset explored here that did not use noise as a type of speech degradation.

I. NELE-Taal

In [61], sentences from the Dutch Matrix-test [57] were processed by two near-end listening enhancement (NELE) algorithms based on linear [62] and non-linear [61] approximations of the SII. The signals were then degraded by unmodulated SSN and babble noise at three different SNRs: [−20, −17, −14] dB and [−11 −8 −5] dB for the processed and unprocessed speech, respectively. The SNRs were chosen to produce roughly similar intelligibility for the processed and unprocessed conditions. This resulted in a total of 2 (noise types) x 3 (1 unprocessed + 2 algorithms) x 3 (SNRs) = 18 conditions. N=16 subjects participated in the listening test. This dataset was used as the validation set in optimizing the parameters of wSTMI.

J. NELE-Cooke

In [63], IEEE sentences were processed by 19 NELE algorithms and degraded by two types of noise. Unmodulated SSN and competing speaker (CS) noise were added to the processed speech at [1, −4, −9] dB and [−7, −14, −21] dB SNR, respectively. In [24], a subset of the dataset comprising 10 of the IEEE sentences for each degradation condition and nine of the NELE algorithms was used for SIP assessment. This resulted in a total of 2 (noise types) x 10 (1 unprocessed + 9 processed) x 3 (SNRs) = 60 conditions for our assessment.

K. NELE-Chermaz

In [64], a recording of the IEEE sentences [65] were processed by three NELE algorithms: SSDRC [66], AdaptDRC [67], [68], and AdaptDRC + OE [69]. The algorithms were chosen based on their performances in the Hurricane challenge [63]. The processed sentences were degraded in two realistic acoustic environments. The first environment was a small space with a short reverberation time (T60 = 300 ms) and CS noise. The second environment was a wide space with a long reverberation time (T60 = 1250 ms) and unmodulated SSN. Additive noises were presented at [−12.6, −7.6, −2.6] and [−1.6, 1.8, 5.6] dB SNRs for the small and large environments, respectively. This resulted in a total of 2 (environments) x 4 (1 unprocessed + 3 NELE algorithms) x 3 (SNRs) = 24 degradation conditions. Stimuli were presented binaurally to N=34 listeners. In our work, speech signals associated with the left channel (the better ear) were used for monaural SIP.

L. Int-Miller

In contrast to the previous datasets of degraded continuous speech, this dataset [70] presented temporally interrupted segments of clean speech alternating with noise. Interruption intervals for IEEE sentences were defined by calculating the running SNR between the target speech and a single-talker speech-modulated SSN at an average SNR of 0 dB. Two methods of interruption were employed based on the running SNR: positive (high-intensity speech) and negative (low-intensity speech) local SNR intervals which defined temporal intervals of speech either exceeding, or exceeded by, the competing noise level, respectively. The detected intervals were deleted and filled with one of the following noise types: (1) unmodulated SSN, or speech-modulated SSN based on (2) the missing portion of the speech, (3) the preceding speech, (4) a random segment of a different sentence, and (5) a time-compressed version of (4). The noise replacements were presented at two levels: according to the level of the replaced segment or according to the overall level of the sentence. This resulted in a total of 2 (interruption methods) x 5 (noise types) x 2 (noise levels) = 20 conditions. Stimuli were presented to 11 normal-hearing listeners.

A. Figures of Merit

SIP algorithms are conventionally evaluated using the Pearson and Spearman correlation coefficients or variants. The reason is that the subjective intelligibility score scale generally depends on listening test parameters not known to the algorithm, such as the corpus, subject response protocol, and scoring method, e.g., word scoring versus sentence scoring. Thus, a SIP algorithm can be made more widely applicable to different listening test protocols and scoring scales by requiring the algorithm’s output (“objective” scores) to track the trend of the subjective scores instead of the latter’s absolute values. To reconcile the objective score and the subjective score scales, a common practice is to use a monotonic mapping with few parameters, such as a sigmoid or a third-degree polynomial, to map the objective scores to the subjective scale. In [21], a logistic mapping is used to map model outputs to subjective score estimates:

where $\tilde{I}$ is the SIP algorithm output, and c < 0 and d are constants whose values are chosen with least square fitting. The parameters of the logistic mapping are calculated separately for each dataset and each algorithm.

We use five figures of merit to evaluate the performance of SIP algorithms: i) the Pearson correlation coefficient between the subjective I and objective $\tilde{I}$ intelligibility scores, ii) the Pearson correlation coefficient between the subjective I and logistic mapped $\overline{I}$ intelligibility scores, iii) the Spearman rank correlation coefficient, iv) the RMSE between I and $\tilde{I}$, and v) the RMSE between intelligibility score I and the logistic mapped predicted intelligibility $\overline{I}$.

B. Performance Evaluation

We compare the performance of wSTMI to the other SIP algorithms listed in Table III. Included are the best-performing SIP algorithms known to-date. Different from other algorithms, SIIB needs to input speech samples with at least 20 s length. To meet this requirement, SIIB-1 uses sample repetition, and SIIB-2 entails concatenation of several distinct short samples.

Figure 6 shows scatter plots for each dataset and wSTMI’s output before applying a logistic mapping. Each point in the plot represents a different condition in the dataset. Tables IV and V show the Pearson correlation before and after applying the logistic mapping in Equation 16, respectively. To determine statistically significant differences between correlation values (Tables IV and V), pairwise comparisons using the Williams’s test [72] were performed for each dataset between the best performing SIP method and the others. Methods which did not perform significantly worse than the best performing algorithm (p < 0.05) are marked with (*) in Table V. Table VI displays the Spearman rank correlation. Tables VII and VIII show the RMSE before and after applying the logistic mapping in Equation 16, respectively.

Not surprisingly, wSTMI shows excellent performance for the training and validation datasets, for which the model parameters W and b were optimized (ITFS-Kjems and NELE-Taal). However, importantly, wSTMI also performs well for datasets not used in the training phase. Also interesting to note that wSTMI performs well with strongly modulated noise maskers, despite the fact that such signals were not present in the training dataset. The top performance of wSTMI is closely followed by eSTOI, SIIB-2, and OSTMI. Focusing on the modulated noise conditions, i.e., ModN-Jensen, ModN-Fogerty, ModN-Gibbs, and ModFN-Fogerty, it is clear that wSTMI outperforms existing SIP methods.

STMI, OSTMI, and wSTMI all compare the spectro-temporal modulation envelopes of the clean and degraded speech signals to estimate the intelligibility of the latter. STMI integrates the filtered spectrograms across the acoustic frequency axis in order to capture the temporal fluctuation of energy in each modulation channel, while OSTMI and wSTMI avoid this integration by making individual comparisons between acoustic frequency bins within each modulation channel. Moreover, while STMI assigns equal weights to different spectro-temporal modulation frequencies, OSTMI uses a heuristic feature-selection scheme which assigns larger weights to intermediate spectral modulation frequencies. wSTMI further improves the performance of OSTMI by optimizing the weights assigned to different spectro-temporal modulation channels to maximize the SIP performance across a training dataset. The top SIP performance of wSTMI signifies the importance of taking into account different contributions of distinct spectro-temporal modulation channels to speech intelligibility.

Recall that STOI, eSTOI, OSTMI, and wSTMI estimate the intelligibility by comparing the modulation envelopes of the clean and degraded input signals. While STOI compares the temporal modulation envelopes of the clean and degraded signals, the other three algorithms use the joint spectro-temporal modulation envelopes. It is interesting to note that eSTOI, OSTMI, and wSTMI outperform STOI in the presence of modulated noise (ModN-Jensen, ModN-Fogerty, ModN-Gibbs, NELE-Cooke, and NELE-Chermaz). This suggests the significance of spectral modulation analysis in modulated noise.

TFSS uses a Hilbert-derived temporal fine structure (TFS) waveform for SIP. The TFS waveform in [20] is the phase-modulated carrier of the Hilbert envelope. This contrasts with SIP methods that employ primarily envelope information, such as STOI, eSTOI, and wSTMI. Though TFSS was tested only on the NR-Hu dataset in [20], TFSS performs quite well for a wide range of distortions, including modulated noise and reverberation. The decent performance of TFSS for modulated noise conditions may be explained by the significant contribution of TFS information to speech perception in the presence of modulated noise [18], [73], [74].

We note that HASPI is computed as a linear combination of cepstral correlation and auditory coherence terms (low, mid, and high-level coherence). Here, we used a fixed set of parameters proposed in [28] to evaluate the performance of HASPI. In [24], the parameters of HASPI were optimized for each dataset to maximize performance. By comparing the performance of HASPI with and without parameter tuning, a few observations can be made. Even though HASPI achieves top SIP performance in many degradation conditions in both setups, it performs less well for NR-Hu and ModN-Jensen when used without parameter tuning. Moreover, when used without adaptation, HASPI performs poorly for NELE-Taal while delivering high performance for similar distortions in NELE-Cooke and NELE-Chermaz. These observations indicate that a careful tuning of HASPI’s parameters is crucial for achieving robust SIP performance.

Many SIP algorithms performed poorly on ModN-Fogerty. Recall that ModN-Fogerty consists of speech degraded by time-compressed/expanded speech shaped modulated noise presented at a fixed SNR. Therefore, the time-compression/expansion rate of noise is the only factor governing the intelligibility in this dataset. None of the SIP methods investigated in this study were optimized for such distortions. STOI and SI-SDR show negative correlation with intelligibility across this dataset. In general, we desire a SIP algorithm whose outputs show strong positive correlation across many types of speech distortions.

Focusing on the interrupted speech conditions, i.e., Int-Miller dataset, it is evident that wSTMI, CSII-high, SI-SDR, and HASPI outperform other SIP methods. Interestingly, OSTMI, SIIB, CSII-I3, and TFSS are negatively correlated with intelligibility for this dataset. Figure 7 shows the scatter plots for each algorithm and the Int-Miller dataset before applying a logistic mapping. Recall that Int-Miller consisted of temporally interrupted speech. Also, the +SNR conditions (high-intensity speech) were more intelligible than their −SNR (low-intensity speech) counterparts. As the scatter plots in Figure 7 imply, many SIP algorithms predicted lower intelligibility for the +SNR conditions, resulting in a negative correlation. In addition, even though CSII-high and SI-SDR achieved high overall correlation for this dataset, they might not be suitable for this type of degradation. The scatter plots show that the predictions within each cluster are flat or negatively correlated with intelligibility for CSII-high and SI-SDR.

A. Comparison to MTFs of Human Auditory System

The optimization in Section IV-B assigns larger weights to the modulation channels that are important for SIP. As MTFs represent the relative importance of the spectro-temporal modulation frequencies for human auditory perception, we expect to observe similarities between the MTFs of the human auditory system and the optimized weights W.

In [29] and [30], spectro-temporal MTFs were estimated by measuring the detection thresholds of different modulation frequencies for the human auditory system. The MTFs in [30] show a lowpass behavior in both the spectral and temporal modulation directions with cut-off frequencies of about 2 cyc/octave and 16 Hz, respectively. Moreover, the MTF is relatively flat in the bandpass region. In [34], MTFs were measured by evaluating the relative importance of the modulation frequencies for speech perception (in contrast to discrimination thresholds in [30]). A modulation filtering approach was used to restrict the modulation patterns in the speech signal and a joint spectro-temporal MTF was derived. In [40], the envelope modulation fidelity was calculated using the normalized cross-covariance of the degraded and reference signal envelopes. Even though Kates et al. used temporal modulation frequencies up to 325 Hz, the results indicated that the low temporal modulation frequencies provided the highest amount of information for speech intelligibility. The findings in [29], [30], [34], [40] are consistent with the optimized W depicted in Figure 5, but with one key difference. While the cut-off spectral and temporal modulation frequencies agree, Elliot et al. [34] found that the lowest spectral modulation frequencies (near DC) are important for speech perception. In contrast, our optimization method tends to favor the intermediate spectral modulation frequencies.

From Figure 8 it follows that for larger values of λ (larger sparsity), the optimization scheme favours a lowpass behaviour with respect to the temporal modulations. Specifically, temporal modulation frequencies below 10 Hz tend to be selected. This is in slight contrast to previous studies of the relative importance of modulation frequencies for speech perception. For example, Drullman et al. [13] examined the effect of varying the amount of temporal modulation frequencies available to human subjects for speech recognition and found that they needed temporal modulation frequencies up to 16 Hz. The weights selected for wSTMI are an outcome of predicting speech intelligibility using a referenced-based STMA framework. One may argue that the task of SIP is different than that of speech recognition, and hence, one might not expect the important spectro-temporal modulations for the two tasks to be identical.

B. Comparison to Automatic Speech Recognition

In (Schadler et al. [35] and [36]), the relative importance of different spectro-temporal modulation frequencies for automatic speech recognition (ASR) was determined using the STMA front-end (Section II-A) and hidden Markov models with Gaussian mixture emissions as the back-end. Schadler et al. evaluated the performance sensitivity of their ASR algorithm, when features from a single spectro-temporal modulation channel were dropped. The results showed that the intermediate spectral modulation frequencies (between 1.34 and 2.8 cyc/kMel) and low temporal modulation frequencies were more important for ASR. This is consistent with our findings and implies that the relatively important spectral modulation frequencies exhibit a bandpass behavior.

C. Selected Spectral Modulation Frequencies

We present a filtering example to help further understanding the role of spectral modulation analysis in SIP. Figure 9 shows a series of spectral modulation filtered speech spectrograms for the utterance “should we chase?” spoken by a male (no temporal modulation filtering is employed). We can see that the filtered spectrograms with intermediate spectral modulation frequencies between 0.72 and 4.54 cyc/kMel best preserve the clarity of the formants and their transition patterns. A similar observation was made for several other speech signals. The importance of formant transitions in the perception of natural speech has been shown in several studies, including [75]–[77]. Thus, based on the above observation, we argue that the intermediate spectral modulation frequencies are crucial for SIP.