Introduction

The information in sound is based on variations in the air pressure over time, which for many sound sources can be modelled as a superposition of sine wave oscillations of different frequencies. To capture this information by auditory perception or signal processing, the sound signal has to be processed over some non-infinitesimal amount of time and in the case of a spectral analysis also over some range of frequencies. Such a region over time or over the spectro-temporal domain is referred to as a temporal or spectro-temporal receptive field (Aertsen and Johannesma [1]; Theunissen et al. [2]; Miller et al. [3]; Fritz et al. [4]).

If one considers the theoretical or algorithmic problem of designing an auditory system that is going to analyse the variations in air pressure over time, one may ask what types of auditory operations should be performed on the sound signal. Would any operation be reasonable? Specifically, regarding the notion of receptive fields, what types of temporal or spectro-temporal receptive field profiles would be reasonable? Is it possible to derive a theoretical model of how receptive fields “ought to” respond to auditory signals?

In vision, the corresponding problem of formulating a theoretical model for visual receptive fields (Lindeberg [5]) can be addressed based on a framework developed in the area of computer vision known as scale-space theory (Iijima [6]; Witkin [7]; Koenderink [8]; Koenderink and van Doorn [9, 10]; Lindeberg [11–14]; Sporring et al. [15]; Florack [16]; ter Haar Romeny [17]). A paradigm that has been developed in this field is to impose structural constraints on the first stages of processing that reflect symmetry properties of the environment. Interestingly, it turns out to be possible to substantially reduce the class of permissible image operations from such arguments, and it has been shown that biological receptive fields as measured in the lateral geniculate nucleus (LGN) and the primary visual cortex (V1) of higher mammals (Hubel and Wiesel [18–20]; DeAngelis et al. [21, 22]; Conway and Livingstone [23]; Johnson et al. [24]) can be well modelled by idealized scale-space operations (Young et al. [25, 26]; Lindeberg [5, 27]).

The subject of this article is to show how a corresponding normative theory for receptive fields can be developed for auditory stimuli, and how idealized models of auditory receptive fields can be derived in a principled manner by applying scale-space theory to auditory signals. Our aim is to express auditory operations that are well localized over time and frequencies and which allow for well-founded handling of temporal phenomena that occur at different temporal scales as well as receptive fields that operate over different ranges of frequencies in such a way that operations over different ranges of frequencies can be related in a well-defined manner.

When applied to the definition of spectrograms, alternatively to the formulation of an idealized cochlea model, the scale-space approach can be used for deriving the Gabor (Gabor [28]; Wolfe et al. [29]; Lobo and Loizou [30]; Qiu et al. [31]; Wu et al. [32]) and Gamma-tone (Johannesma [33]; Patterson et al. [34]; Hewitt and Meddis [35, 36]) approaches for computing local windowed Fourier transforms as specific cases of a complex-valued scale-space transform over different frequencies. In addition, the scale-space approach to defining spectrograms leads to a new family of generalized Gamma-tone filters, where the time constants of the individual first-order integrators coupled in cascade are not equal as for regular Gamma-tone filters but instead distributed logarithmically over temporal scales and allowing for different trade-offs in terms of e.g. the frequency selectivity of the spectrogram and the temporal delay of time-causal receptive fields.

When applied to a logarithmic transformation of the spectrogram, as motivated from the desire of handling sound signals of different strength (sound pressure) in an invariant manner and with a logarithmic transformation of the frequencies as motivated by the desire of enabling invariance properties under a frequency shift, such as transposing a musical piece by one octave, we will show how this theory also allows for the formulation of spectro-temporal receptive fields at higher levels in the auditory hierarchy in terms of spectro-temporal derivatives of spectro-temporal smoothing operations as obtained from scale-space theory.

It will be demonstrated how such second-layer receptive fields can be used for computing basic auditory features such as onset detection, partial tone enhancement and formants, and specifically how different types of features can be obtained at different temporal scales τ, spectral scales s and how this theory naturally also leads to a glissando parameter v that represents how logarithmic frequencies ν may vary over time t according to a local linear approximation ν′ = ν + vt.

Compared to the more common approach of computing auditory features in digital signal processing by local windowed fast Fourier transforms (FFT), we argue that the proposed theory provides a way to avoid artifacts of performing the computation in temporal blocks that later have to be combined again. Furthermore, by the built-in covariance properties of the model under temporal shifts, variations in sound pressure, frequency shifts and glissando transformations, the proposed approach allows for provable invariance properties under such transformations of sound signals.

It will also be shown how idealized models of spectro-temporal receptive fields as obtained from the presented theory in terms of spectro-temporal derivatives of spectro-temporal scale-space kernels can be used for generating predictions of auditory receptive fields that are qualitatively similar to biological receptive fields as measured by cell recordings in the inferior colliculus (ICC) and the primary auditory cortex (A1) (Miller et al. [3]; Qiu et al. [31]; Machens et al. [37]; Andoni et al. [38]; Elhilali et al. [39]; Atencio and Schreiner [40]).

Structural requirements on temporal receptive fields

In the following, we will describe a set of structural requirements concerning temporal receptive fields for a general sensory system that processes a scalar time-dependent signal regarding (i) the measurement of sensory data with its close relationship to the notion of temporal scale, (ii) internal derived representations of the signal that are to be computed by a general sensory system, and (iii) the special nature of time in terms of temporal causality and temporal recursivity.

If we regard the sensory signal f as defined on a one-dimensional continuous temporal axis f : ℝ → ℝ, then the problem of defining a set of early sensory operations can be formulated as finding a family of operators 𝓣_τ that are to act on f to produce a family of new intermediate representations of the signal (1) which are also to be defined as functions on ℝ, i.e., L(⋅; τ) : ℝ → ℝ.

(In Equation (1), the symbol “⋅” at the position of the first argument of L is a place holder to emphasize that in this relation, L is regarded as a function and not evaluated with respect to its first argument t. The following semi-colon emphasizes the different natures of the temporal coordinate t and the filter parameter τ.)

The evaluation of one (specific example) function L can be interpreted as the response of a (set of) sensory neurons in biology or as an internal representation in a temporal signal processing system that processes temporal information. Combined with additional processing over frequencies, we will later use such internal representations for modelling neurons in the inferior colliculus (ICC) and the primary auditory cortex (A1).

Specific scale-space axioms for a non-causal temporal domain

Depending on the conditions under which the sensory data is processed, we can consider two types of cases. For pre-recorded signals, we may in principle assume access to the data at all temporal moments simultaneously and thereby apply operations to the signal that would correspond to access to virtual future. For real-time signal processing or when modelling biological perception, there is, however, no way of having access to the future, which imposes fundamental additional structural requirements on a temporal front-end. For pre-recorded temporal signals, we require the following:

Non-enhancement of local extrema.

In the case of a continuous scale parameter, one way of formalizing the requirement of non-creation of new structures in the signal with increasing scale is that local extrema must not be enhanced with increasing scale. In other words, if a point (t₀; τ₀) is a local (spatial) maximum of the mapping t ↦ L(t; τ₀) then the value must not increase with scale. Similarly, if a point (t; τ₀) is a local (spatial) minimum of the mapping t ↦ L(t; τ₀), then the value must not decrease with scale (see fig. 1). Given the above mentioned differentiability property with respect to scale, we say that the multi-scale representation constitutes a scale-space representation if for a scalar scale parameter it satisfies the following conditions (Lindeberg [14, 43]): (22) (23) By considering the response to a constant signal, it follows from the requirement of non-enhancement of local extrema that a scale-space kernel should be normalized to unit L₁-norm, corresponding to the normalization requirement in Equation (9).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Illustration of the notion of non-enhancement of local extrema for a 1-D signal.

The requirement of non-enhancement of local extrema is a way of restricting the class of possible filtering operations by formalizing the notion that new structures in the signal must not be created with increasing scale, by requiring that the value at a local maximum must not increase with scale and that the value at a local minimum must not decrease. (Figure from Lindeberg [27].)

https://doi.org/10.1371/journal.pone.0119032.g001

Scale-space concepts for purely temporal domains

In this section we will describe how the structural requirements listed in the section “Structural requirements on temporal receptive fields” restrict the class of temporal scale-space kernels and thus the class of possible temporal receptive fields.

Non-causal Gaussian temporal scale-space

If, for the purpose of analyzing pre-recorded auditory data, we allow for unlimited freedom of accessing the sensory data at all temporal moments simultaneously, we can apply a similar way of reasoning as has been used for deriving scale-space concepts for image data over a spatial domain (Iijima [6]; Witkin [7]; Koenderink [8]; Lindeberg [11–14, 43]; Sporring et al. [15]; Florack [16]; Weickert et al. [45]; ter Haar Romeny [17]):

Given time-dependent sensory data f : ℝ → ℝ defined over a one-dimensional temporal domain, let us assume that the first stage of sensory processing as represented by the operator 𝓣_τ should satisfy the following structural requirements: (i) linearity, (ii) shift invariance and (iii) obey a semi-group structure over temporal scales τ, where we also have to assume (iv) certain regularity properties of the semi-group 𝓣_τ over scale τ to guarantee sufficient differentiability properties with respect to time t and temporal scales τ. Let us furthermore require (iv) non-enhancement of local extrema to hold for any smooth function f ∈ C^∞(ℝ) ∩ L¹(ℝ) and for any positive scale direction s.

Then, it follows from (Lindeberg [14], theorem 5) that these conditions together imply that the scale-space family L must satisfy a diffusion equation of the form (25) with initial condition L(t; 0) = f(t) for some positive constant Σ₀ and some constant δ₀. Equivalently, this spatio-temporal scale-space representation at scale τ can be obtained by convolution with temporal Gaussian kernels of the form (26) with Σ_τ = τΣ₀ and δ_τ = τ δ₀. Since the parameter Σ₀ only corresponds to an unessential rescaling of the temporal scale parameter τ, we will set Σ₀ = 1.

To verify that the solutions of the diffusion equation obey non-enhancement of local extrema is straightforward. For a one-dimensional signal the first-order derivative is zero, whereas the value of the second derivative will be positive at local minima and negative at local maxima. Hence, for Gaussian smoothing as governed by the diffusion Equation (25), the derivative with respect to scale is guaranteed to be positive at local minima and negative at local maxima (sufficiency). A less immediate result is that non-enhancement of local extrema also implies that the evolution over scale must be governed by the diffusion equation (necessity) and is proved in (Lindeberg [14]).

Graphs of these Gaussian kernels are shown in fig. 2. Notably, these kernels are not strictly time causal. To arbitrary degree of accuracy, however, they can be approximated by truncated time-causal kernels, provided that the time delay δ is chosen sufficiently long in relation to the temporal scale τ. Hence, the choice of δ leads to a trade-off between the computational accuracy of the implementation and the temporal response properties as delimited by a non-zero time delay. This problem, however, arises only for real-time analysis. For off-line computations, the time delay may be set to zero, corresponding to kernels that are mirror symmetric T(−t; s) = T(t; s) through the origin. Thus, the truncated and time-shifted Gaussian kernels can serve as a simplest possible model for a temporal scale-space representation, provided that the requirements of temporal causality and temporal recursivity can be relaxed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The time-shifted Gaussian kernel $g(t;\tau ,\delta )=1/\sqrt{2\pi \tau }\exp (-{(t-\delta )}^2/2\tau )$ for τ = 1 and δ = 4 with its first- and second-order temporal derivatives.

https://doi.org/10.1371/journal.pone.0119032.g002

Derived receptive fields in terms of temporal derivatives.

In addition to the zero-order smoothing kernel T, we have in fig. 2 also shown its first- and second-order temporal derivatives T_t and T_tt. Such derivatives of scale-space kernels do also obey desirable structural properties in terms of linearity, shift invariance and nice properties over scale in terms of non-enhancement of local extrema, with the semi-group property replaced by a cascade property over scale (27) and with the limit case when the temporal scale goes to zero (14) replaced by (28) provided that the corresponding derivative of f exists. Regarding temporal receptive fields that are expressed in terms of derivatives of scale-space kernels, the normalization condition (9) is replaced by the integral of the receptive field being zero (29) In all other major respects, such receptive fields satisfy essential scale-space properties in terms of non-creation of new structures with increasing scale in the sense that local extrema in the receptive field response are not enhanced from a fine to a coarser scale or that the number of local extrema or zero-crossings in the signal is guaranteed to not increase from any fine to any coarser scale.

Additionally, receptive fields that are expressed in terms of temporal derivatives are invariant under additive transformations of the signal (30) and thereby provide a mechanism for capturing local variations in the signal under variabilities of its baseline.

Time-causal temporal scale-space

When constructing a system for real-time processing of sensory data, a fundamental constraint on the temporal smoothing kernels is that they have to be time-causal. The ad hoc solution of using a truncated symmetric filter of finite temporal extent in combination with a temporal delay is not appropriate in a time-critical context. Because of computational and memory efficiency, the computations should furthermore be based on a compact temporal buffer that contains sufficient information for representing sensory information at multiple temporal scales and computing features therefrom. Corresponding requirements are necessary in computational modelling of biological perception.

Time-causal scale-space kernels for pure temporal domain.

Given the requirement on temporal scale-space kernels by non-creation of local extrema over a pure temporal domain, truncated exponential kernels (31) can be shown to constitute the only class of time-causal scale-space kernels over a continuous domain (Lindeberg [42]; Lindeberg and Fagerström [44]). The Laplace transform of such a kernel is given by (32) and coupling K such kernels in cascade leads to a composed filter (33) having a Laplace transform of the form (34) The composed filter has temporal mean and variance (35) (36) In terms of physical models, repeated convolution with such kernels corresponds to coupling a series of first-order integrators with time constants μ_k in cascade: (37) with L(t; 0) = f(t). These temporal smoothing kernels satisfy scale-space properties in the sense that the number of local extrema or the number of zero-crossings in the temporal signal are guaranteed to not increase with the temporal scale. In this respect, these kernels have a desirable and well-founded smoothing property that can be used for defining multi-scale observations over time. A limitation of this type of temporal scale-space representation, however, is that the scale levels are required to be discrete and that the scale-space representation does hence not admit a continuous scale parameter. Computationally, however, the scale-space representation based on truncated exponential kernels can be highly efficient and admits for direct implementation in terms of hardware (or wetware) that emulates first-order integration over time (see fig. 3 for an illustration of a corresponding electric wiring diagram).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Electric wiring diagram consisting of a set of resistors and capacitors that emulate a series of first-order integrators coupled in cascade.

Here, we regard the time-varying voltage f_in as representing the time varying input signal and the resulting output voltage f_out as representing the time varying output signal at a coarser temporal scale. According to the theory of temporal scale-space kernels for one-dimensional signals (Lindeberg [42]; Lindeberg and Fagerström [44]), the corresponding equivalent truncated exponential kernels are the only primitive temporal smoothing kernels that guarantee both temporal causality and non-creation of local extrema (alternatively zero-crossings) with increasing temporal scale. Such first-order temporal integration can be used as a straightforward computational model for temporal processing in biological neurons; see also (Koch [46], chapters 11–12) regarding physical modelling of the information transfer in dendrites of neurons.

https://doi.org/10.1371/journal.pone.0119032.g003

When implementing this temporal scale-space concept, a set of intermediate scale levels has to be distributed between some minimum and maximum scale levels τ_min and τ_max. Assuming that a total number of K scale levels is to be used, it is natural to distribute the temporal scale levels according to a geometric series, corresponding to a uniform distribution in units of effective temporal scale τ_eff = logτ (Lindeberg [47]). Using such a logarithmic distribution of the temporal scale levels, the different levels in the temporal scale-space representation at increasing temporal scales will serve as a logarithmic memory of the past, with qualitative similarity to the mapping of the past onto a logarithmic time axis in the scale-time model by Koenderink [48]. If we have the freedom of choosing τ_min freely, a natural way of parameterizing these temporal scale levels is by using a distribution parameter c > 1 such that (38) which by Equation (36) implies that time constants of the individual first-order integrators will be given by (39) (40) If the temporal signal is on the other hand given at some minimum temporal scale level τ_min, we can instead determine c in (38) such that τ₁ = τ_min (41) and add K − 1 temporal scale levels with μ_k according to (40). Alternatively, if one chooses a uniform distribution of the intermediate temporal scale levels (42) implying (43) then it becomes straightforward to compute the explicit expression for the composed kernel (44) having temporal mean value m_k = k μ and variance τ = k μ². In contrast to the primitive truncated exponentials, which are discontinuous at the origin, these kernels are continuous of order k − 1, allowing for differentiation up to order k − 1. The corresponding expressions for the first- and second-order derivatives are: (45) (46) Fig. 4 shows graphs of these kernels for two combinations of μ and K that correspond to the same value of the composed variance τ = K μ². Notably, these kernels are highly asymmetric for small values of K, whereas they become gradually more symmetric as K increases. Figs. 5–6 show corresponding compositions of truncated exponential kernels for self-similar distributions of the intermediate time constants according to Equations (38), (39) and (40) for $c=\sqrt{2}$ and c = 2^3/4. Comparing fig. 4 and figs. 5–6, the use of a self-similar distribution of the time constants (in figs. 5–6) allows for smoother behaviour near the origin with increasing K while not increasing the temporal delay as much as for the kernels corresponding to a uniform distribution of the intermediate temporal scale levels (in fig. 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Equivalent kernels $h_{composed}(t;\mu )={*}_{k=1}^K{h}_{exp}(t;\mu )$ with temporal variance τ = 1 corresponding to the composition of K truncated exponential kernels with equal time constants μ and their first- and second-order derivatives.

(top row) k = 4 and $\mu =\sqrt{1/4}$. (bottom row) k = 7 and $\mu =\sqrt{1/7}$.

https://doi.org/10.1371/journal.pone.0119032.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Equivalent kernels $h_{composed}(t;\mu )={*}_{k=1}^K{h}_{exp}(t;{\mu }_k)$ with temporal variance τ = 1 corresponding to the composition of K = 4 or K = 7 truncated exponential kernels with different time constants defined from a self-similar distribution of the temporal scale levels according to Equations (38), (39) and (40) and corresponding to a uniform distribution in terms of effective temporal scale τ_eff = log τ for $c=\sqrt{2}$ and with their first- and second-order derivatives.

https://doi.org/10.1371/journal.pone.0119032.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Equivalent kernels $h_{composed}(t;\mu )={*}_{k=1}^K{h}_{exp}(t;{\mu }_k)$ with temporal variance τ = 1 corresponding to the composition of K = 4 or K = 7 truncated exponential kernels with different time constants defined from a self-similar distribution of the temporal scale levels according to Equations (38), (39) and (40) and corresponding to a uniform distribution in terms of effective temporal scale τ_eff = log τ for c = 2^3/4 and with their first- and second-order derivatives.

https://doi.org/10.1371/journal.pone.0119032.g006

Time-recursive computation of temporal derivatives.

Temporal scale-space derivatives of order r can be defined from this scale-space model according to (47) where the Laplace transform of the composed (equivalent) derivative kernel is (48) For this kernel to have a net integration effect (to enable well-posed derivative operators), the total order of differentiation must not exceed the total order of integration. Thereby, r < k is a necessary requirement. The composed transfer function must have finite L₂-norm.

A very useful observation that can be made concerning derivative computations is that temporal derivatives can equivalently be computed from differences between different temporal channels. Let us first assume that all time constants μ_i are different in (48). Then, a partial fraction division gives (49) where (50) showing that each temporal derivative can be computed as a linear combination of the representations at the different time-scales.

More realistically, the channels that we can regard as available at a certain temporal scale with index k will not be the results of direct filtering with different time constants μ_i. Rather, we would like to use the intermediate outputs from the cascade coupled recursive filters H_composed(q; τ_i) for k − r ≤ i ≤ k. Decomposition of $H_{composed}^{(r)}$ into a sum of r such transfer functions (51) shows that the weights B_i are given as the solution of a triangular system of equations provided that the necessary condition r < k is satisfied (52) It can be shown that the Laplace transforms of the equivalent derivative computation kernels satisfy the following recurrence relation (Lindeberg and Fagerström [44]) (53) implying that higher-order temporal derivatives can be computed from small-support finite differences of lower-order derivatives, where the temporal scale-space representations at different temporal scales serve as a sufficient temporal buffer of what has occurred in the past. Derivative computations will therefore be highly efficient. Both the temporal smoothing operation and the computation of temporal derivatives are time-recursive.

Multi-scale spectrograms for auditory signals

The above treatment concerning is general and can be used for modelling desirable properties of temporal receptive fields for a variety of time-dependent sensory signals. For auditory signals, an additional structural requirement arises from the fact that the auditory information is transferred in terms of sound waves that travel from the transmitter to the receiver and the auditory information can be encoded in terms of oscillation frequencies of the air pressure that generates the sensory signal. For this reason and from our knowledge that the variations due to the geometry and other properties of the cochlea leads to physical resonances whose effect can be modelled as a physical Fourier transform, spectrograms are a common tool for analyzing auditory information.

Note that our primary aim is not to specifically model, for example, the measured response of the nerves coming from the cochlea as typically done in previous auditory models (Patterson et al. [36]). Instead we are following the scale-space theory using the principle of invariance as outlined in the sections “Structural requirements on temporal receptive fields” and “Scale-space concepts for purely temporal domains”.

Based on the two models for temporal receptive fields (non-causal in the section “Non-causal Gaussian temporal scale-space” and time-causal in the section “Time-causal temporal scale-space”), we can use the temporal smoothing functions in these two temporal scale-space models as scale-dependent window functions for defining two types of complex-valued multi-scale spectrograms according to (54) (55) where

• g(t; τ) is a temporal Gaussian kernel of the form (26),
• h_composed(t; μ) with μ = (μ₁, …, μ_K) is the equivalent convolution kernel corresponding to a cascade of truncated exponential filters of the form (33).

For a given value of τ, the spectrogram becomes a 2-D function. With the definition extended to all values of τ, the spectrogram based on Gaussian window functions instead becomes a 3-D volume over all temporal extents of the window function or alternatively a set of discrete 2-D slices for the window functions based on truncated exponential functions coupled in cascade for vectors μ = (μ₁, …, μ_K) of different length K.

Note that a priori there may be no principled reason for preferring a particular duration of the temporal window function for the windowed Fourier transform over some other temporal duration. Specifically, different temporal durations may be appropriate for different auditory tasks, such as a preference for a short temporal duration for onset detection and a preference for a longer temporal duration to separate sounds with nearby frequencies. Thereby, the scale-space approach allows for the definition of windowed Fourier transforms for all temporal extents in such a way that any windowed Fourier transform at a coarse temporal scale can be related to a windowed Fourier transform at any finer scale using the cascade property (56) (57) derived from the semi-group structure (15) or the Markov property (18) of the underlying scale-space kernels. Combined with the additional scale-space properties of non-creation of new structures with increasing scale, this guarantees well-founded theoretical properties between corresponding windowed Fourier transforms at different temporal scales.

In most other work on auditory signal processing, there is often an implicit assumption that one chooses a scale for computing the auditory features that seems to work and on which later stage analysis is then based. By the presented formulation of multi-scale spectrograms, we aim at making the consequences of such assumptions explicit, and emphasizing the possibility of computing auditory features at multiple temporal scales as an integrated part of the analysis. Compared to the more traditional approach of computing spectrograms from local fast Fourier transforms combined with local windowing operations, this formulation of multi-scale spectrograms also avoids the concatenation of such windowing operations altogether and thereby the artifacts caused by these.

The scale-space approach for defining multi-scale auditory spectrograms implies that instead of computing a scale-space representation of the original auditory signal, the auditory signal is first projected onto the two orthogonal dimensions cosωt and isinωt of a complex sine wave e^−iωt (58) for which temporal scale-space representations are then defined, implying that the multi-scale spectrogram can be seen as a complex-valued scale-space transform.

Invariance and covariance properties.

Concerning the symmetry requirements of a general temporal sensory front-end described in the section “Structural requirements on temporal receptive fields”, the linearity of the scale-space operations is transferred to a linearity in the complex multi-scale spectrograms (54)–(55). This implies that multiple sources of sound will be combined in an additive manner in terms of their complex-valued responses and that sound sources of different strength (sound pressure) will be handled in a similar manner up to a multiplication of the strength of the signal.

Regarding temporal shift invariance, the magnitude maps ∣S_g∣ and ∣S_h∣ are invariant under a shift of the temporal axis, whereas the phase of the truly complex spectrograms S_g and S_h will be transformed in a predictable manner between similar sound signals at different time moments or from different distances to the observer.

Under a local rescaling of the temporal axis (59) the temporal receptive fields from the Gaussian scale-space model are fully scale covariant and the corresponding complex-valued multi-scale spectrograms are transformed according to (60) If we let the window scale $\sigma =\sqrt{\tau }$ for any angular frequency ω be proportional to the wavelength λ = 2π/ω corresponding to that frequency, then the corresponding spectrograms within the same 2-D slice of the extended 3-D multi-scale spectrogram can therefore be matched by a corresponding frequency shift (61) If the temporal window functions on the other hand do not have the temporal extent proportional to the wavelength, then temporal covariance does not hold within the same 2-D slice but still holds within the 3-D multi-scale spectrogram based on Gaussian window functions because of their self-similarity over scale, whereas the corresponding scaling relations can only be approximate for the truncated exponential functions coupled in cascade, because of the temporal scale levels being restricted to a discrete set of values.

Again there may not be any principled reason for preferring a particular temporal scale over another. The multi-scale nature of these spectrograms makes this aspect explicit and opens up for using different temporal scales for different auditory tasks, where different temporal scales may have complementary advantages.

Relations to Gabor functions

By rewriting the expression (54) for the complex-valued spectrogram based on the Gaussian temporal scale-space concept as (62) it can be seen that up to a phase shift, this multi-scale spectrogram can equivalently be interpreted as the convolution of the original auditory signal f by Gabor functions [28] of the form (63) Such Gabor functions have been previously used for analyzing auditory signals by several authors, including (Wolfe et al. [29]; Kleinschmidt et al. [49, 50]; Lobo and Loizou [30]; Qiu et al. [31]; van de Boogart and Lienhart [51]; Ezzat et al. [52]; Domont et al. [53]; He et al. [54]; Heckmann et al. [55]; Wu et al. [32]; Schädler et al. [56]; Sameh and Lachiri [57]). Our theory provides a new way of deriving this representation with special emphasis on the multi-scale nature of the Gaussian window functions and their resulting cascade properties between spectrograms at different temporal scales.

Relations to Gammatone filters

In the special case when the time constants of all the K truncated exponential filters that are coupled in cascade are all equal μ_k = μ, it follows from combination of Equations (55) and (44) that the multi-scale spectrogram is given by (64) and does up to a phase shift correspond to convolution of the input signal f by filters of the form (65) (66) For comparison, the Gammatone filter with parameters a and b and frequency ϕ is defined according to (67) By identification of the parameters (68) and using ω = 2π ϕ it follows that we can derive the Gammatone filter as a special case of applying a time-causal scale-space representation with discrete scale levels to the projections f_cos(t, ω) and f_sin(t, ω) of an auditory signal f(t) onto a complex sine wave e^−iωt.

Gammatone filter banks are also commonly used in audio processing (Johannesma [33]; Patterson et al. [34, 36]; Hewitt and Meddis [35]; Irino and Patterson [58]; Ambikairajah [59]; Hohmann [60]; van Immerseel and Peeters [61]; Schlute et al. [62]; Ngamkham et al. [63]). The present treatment provides a new way of deriving them in a principled and conceptually similar way as the Gabor filters can be derived, with the differences that the temporal filtering operations are required to be truly time-causal and that only a discrete set of temporal scale levels is to be used.

Generalized Gammatone filters

In addition, by allowing for different time constants in the primitive truncated exponential filters, this scale-space concept leads to generalized Gammatone filters (69) (70) with h_composed according to (34). By comparing graphs of the two classes of auditory receptive fields based on time-causal window functions (Lindeberg and Friberg [64], fig. 6), it can be seen that the frequency selective filters based on truncated exponential filters having a logarithmic distribution of the intermediate temporal scale levels allow for a faster temporal response compared to the corresponding filters based on truncated exponential filters with equal time constants. Thereby, these generalized Gammatone filters allow for additional degrees of freedom to obtain different trade-offs between the frequency selectivity and the temporal delay of time-causal window functions by varying the number of levels K and the distribution parameter c—see the appendices “Frequency selectivity of the spectrograms” and “Temporal dynamics of the time-causal kernels” for an in-depth analysis of the frequency selectivity and the temporal delay of such kernels.

Figs. 7–8 show spectrograms computed in this way for two sound signals using the three different types of temporal window functions and using fixed vs. frequency-dependent window scales.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Spectrograms with a fixed temporal window scale ${\sigma }_t=\sqrt{\tau }=20\mathrm{\text{ms}}$ computed using different temporal scale-space concepts for (left column) the first 3 seconds of “Tom’s diner” by Suzanne Vega with the lyrics “I am sitting in the morning at the …” and (right column) a synthetic signal containing harmonic spectra with different fundamental frequencies over a logarithmic frequency scale from 80 Hz to 16 kHz using 48 frequency levels per octave: (top row) the discrete analogue of the Gaussian kernel, (middle row) a cascade of seven time-causal recursive filters having a uniform distribution of the temporal scale levels and (bottom row) a cascade of seven time-causal recursive filters having a logarithmic distribution of the temporal scale levels with $c=\sqrt{2}$.

The vertical axis shows the logarithmic frequency expressed in semitones with 69 corresponding to the tone A4 (440 Hz). Notice that while the different types of spectrograms largely capture similar types of spectro-temporal structures, there is a significant difference in temporal delay and temporal response characteristics between the non-causal and the time-causal spectrograms.

https://doi.org/10.1371/journal.pone.0119032.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Spectrograms with the temporal window scale proportional to the wavelength according to Equation (73) for n = 8 and a soft lower threshold ${\sigma }_0=\sqrt{{\tau }_0}=1\mathrm{\text{ms}}$ computed using different temporal scale-space concepts for (left column) the first 3 seconds of “Tom’s diner” by Suzanne Vega with the lyrics “I am sitting in the morning at the …” and (right column) a synthetic signal containing harmonic spectra with different ground tones over a logarithmic frequency range from 80 Hz to 16 kHz using 48 frequency levels per octave: (top row) the discrete analogue of the Gaussian kernel, (middle row) a cascade of seven time-causal recursive filters having a uniform distribution of the temporal scale levels and (bottom row) a cascade of seven time-causal recursive filters having a logarithmic distribution of the temporal scale levels with $c=\sqrt{2}$.

The vertical axis shows the logarithmic frequency expressed in semitones with 69 corresponding to the tone A4 (440 Hz). The spectrograms computed with time-causal kernels have been delay compensated by a temporal delay defined from the position of the first inflection point of the temporal window function. Notice that while the different types of spectrograms to some extent capture qualitatively similar types of spectro-temporal structures, there is a significant difference in temporal delay and temporal response characteristics between the non-causal and the time-causal spectrograms. Compared to the spectrograms in fig. 7 that are computed with a fixed temporal scale implying that the spectral bands become more narrow at higher frequencies, the use of a temporal scale proportional to the wavelength specifically implies that the widths of the spectral bands are here much more uniform over frequencies (see the section “Frequency selectivity of the spectrogram” for a theoretical analysis).

https://doi.org/10.1371/journal.pone.0119032.g008

Frequency-dependent window scale

To guarantee basic covariance properties of the spectrogram under a frequency shift (71) it is as earlier mentioned natural to let the temporal window scale vary with the frequency ω in such a a way that the temporal window scale in units of $\sigma =\sqrt{\tau }$ is proportional to the wavelength λ = 2π/ω (72) where n is a parameter. By such frequency dependency of the temporal window scale, the spectral selectivity in the spectrogram (the width of a spectral band) will be independent of the frequency ω (see the section “Frequency selectivity of the spectrograms”). This is a prerequisite for the desirable property that a shift by one octave of a musical piece should imply that the corresponding spectrogram should appear similar while shifted by one octave, if the frequency axis of the spectrogram is parameterized on a logarithmic scale.

To prevent the temporal window scale from being too short for high frequency sounds, we have additionally chosen to add a soft lower bound such that the temporal extent is instead chosen according to (73) where ${\tau }_0={\sigma }_0^2$ denotes a lower bound on the temporal window scale. Thereby, frequency covariance of a 2-D spectrogram will only be approximate, while being a good approximation if τ ≫ τ₀. If we quantify τ ≫ τ₀ as τ = β² τ₀, then the soft lower bound corresponds to (74) which with σ₀ = 1 ms, n = 8 and β = 2 corresponds to a frequency of about 4 600 Hz. By varying the parameters σ₀ and n, we can move the frequency where deviations from true invariance begin to occur for a given value of the tolerance parameter β.

To prevent the temporal delay from being too long at low frequencies, one can also introduce a soft upper bound on the temporal scale (75) for suitable values of τ_∞ and p. Then, approximate frequency covariance will hold over some subset of frequencies defined by the parameters n, τ₀, τ_∞ and p.

In human hearing, there is different evidence that the resolution of pitch perception is the highest in the area around 0.6-2 kHz and then decreases for both lower and higher frequencies (see e.g. Hartmann [65]; Moore [66]). Furthermore, within the middle area 0.6-2 kHz the relative pitch sensitivity appears to be approximately constant. The synchrony in the neural firing in the auditory nerve decreases with increasing frequency (Johnson [67]). The ability to identify the pitch of a mistuned harmonic decreases with increasing frequency exhibiting a knee at around 2 kHz (Hartmann et al. [68]). A lower frequency boundary can also be motivated from the size of the critical bands which according to the classic Zwicker data changes from being proportionally constant (about a musical minor third) for frequencies above 500 Hz to being constant (about 100 Hz) for frequencies below 500 Hz (Zwicker [69]). More recent data exhibit a similar but less strong tendency (Moore [70], page 77). Thus, in summary, there should presumable be both an upper and a lower limit for self-similarity.

Receptive fields defined over the spectrogram

Given that a spectrogram has been computed by a first layer of auditory receptive fields, we define a second layer of receptive fields by operating on the spectrogram with 2-D spectro-temporal filters (see fig. 9) in a structurally similar way as visual receptive fields are applied to time-varying visual input (Lindeberg [5, 27]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Schematic illustration of the definition of auditory features from a second layer of receptive fields over the spectrogram, where we also allow for a logarithmic transformation of the magnitude values ∣S∣ of the spectrogram prior to the application of the second layer of linear receptive fields and make use of a logarithmic transformation of the frequencies $\nu ={\nu }_0+C\log \left(\frac{\omega }{{\omega }_0}\right)$ before defining the linear receptive fields over the spectro-temporal domain.

Regarding the scale parameters, the first layer of temporal receptive fields depends on a single temporal scale parameter τ_f for the frequency selective temporal filters, whereas the second layer of auditory receptive fields also depends on an additional temporal scale parameter τ_a, a logspectral scale parameter s over the logarithmic frequencies ν and a glissando parameter v representing the rate by which the logarithmic frequencies may vary over time.

https://doi.org/10.1371/journal.pone.0119032.g009

Structural requirements on second-layer receptive fields

Given a transformed spectrogram defined in this way, let us define a family of second layer spectro-temporal receptive fields A(t, ω; Σ) that are to operate on the transformed spectrogram S_dB(t, ν;τ) and be parameterized by some multi-dimensional spectro-temporal scale parameter Σ that includes smoothing over time t and logarithmic frequencies ν, and for which the corresponding operator 𝓐_Σ is required to obey:

1. (i). linearity over the logarithmic spectrogram (81) to ensure that (a) the multiplicative relations of the magnitude of the spectrogram (77) that are mapped to linear relations by the logarithmic transformation (76) are preserved as linear relations over the receptive field responses and (b) the scale-space properties imposed to ensure non-creation of new structures in smoothed spectrograms as defined by spectro-temporal smoothing kernels do also transfer to spectro-temporal derivatives of these,
2. (ii). shift-invariance with respect to translations over time t ↦ t + Δt and logarithmic frequencies ν ↦ ν + Δν (82) such that all temporal moments and all logarithmic frequencies are treated in a similar manner. Temporal shift invariance implies that an auditory stimulus should be perceived in a similar manner irrespective of when it occurs. Shift-invariance in the logarithmic frequency domain implies that, for example, a piece of music should be perceived in a similar manner if it is transposed by e.g. one octave.

These conditions together imply that the spectro-temporal receptive fields should be given by convolution with some two-dimensional kernel over the spectro-temporal domain (Hirschmann and Widder [41]): (83) To characterize what types of receptive fields are compatible with scale-space properties, we will next impose additional structural requirements, which will take different forms depending on whether the temporal dimension is treated in a time-causal or non-causal manner:

Relations between receptive fields at different spectro-temporal scales.

For pre-recorded sound signals, for which we can take the freedom of accessing data from the virtual future in relation to any time moment, we impose a

(iii.a).
1. continuous semi-group structure over spectro-temporal scales on the second layer receptive fields (84) corresponding to an additive structure over the multi-dimensional scale parameter Σ.

For time-causal signals, we require:

(iii.b).
1. a continuous semi-group structure over logspectral scales s (85) and a Markov property between adjacent temporal scales (86)

These requirements are analogous to the previous treatment in the section “Structural requirements on temporal receptive fields”, with extensions from a purely temporal domain to a spectro-temporal domain.

Non-creation of new spectro-temporal structures with increasing scale.

When processing the spectrogram at different spectro-temporal scales, we want to ensure that the spectro-temporal receptive fields do not create new structures at coarser scales that do not correspond to simplifications of corresponding structures at finer scales. Depending on whether the temporal dimension is treated in a time-causal or non-causal manner, we formalize this condition as:

1. (iv.a). For the non-causal Gaussian spectrogram (54), for which temporal causality of the temporal smoothing kernels is disregarded, we require non-enhancement of local extrema in the sense that if for some scale Σ₀ the point (t₀, ν₀) is a local maximum (minimum) for the mapping (t, ν) ↦ (𝓐_Σ S_dB)(t, ν; Σ₀) then the value at this point must not increase (decrease) with increasing scale Σ.
2. (iv.b). For the time-causal spectrogram (55) based on truncated exponential filters coupled in cascade (33), we require: (iv.b1) the smoothing operation over the logspectral domain to satisfy non-enhancement of local extrema in the sense that if at some logspectral scale s₀ a point ν₀ is a local maximum (minimum) of the mapping ν ↦ (𝓐_Σ S_dB)(ν; s₀) obtained by disregarding the temporal variations, then the value at this point must not increase (decrease) with increasing logspectral scale s, and (iv.b2) the smoothing operation over time to be a time-causal scale-space kernel guaranteeing non-creation of new local extrema under an increase of the temporal scale parameter τ.

Glissando covariance.

In musical performance, the frequencies may vary continuously over time in such a way that the fundamental frequency ω₁ and the harmonics (overtones) ω_j are all multiplied by the same time-varying factor ω_j(t) = ψ(t)ω_j. This is in particular prominent in singing, but may occur in all instruments with continuous pitch control. In terms of logarithmic frequencies, we can model a local linearization of this temporal variability as a glissando transformation of the form (87) analogous to the way spatial image data may be subject to local Galilean transformations over time. Comparing two spectrograms, one with constant frequencies over time and one with linearly varying logarithmic frequencies, the glissando transformation can be expressed in operator form as (88) for ν′ = ν + v t. In relation to receptive field responses that are computed over the two domains with spectro-temporal scale parameters Σ and Σ′, we may require:

(v).
1. If two local patches of two spectrograms are related by a local glissando transformation, then it should be possible to relate the local spectro-temporal receptive field responses such that (89) for some transformation Σ′ = G_v(Σ) of the spectro-temporal scale parameters Σ.

Idealized models for spectro-temporal receptive fields

Given the structural requirements above, it can from derivations similar to those that are used for constraining visual receptive fields given structural requirements on a visual front-end (Lindeberg [5]) be shown that the second layer of auditory receptive fields should be based on spectro-temporal receptive fields of the form (90) where

• ∂_t^α represents a temporal derivative operator of order α with respect to time t which could alternatively be replaced by a glissando-adapted temporal derivative of the form ${\partial }_{\overline{t}}={\partial }_t+v{\partial }_{\nu }$,
• ∂_ν^β represents a logspectral derivative operator of order β with respect to logarithmic frequency ν,
• T(t; τ_a) represents a temporal smoothing kernel with temporal scale parameter τ_a, which should either be (i) a temporal Gaussian kernel g(t; τ_a) of the form (26) or (ii) the equivalent time-causal kernel h_composed(t; μ) according to (34) and corresponding to a set of truncated exponential kernels coupled in cascade,
• g(ν − vt; s) represents a Gaussian spectral smoothing kernel over logarithmic frequencies ν with logspectral scale parameter s and v representing a glissando parameter making it possible to adapt the receptive fields to variations in frequency ν′ = ν + vt over time, and
• the spectro-temporal covariance matrix Σ in the left hand side expression for spectro-temporal receptive fields comprises both the temporal scale parameter τ_a, the logspectral scale parameter s and the glissando parameter v.

Thereby, the spectro-temporal receptive fields according to (90) constitute a combination of the purely temporal receptive fields according to the theory in the sections “Structural requirements on temporal receptive fields” and “Scale-space concepts for purely temporal domains” with a Gaussian scale-space concept over the logspectral dimension.

The proofs concerning these spectro-temporal receptive fields are similar to those regarding the spatio-temporal receptive fields over a 1+1-D spatio-temporal domain with the spatial dimension replaced by a logspectral dimension.

Fig. 10 shows examples of spectro-temporal receptive fields obtained in this way for the two different types of underlying temporal scale-space concepts. For ν = 0, the resulting receptive fields are separable over the spectro-temporal domain, whereas ν ≠ 0 leads to non-separable glissando-adapted receptive fields.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Examples of idealized spectro-temporal receptive fields as obtained from spectro-temporal derivatives of spectro-temporal smoothing kernels based on (top row) the non-causal Gaussian scale-space concept and (bottom row) the time-causal scale-space concept corresponding to first-order integrators coupled in cascade (here using five temporal scale levels and $c=\sqrt{2}$).

The first four columns show separable receptive fields whereas the fifth column shows non-separable glissando-adapted receptive fields. (Horizontal dimension: time t in ms. Vertical dimension: Logarithmic frequency: ν. Temporal scale: ${\sigma }_t=\sqrt{\tau }=10\mathrm{\text{ms}}$. Logspectral scale: ${\sigma }_{\nu }=\sqrt{s}=6$ semitones.)

https://doi.org/10.1371/journal.pone.0119032.g010

Filter parameters of auditory receptive fields.

The auditory features computed from these types of receptive fields depend on three different scale parameters:

• a temporal window scale parameter τ_f defining the temporal extent of the windows over which the windowed Fourier transforms in the spectrograms are defined,
• a secondary temporal integration scale parameter τ_a defining the temporal extent over which the magnitude values in the spectrogram are integrated over time and
• a logspectral scale parameter s defining the amount of smoothing over logarithmic frequencies ν.

In addition, this family of spectro-temporal receptive fields comprises:

• a glissando parameter v that makes it possible to adapt the receptive fields to variations on the logarithmic frequencies ν over time t.

Each parameterized spectro-temporal receptive field may occur for different orders of differentiation α and β with respect to time and logarithmic frequencies, respectively.

Auditory features from second layer receptive fields

In the following, we will show examples of auditory features that can be defined from a second layer of auditory receptive fields of this form.

Spectro-temporal smoothing.

Auditory receptive fields A based on convolution with a spectro-temporal smoothing kernel T over the spectro-temporal domain: (91)

Onset and offset detection.

Computation of first-order temporal derivatives (92) where $\sqrt{{\tau }_a}$ is a scale normalization factor to approximate scale-normalized derivatives (Lindeberg [77]) by variance normalization (Lindeberg and Bretzner [78]).

This operation is similar to edge detection in image processing and computer vision (Canny [79]; Lindeberg [80]) with the differences that (i) the underlying derivatives are computed in a fixed direction and that (ii) in the case of a time-causal treatment of time, the onset detection will also be associated with a temporal delay. The signed derivative operator responds to an increase in the magnitude of the signal by a positive response and to a decrease in the magnitude by a negative response. To select receptive field responses that correspond to onsets only, this operation is naturally combined with the (non-linear) logical operation: D_t > 0 such that (see fig. 11, middle row) (93) Analogously, offset detection can be performed using (94)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Second layer receptive field responses obtained by applying spectro-temporal scale-space derivatives to the logarithmic spectrogram of (left column) the first 3 seconds of “Tom’s diner” by Suzanne Vega with the lyrics “I am sitting in the morning at the …” or (right column) a synthetic signal containing of harmonic spectra with 20 partial tones and a spectral slope of 6 dB/octave at different fundamental frequencies: (top row) Original spectrogram.

(middle row) Onset detection from the positive part of the first-order temporal derivatives ∂_t log ∣S∣ > 0 using a cascade of four time-recursive filters with temporal scale proportional to the temporal window scale ${\sigma }_t=\sqrt{\tau }=0.75{\sigma }_w$ and with logspectral smoothing scale σ_ν = 0.5 semitones. (bottom row) Spectral band detection from the negative part of the second-order temporal derivatives ∂_νν log ∣S∣ < 0. The vertical axis shows the logarithmic frequency expressed in semitones with 69 corresponding to the tone A4 (440 Hz).

https://doi.org/10.1371/journal.pone.0119032.g011

Spectral sharpening.

Computation of second-order Gaussian derivatives over the logspectral domain (95) where the factor s is a scale normalization factor for scale-normalized derivatives based on the Gaussian scale-space concept (Lindeberg [77]). Depending on the value of the logspectral scale parameter, this operation may either enhance partial tones or formants. This operation is naturally combined with the (non-linear) logical operation 𝒟_νν < 0 (see fig. 11, bottom row) (96) When applied at a fine logspectral scale, this operation can be used for enhancing spectral bands corresponding to the fundamental frequency and its overtones (see fig. 11, bottom row). When applied at a coarser logspectral scale, corresponding spectral sharpening can be used for enhancing the formants of vowels (see fig. 12). A similar approach involving a combination of Gaussian functions was used by Baer et al. [81] for enhancing spectral contrast for listeners with hearing impairment and by Heckmann et al. [55] as a part of feature extraction for automatic speech recognition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Spectral sharpening at a coarse logspectral scale (σ_ν = 4 semitones) applied to the spectrogram of the first 3 seconds of “Tom’s diner” with the lyrics “I am sitting in the morning at the …” using recursive filters at composed temporal scale 20 ms.

Note how this operation reveals the formants of the vowels in the frequency range between roughly MIDI 70 and MIDI 110, corresponding to frequencies beween roughly 450 Hz and 4.7 kHz. The vertical axis shows the logarithmic frequency expressed in semitones with 69 corresponding to the tone A4 (440 Hz).

https://doi.org/10.1371/journal.pone.0119032.g012

By comparing the responses of the partial tones in the second-order logspectral derivatives to the partial tones in the raw logarithmic spectrogram, we can note that the responses to the partial tones are far more similar between different partial tones in the logspectral derivatives compared to the raw spectrogram. This property can be understood from the invariance of spectro-temporal derivatives to local multiplications of the magnitude of a signal described in the section “Logarithmic transformations of the spectrogram”. If we model the partial tones as self-similar copies of each other at different frequencies while having different relative strength (sound pressure), then by combination of the invariance under multiplications of the magnitude in the section “Logarithmic transformations of the spectrogram” with the invariance of the relative bandwidth under multiplicative frequency transformations in the section “Frequency selectivity of the spectrograms”, it follows that the spectro-temporal derivative responses to different overtones can be expected to have a similar appearance.

Fig. 13 shows an extension of this approach, where formant enhancement is performed using glissando-adapted receptive fields, demonstrating how formant variations for different amounts of glissando are enhanced by glissando-adapted receptive fields for corresponding values of the glissando parameter.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Enhancement of the formants using glissando-adapted receptive fields corresponding to second-order derivatives with respect to logarithmic frequency ν for different glissando values v = +80, +40, 20, -20, -40 and -80 and applied to the first 3 seconds of “Tom’s diner” computed with time-causal receptive fields at temporal scale σ_t = 20 ms and logspectral scale σ_ν = 4 semitones (compare with fig. 12 that shows corresponding results for non-adapted separable receptive fields).

Note how the formant variations for different amounts of glissando are enhanced by glissando-adapted receptive fields for corresponding values of the glissando parameter. Such a set of glissando-adapted receptive fields for a logarithmic distribution of the glissando values v can serve as a filter bank for algorithms that operate on these receptive field responses as input. (Horizontal dimension: time t, Vertical dimension: logarithmic frequency ν.)

https://doi.org/10.1371/journal.pone.0119032.g013

Capturing frequency variations over time.

Given that local spectral bands have been enhanced by second-order derivatives over logarithmic frequencies (95), we can compute local extrema over frequencies by differentiating this response (97) (98) By interpolating for the zero-crossings of (97) that satisfy the sign constraint (98) we can obtain subresolution curves of how the frequency of partial tones vary over time (see fig. 14).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Spectro-temporal curves that make explicit how the frequencies of partial tones vary over time, computed as the zero-crossing curves of the logspectral derivative ∂_ν(−_νν log ∣S∣) = 0 that satisfy ∂_νν(−_νν log ∣S∣) < 0 and which thereby become continuous curves over time (drawn in black, thresholded on −_νν log ∣S∣ ≥ C for C = 3 and overlaid on −_νν log ∣S∣).

(Horizontal dimension: time t, Vertical dimension: logarithmic frequency ν.)

https://doi.org/10.1371/journal.pone.0119032.g014

Glissando estimation.

One way to estimate explicitly how the frequencies vary over time is by estimating the temporal variation in the above curves, corresponding to feature tracking in the area of computer vision.

An alternative more receptive field based approach is by computing the receptive field responses for a filter bank of different glissando-receptive fields 𝒟(v) (e.g. second-order logspectral derivatives 𝒟(v) = 𝒟_νν(v)) for different amounts of glissando v analogous to how ridge detection methods in computer vision can be expressed in terms of second-order derivatives of image intensity (Lindeberg [80]) and selecting the maximum response over the filter bank (99) as the glissando estimate (100) preferably complemented by interpolation to estimate the amount of glissando by higher accuracy than the actual sampling (compare with Laptev and Lindeberg [82] and Lindeberg [27] for corresponding filter-based approaches for estimating image velocities using a filter bank approach over different Galilean transformations).

Yet a more direct approach can be obtained by computing a spectro-temporal second-moment matrix (101) by a third layer of spectro-temporal smoothing applied to the products $L_t^2$, L_t L_ν and $L_{\nu }^2$ of the spectro-temporal derivatives L_t = ∂_t𝓣_Σ S_db and L_ν = ∂_ν𝓣_Σ S_db and then computing the glissando estimate as the value (102) that transforms the spectro-temporal moment matrix to diagonal form with the mixed ${\Upsilon }_{t\nu }^{\prime }$ being zero and corresponding to estimation of optic flow and Galilean invariant image descriptors in the area of computer vision (Lukas and Kanade [83]; Laptev et al. [84]; Lindeberg [27]). Specifically, by computing receptive field responses using a glissando estimate according to (100) alternatively for a glissando value that corresponds to a fixed-point of (102), it can be shown that the resulting glissando-adapted receptive field responses will be invariant under glissando transformations, which would not be fully possibly based on separable spectro-temporal receptive fields only (see also Lindeberg [5, 27] for analogous results regarding Galilean invariance in vision).

Relations to biological receptive fields

In the central nucleus of the inferior colliculus (ICC) of cats, Qiu et al. [31] report that about 60% of the neurons can be described as separable in the time-frequency domain (see fig. 15), whereas the remaining neurons are either obliquely oriented (see fig. 16) or contain multiple excitatory/inhibitory subfields. Glissando-adapted receptive fields have also been reported in the inferior colliculus of bats (see fig. 17). This overall structure is nicely compatible with the treatment in the section “Idealized models for spectro-temporal receptive fields”, where the second-layer receptive fields are expressed in terms of spectro-temporal derivatives of either time-frequency separable spectro-temporal smoothing operations or corresponding glissando-adapted features as motivated by the structural requirements in the section “Structural requirements on second-layer receptive fields”.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. (left) A separable monaural spectro-temporal receptive field in the central nucleus of the inferior colliculus (ICC) of cat as reported by Qiu et al.

[31] (schematic simplification only, see fig. 2(a) in [31] or fig. 16 in [64] for the original data). (middle and right) Idealized receptive fields models (90) corresponding to a first-order derivative with respect to time and a second-order derivative with respect to logarithmic frequency centered at ν = 4.7 octave, temporal scale σ_t = 7 ms, logspectral scale σ_ν = 0.17 octave for both models and additionally temporal delay δ = 17 ms for the Gaussian model.

https://doi.org/10.1371/journal.pone.0119032.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. (left) A non-separable spectro-temporal receptive field in the central nucleus of the inferior colliculus (ICC) of cat as reported by Qiu et al.

[31] (schematic simplification only, see fig. 3(a) in [31] or fig. 17 in [64] for the original data). (middle and right) First-order temporal derivative of idealized glissando-adapted receptive fields models (90) centered at ν = 4.3 octave, temporal scale σ_t = 7 ms, logspectral scale σ_ν = 0.20 octave and glissando v = −0.02 octave/ms for both models and additionally temporal delay δ = 23 ms for the Gaussian model.

https://doi.org/10.1371/journal.pone.0119032.g016

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. (left) A non-separable spectro-temporal receptive fields in the inferior colliculus (ICC) of Mexican free-tailed bat as reported by Andoni et al.

[38] (schematic simplification only, see fig. 6(c) in [38] or fig. 18 in [64] for the original data). (middle and right) Second-order temporal derivative of idealized glissando-adapted receptive fields models (90) centered at semitone ν = 138, temporal scale σ_t = 2 ms, logspectral scale σ_ν = 3 semitones and glissando v = 1.5 semitones/ms for both models and additionally temporal delay δ = 4.7 ms for the Gaussian model.

https://doi.org/10.1371/journal.pone.0119032.g017

Qualitatively similar shapes of receptive fields can be measured from neurons in the primary auditory cortex (see figs. 18 and 19 as well as Miller et al. [3] regarding binaural receptive fields). Specifically, the use of multiple temporal and spectral scales as a main component in the model is in good agreement with biological receptive fields having different degrees of spectral tuning ranging from narrow to broad (see fig. 20) and different temporal extent (see fig. 18). Corresponding tradeoffs between spectral and temporal tuning occur in the inferior colliculus (Rodriguez et al. [85]). The distribution of latencies is, however, towards somewhat shorter latencies in the thalamus than in the auditory cortex (Miller et al. [3]) consistent with larger temporal scales in the auditory cortex than in the inferior colliculus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 18. (left column) Separable spectro-temporal receptive fields in the primary auditory cortex (A1) of Sprague Dawley rat as reported by Machens et al.

[37] (schematic simplification only, see figs. 5(a) and 5(c) in [37] or fig. 19 in [64] for the original data). (middle and right columns) Idealized receptive fields models (90) corresponding to first-order derivatives with respect to time, in the top row centered at semitone ν = 91, temporal scale σ_t = 45 ms, logspectral scale σ_ν = 6 semitones for both models and additionally temporal delay δ = 60 ms for the Gaussian model and in the bottom row centered at semitone ν = 84, temporal scale σ_t = 60 ms, logspectral scale σ_ν = 6 semitones for both models and additionally temporal delay δ = 100 ms for the Gaussian model

https://doi.org/10.1371/journal.pone.0119032.g018

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 19. (left) A separable spectro-temporal receptive fields in the primary auditory cortex (A1) of ferret as reported by Elhilali et al.

[39] (schematic simplification only, see fig. 6(a) in [39] or fig. 20 in [64] for the original data). (middle and right) Idealized receptive fields models (90) corresponding to second-order derivatives with respect to logarithmic frequency centered at semitone ν = 116, temporal scale σ_t = 17 ms, logspectral scale σ_ν = 5 semitones for both models and additionally temporal delay δ = 25 ms for the Gaussian model.

https://doi.org/10.1371/journal.pone.0119032.g019

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 20. (left column) Spectro-temporal receptive fields of broadly and narrowly tuned neurons in the primary auditory cortex (A1) of cats as reported by Atencio and Schreiner [40].

In terms of the idealized models of receptive fields, such broad or narrowly tuned receptive fields correspond to different values of the logspectral scale parameter s. (middle and right columns) Idealized receptive fields models (90) corresponding to first-order derivatives with respect to time, in the left column centered at semitone ν = 119, temporal scale σ_t = 12 ms, logspectral scale σ_ν = 8 semitones for both models and additionally temporal delay δ = 20 ms for the Gaussian model and in the bottom column centered at semitone ν = 122, temporal scale σ_t = 45 ms, logspectral scale σ_ν = 0.5 semitones for both models and additionally temporal delay δ = 60 ms for the Gaussian model.

https://doi.org/10.1371/journal.pone.0119032.g020

Whereas spatio-temporal receptive fields estimated from neurons in the auditory cortex have been reported to reasonably well predict the neural responses for subsets of natural stimuli, Machens et al. [37] report that for many natural stimuli the responses of the auditory neurons in the primary auditory cortex cannot be predicted by the estimated linear receptive fields. Atencio et al. [86] also report that the dimensionality and thereby the variability of the receptive fields in the auditory cortex is significantly richer compared to the receptive fields in the inferior colliculus in the midbrain. Thus, the neurons in the primary auditory cortex appear to contain non-linearities whose functionality remains to be understood. In the inferior colliculus in the midbrain, Escabi and Schreiner [87] report that about 60% of the receptive fields can be well described in terms of linearly integrating neurons.

In the work by Qiu et al. [31], the measured biological receptive fields were fitted to Gabor functions as motivated by previous use of Gabor functions for modelling visual receptive fields (Marcelja [88]; Jones and Palmer [89, 90]). In vision, the use of Gabor functions for modelling visual receptive fields can, however, be questioned both on theoretical and empirical grounds (Stork and Wilson [91]; Lindeberg [5, 27]).

Stork and Wilson [91] argue that (i) only complex-valued Gabor functions, which cannot be described by single-unit receptive fields, minimize the uncertainty relation, (ii) the real functions that minimize this relation are Gaussian derivatives rather than Gabor functions and (iii) comparisons among Gabor and alternative fits to both psychophysical and physiological data have shown that in many cases other functions (including Gaussian derivatives) provide better fits than Gabor functions do.

Lindeberg [5, 27] argues that in relation to invariance properties the family of affine Gaussian kernels is closed under affine image deformations, whereas the family of Gabor functions obtained by multiplying rotationally symmetric Gaussians with sine and cosine waves is not closed under affine deformations. Therefore, it is not possibly to compute truly affine invariant representations from such Gabor functions. Instead, given a pair of images that are related by an affine image deformation, the lack of affine covariance implies that there will be a systematic bias in representations derived from such Gabor functions, corresponding to the difference between the backprojected Gabor functions in the two domains (compare with fig. 3 in [5] for an illustration). Using receptive profiles defined from directional derivatives of affine Gaussian kernels, it will on the other hand be possible to compute affine invariant representations. Similar arguments about Galilean invariance hold regarding theoretical modelling of spatio-temporal receptive fields.

In the visual domain, such affine transformations constitute a natural first-order model for image deformations caused by changing the viewing direction relative to an object or seeing an object that moves relative to the observer. In the auditory domain, affine transformations in terms of glissando transformations constitute a natural first-order model for frequencies that vary over time, such as in singing, melodic speech or sound from instruments having continuous pitch control.

Applied to the auditory domain, this means that invariance under glissando transformations cannot be accomplished with the regular family of Gabor functions, whereas glissando invariance is possible based on the presented theory for glissando-adapted receptive fields.

In addition, Gaussian receptive fiels as well as the derivatives of these can be modelled by diffusion equations, and can therefore be implemented by computations between neighbouring computational units, which is biologically plausible in terms of connections between neighbouring neurons and of minimizing the wiring length between them.

Specifically, biological spectro-temporal receptive fields show a marked temporal asymmetry that cannot be captured by Gabor functions for which the locations of excitatory and inhibitory subregions are uniformly spaced. Therefore, Qiu et al. [31] performed additional non-linear time warping to be able to fit the model to the data. Then, they modelled oblique receptive fields over the time-frequency domain using singular value decomposition to express any oblique receptive field as a sum of separable Gabor-based receptive fields.

By modelling the spatio-temporal receptive fields by a combination of time-causal scale-space kernels over time and Gaussian receptive fields over logarithmic frequencies, the temporal asymmetry of the kernels constitutes an integrated part of the theory, which cannot be accomplished by temporally symmetric smoothing operations as used in the Gabor model of receptive fields. Furthermore, oblique receptive fields in the time-frequency domain do also constitute an integrated part of the theory in terms of glissando transformations, and there is no need to decompose a glissando-receptive field as a sum of a possibly rather large number of spectro-temporal receptive fields, to be able to model the spectro-temporal receptive field in a quantitative manner. In addition, the model has been derived in a mathematically principled way from a set of structural requirements and the idealized receptive fields can be computed by a combination of diffusion equations and first-order integrators, and therefore by a biologically plausible neural architecture.

Relations to previous work in audio processing

Previous auditory models of human hearing have to a large extent focused on the first stages of processing including the acoustical response of the cochlea and the following neurological responses in the auditory nerve. The auditory periphery is also the stage at which it was possible to collect the first physiological and neurological data. Recently, partly due to the fast technological progress in measurement techniques, models of more high-level functions in the auditory cortex begin to emerge (Meddis et al. [92]). Thus, the purpose has been to convert the incoming audio into a frequency-time representation in a similar way as is done in the physical-neural system in the cochlea. This is typically done in several stages taking into account both biological measurements and psychoacoustic listening test data.

The main stage is to simulate the physical resonance system in the cochlea. It is often implemented as filter bank in which the bandwidth and frequency position of each band is separated in a similar way as the cochlear nerves. A gammatone filter is often used since it has been show to be a reasonable approximation of the acoustic properties in the cochlea leading to the first neural input (Patterson et al. [93, 94]). Many other types of filters have been proposed, such as the gammachirp providing better fit to non-linearities with regard to the asymmetric frequency response at loud sound pressures (Irino and Patterson [58]; see also Chen et al. [95] and Lopez-Poveda and Meddis [96]). In a second stage, the response of the auditory nerves arriving from the inner hair cells in the cochlea are modelled. A common approach is to make a half-wave rectification, compression (square-root or logarithmic) and low-pass filtering (Patterson et al. [36]). In addition the local contrast can be enhanced both in time and frequency using an adaptive procedure (Patterson and Holdsworth [97]).

Previous computational toolboxes include the auditory image model (Aim-mat) by Bleek et al. [98]; see also Patterson et al. [99] which include all of the different parts above as well as some additional parts and the auditory model by Slaney [100].

Thus, these auditory image models are quite advanced and take into account a number of biological and perceptual phenomena. However, the biological data supporting these stages seems rather scarce in particular for complex sound signals such as music. The nerve responses have often been measured in animals rather than humans and with rather simple stimuli such as stationary sinusoids (Ruggero [101]). Many of the parts are modelled after psychoacoustic data again with simple stimuli and thus involving the whole auditory cortex and brain. A limitation is that advanced perceptual models can be adapted to closely model a certain type of perceptual data but will typically not extrapolate to other perceptual conditions. The recent loudness model by Chen et al. [95] closely approximates perceptual data concerning several aspects of loudness but applies only to steady-state sounds. This is an indication that the formulation of the underlying model(s) is still potentially open to alternative solutions. This is not surprising giving the complexity of the task and difficulties in obtaining biological data.

The traditional auditory models are not necessarily the best choice as a front-end for modelling more high-level perceptual aspects of music and speech. These models are often very demanding in terms of computer power and memory. In addition, the resulting data may not be suitable for further processing. Therefore, in practical applications, a front-end used for analyzing perceptual phenomena is often a simplification of the complete model. Recently, alternative models have been suggested which apply general principles of auditory perception but leaving out the detailed aspects. Such a model can be a good compromise between biological/perceptual reality and computational clarity and efficiency. For example, Chi et al. [102] used for the first stage 128 overlapping constant-Q bandpass filters with 24 filters/octave, a hair cell model with a high-pass filter, a non-linear compression, a membrane leakage low-pass filter, and a simplification of a lateral inhibitory network. In addition, they also modelled a second stage of cortical processing using spectro-temporal receptive fields (STRFs) applied on the spectrogram derived in the first stage (Chi et al. [102]). Another example of such a simplified two-stage model including an auditory spectrogram with STRFs applied to speech recognition was presented by Heckmanns et al. [55].

Traditionally, the short-time Fourier transform (STFT) has been used extensively for converting the signal to a time-frequency representation, presumably due to its efficient computer implementation, the fast Fourier transform (FFT). One of the major drawbacks with the STFT is the frequency resolution, which is constant in terms of Hertz. Since the ear is approximately logarithmic with regard to the frequency band distribution, a major part of the frequency data in the upper treble region is less relevant for a perceptual analysis. Similarly the resolution in the bass range is not enough for example for determining the musical pitch using a time window that can capture the onsets of fast notes (Muller et al. [103]).

One possibility to achieve a log-frequency spectrum is to use a bank of individual bandpass filters as discussed above and also in line with the currently proposed model. This can, however, be rather time consuming. An interesting computationally efficient compromise is therefore the constant-Q transform which uses traditional STFTs applied in a combination of downsampling and different time resolutions for different octaves (Brown and Puckette [104]). A computational toolbox was recently presented by Schörkhuber and Klapuri [105]. Using the constant-Q method, the frequency resolution will be the same across the spectrum. The time resolution will, however, vary significantly across the spectrum and still exhibit poor time resolution in the bass region.

Similar approaches using a second layer of receptive fields applied on the spectrogram have been used in particular in speech research using Gabor functions (Kleinschmidt [49]; Ezzat et al. [52]; Meyer and Kollmeier [106]; Heckmann et al. [55]; Wu et al. [32]). Heckmann et al. [55] used Gabor-based receptive fields of different orientations in the time-frequency plane of the spectrogram in combination with different transformations inspired by visual object recognition to capture the formant trajectories over time. The resulting features were shown to improve the performance of a speech recognition system in combination with traditional features such as mel frequency cepstral components (MFCC). In this article, we show how such and related auditory operations can be derived in principled manner.

Summary and discussion

We have presented a theory for how idealized models of auditory receptive fields can be formulated based on structural constraints on the first stages of auditory processing. The theory includes (i) the definition of multi-scale spectrograms at different temporal scales in such a way that a spectrogram at any coarser temporal scale can be related to a corresponding spectrogram at any finer temporal scale using theoretically well-defined scale-space operations, and additionally (ii) how a second-layer of spectro-temporal receptive fields can be defined over a logarithmically transformed spectrogram in such a way that the resulting spectro-temporal receptive fields obey invariance or covariance properties under natural sound transformations including temporal shifts, variations in the sound pressure, the distance between the sound source and the observer, a shift in the frequencies of auditory stimuli or glissando transformations. Specifically, theoretical arguments have been presented showing how these idealized receptive fields are constrained to the presented forms from symmetry properties of the environment in combination with assumptions about the internal structure of auditory operations as motivated from requirements of handling different temporal and spectral scales in a theoretically well-founded manner.

By combining the scale-space approach with a local frequency analysis, we obtain a new way of deriving the Gabor filters as a complex-valued scale-space transform resulting from the Gaussian scale-space concept being applied to a temporal signal multiplied by a complex sine wave. We can also derive the Gamma-tone filters in a corresponding manner, as a time-causal complex scale-space transform obtained by applying a set of time-causal scale-space kernels based on first-order integrators with equal time constants coupled in cascade and applied to a temporal signal multiplied by a complex sine wave. In addition, the scale-space approach to multi-scale spectrograms leads to a new family of generalized Gamma-tone filters obtained from a logarithmic distribution of the intermediate temporal scales, and allowing for different trade-offs between filter characteristics such as frequency selectivity and temporal delay.

Then, given that a multi-scale spectrogram has been defined and transformed by taking the logarithm of the magnitude values and expressing the frequencies on a logarithmic frequency scale to ensure natural covariance properties under variations of the sound pressure or a frequency shift in the stimulus, the theory provides a second layer of receptive fields applied to the spectrogram, based on spectro-temporal derivatives of spectro-temporal scale-space kernels. We have shown how the derived models of idealized spectro-temporal receptive fields are uniquely determined given natural symmetry properties (scale-space axioms) and we have shown examples of how basic auditory features can be computed in this way.

Thus, the presented scale-space theory for auditory signals can be both related to existing models for auditory analysis and additionally leads to the formulation of a set of new models. Specifically, the presented theory provides a coherent framework by which auditory receptive fields at the first levels of processing in the auditory hierarchy can be expressed within the same theoretical framework. Moreover, the theory allows for provable invariance properties under temporal shifts, variations in sound pressure and logarithmic frequency shifts.

In relation to biology, the theoretical framework thus shows that idealized receptive field profiles can be derived by necessity given a set of idealized assumptions that reflect structural properties of the environment, and lead to predictions about receptive field profiles that are qualitatively very similar to receptive fields found by cell recordings in the inferior colliculus (ICC) and the primary auditory cortex (A1) of mammals. From this theoretical background, the auditory receptive fields in these auditory areas can therefore be seen as highly adapted to the structure of the environment and enabling the computation of invariant representations of the auditory input.

In relation to audio processing, the paper shows how one can derive three types of temporal filters for defining spectrograms, which provides a new way to derive Gabor filters and Gammatone filters in conceptually similar ways, and also to derive a new family of generalized Gammatone filters with additional degrees of freedom to obtain different trade-offs between the spectral selectivity and the temporal delay of time-causal filters. The paper also presents a theory for defining a second layer of receptive fields on such spectrograms and gives examples of how basic auditory features can be computed in this way. These results are derived by mathematical necessity from the assumptions. It is furthermore shown how such audio processing operations can be related to biological receptive fields in auditory perception as well as to a corresponding theory for visual receptive fields.

We propose that this theory should be of wide general interest for researchers in the audio processing community by providing theoretically well-founded and provably invariant/covariant audio operations for processing sound signals and to researchers working with computational modelling or measurements of receptive fields, auditory invariances, theoretical biology and psychophysics, by serving as a general theoretical foundation and understanding of how receptive fields in ICC and A1 support invariant visual processes at higher levels in the auditory hierarchy.

Frequency selectivity of the spectrograms

Consider a sine wave signal with angular frequency ω₀: (103) When computing the windowed spectrogram, we multiply this signal by sine and cosine waves of different angular frequencies ω and integrate by a window function h(t; τ) with temporal extent τ: (104) (105) By the use of basic rules for trigonometric functions (106) (107) the result of convolving these components with the window function h(t; τ) can be expressed by multiplication with the Fourier transform $\widehat{h}(\omega ;\tau )$: (108) (109) Concerning the magnitude of the spectrogram (110) it follows that (111) Assuming that the window function h should be a low-pass filter, then for ω close to ω₀ let us assume (112) Thereby, the dominant component of the spectrogram near ω₀ will be given by (113) By normalizing this entity such that the maximum value at ω = ω₀ equals one, we quantify the frequency selectivity for a frequency dependent window scale τ(ω) as (114) which on a logarithmic dB scale assumes the form (115) where we would ideally choose the temporal extent of the kernel in units of $\sigma =\sqrt{\tau }$ proportional to the wavelength λ = 2π/ω for any angular frequency ω: (116)

Gaussian window functions.

For a Gaussian window function we have (117) With the temporal extent of the window function proportional to the wavelength for any frequency according to (116), the frequency selectivity is given by (118) or in dB (119)

Window functions defined from cascade of truncated exponential functions

For the truncated exponential filters coupled in cascade, the Laplace transform is (120) implying that the Fourier transform is given by (121) In the special case when all time constants μ_k are equal (122) and with the temporal extent of the window function proportional to the wavelength according to (116), the frequency selectivity is given by (123) or in dB (124) In the special case when the intermediate temporal scale levels τ_k are instead distributed according to a logarithmic distribution with τ_k = c^{2(k − K)} τ and μ_k according to (39) and (40), and with the temporal extent of the window function proportional to the wavelength according to (116), we obtain (125) or in dB (126) Fig. 21 shows graphs of the frequency selectivity for the different types of temporal window functions and a few combinations of the underlying filter parameters. The non-causal Gaussian kernel has sharper frequency selectivity compared to the time-causal kernels. Within the class of time-causal kernels, the frequency selectivity increases with the number of truncated exponential kernels that are coupled in cascade. For the logarithmic distribution of the intermediate temporal scale levels, the frequency selectivity also increases with decreasing values of the distribution parameter c.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 21. Graphs of the frequency selectivities of Gaussian and time-causal window functions with the temporal extent proportional to the corresponding wavelength of the spectrogram with n = 8.

For the time-causal filters, K = 4 or K = 7 filters have been coupled in cascade. For the logarithmic distribution, the ratio between successive temporal scale levels τ_k = c^{2(k − K)} τ has been determined from $c=\sqrt{2}$ or c = 2. (Horizontal axis: Angular frequency ω in units of ω₀. Vertical axis: dB values down to -60 dB.)

https://doi.org/10.1371/journal.pone.0119032.g021

Dependency of the relative bandwidth on n

Notably all these expressions are functions of the ratio (127) By for a fixed value of θ solving for ω and assuming θ > 0 (128) we get explicit expressions for how the relative bandwidth of the spectrogram corresponding to that specific value of θ (129) alternatively in logarithmic MIDI units (130) depends on n for any θ, implying that the relative bandwidth decreases approximately inversely proportional to n, where θ is related to the dB level R_dB < 0 according to (131) for the Gaussian window functions and according to (132) for the time-causal kernels having a uniform distribution of the intermediate temporal scale levels. For the time-causal kernels having a logarithmic distribution of the intermediate scale levels, the parameter θ can be determined by solving the following equation (133) numerically for specific values of K and c. Table 1 shows such values for K = 4 and K = 7 for $c=\sqrt{2}$ and c = 2 as well as corresponding values for a uniform distribution of the intermediate scale levels and a Gaussian window function.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Numerical values of the parameter θ determining the relative bandwidth $\frac{2\theta }{n}$ of the spectrogram according to (129), for a Gaussian function and K truncated exponential kernels in cascade with a uniform distribution of the intermediate temporal scale levels τ_k = τ/K or a logarithmic distribution τ_k = c^{2(k − K)} τ with c > 1.

https://doi.org/10.1371/journal.pone.0119032.t001

Frequency invariance

From the invariance of the expressions (118), (123) and (125) under frequency transformations of the form (134) (135) for any α > 0, it follows that the relative bandwidth of the spectrogram will be independent of the angular frequency ω. Thereby, over the range of frequencies for which the temporal extent of the window function is proportional to the wavelength, the spectral sensitivity will be invariant under a shift in frequency ω ↦ α ω, providing a foundation for frequency covariant receptive fields at higher levels in the auditory hierarchy.

Temporal dynamics of the time-causal kernels

For the time-causal filters obtained by coupling truncated exponential kernels in cascade, there will be an inevitable temporal delay depending on the time constants μ_k of the individual filters.

A most straightforward way of estimating this delay is by using the additive property of mean values under convolution (136) In the special case of all the time constants being equal ${\mu }_k=\sqrt{\tau /K}$, this measure is given by (137) showing that the temporal delay increases if the temporal smoothing operation is divided into a larger number of smaller individual smoothing steps.

In the special case when the intermediate temporal scale levels are instead distributed logarithmically according to (38), with the individual time constants given by (39) and (40), this measure for the temporal delay is given by (138) with the limit value (139) when the number of filters tends to infinity.

By comparing Equations (137), (138) and (139), we can specifically note that with increasing number of intermediate temporal scale levels, a logarithmic distribution of the intermediate scale levels implies shorter temporal delays than a uniform distribution of the intermediate scale levels.

Table 2 shows numerical values of these measures for different values of K and c. As can be seen from the table, the logarithmic distribution of the intermediate scales allows for significantly faster temporal dynamics than a uniform distribution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Numerical values of the temporal delay in terms of the temporal mean $m={\sum }_{k=1}^K{\mu }_k$ in units of $\sigma =\sqrt{\tau }$ for time-causal kernels obtained by coupling K truncated exponential kernels in cascade in the cases of a uniform distribution of the intermediate temporal scale levels τ_k = kτ/K or a logarithmic distribution τ_k = c^{2(k − K)} τ with c > 1.

https://doi.org/10.1371/journal.pone.0119032.t002

Additional temporal characteristics.

Because of the asymmetric tails of the time-causal temporal smoothing kernels, temporal delay estimation by the mean value may however lead to substantial overestimates compared to e.g. the position of the local maximum. To provide more precise characteristics in the case of a uniform distribution of the intermediate temporal scale levels, for which a compact closed form expression is available for the composed kernel (140) let us differentiate this function (141) and solve for the positions of the local maximum and the inflection points (142) (143) (144) Table 3 shows numerical values for the position of the local maximum for both types of time-causal kernels. As can be seen from the table, the temporal response properties are significantly faster for a logarithmic distribution of the intermediate scale levels compared to a uniform distribution, and the difference increases rapidly with K. These temporal delay estimates are also significantly shorter than the temporal mean values, in particular for the logarithmic distribution of the intermediate scale levels.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Numerical values for the temporal delay of the local maximum in units of $\sqrt{\tau }$ for time-causal kernels obtained by coupling K truncated exponential kernels in cascade in the cases of a uniform distribution of the intermediate temporal scale levels τ_k = kτ/K or a logarithmic distribution τ_k = c^{2(k − K)} τ with c > 1.

https://doi.org/10.1371/journal.pone.0119032.t003

If we consider a temporal event that occurs as a step function over time (e.g. an onset in the magnitude of the spectrogram which is then processed by a second layer of spectro-temporal receptive fields) and if the temporal position of this onset is estimated from the local maximum over time in the first-order temporal derivative response, then the temporal variation in the response over time will be given by shape of the temporal smoothing kernel. The local maximum over time will occur at a time delay equal to the time at which the temporal kernel has its maximum over time. Thus, the position of the maximum over time of the temporal smoothing kernel is highly relevant for quantifying the temporal responses characteristics of time-causal filters.

Computational implementation

The computational model for auditory receptive fields presented in this paper is based on auditory signals that are assumed to be continuous over time and with frequencies that are also assumed to take values over a continuous frequency domain. When implementing this model on sampled sound signals, the continuous theory must be transferred to discrete time and be restricted to a finite set of discrete frequencies.

In this section we describe how the temporal and spectro-temporal receptive fields can be implemented in terms of corresponding discrete scale-space kernels that possess scale-space properties over discrete temporal and spectro-temporal domains.

Acknowledgments

We would like to thank Anders Elowsson, Pawel Herman, Bernhard Kaplan and the anonymous reviewer for valuable comments that improved the presentation.