Depression Classification Using Token Merging-Based Speech Spectrotemporal Transformer | SpringerLink
Skip to main content

Depression Classification Using Token Merging-Based Speech Spectrotemporal Transformer

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15299))

Included in the following conference series:

  • 162 Accesses

Abstract

This paper introduces a novel approach for depression classification, utilizing multimodal token merging (ToMe) within a speech spectrotemporal transformer framework. The model’s efficacy is evaluated with log-mel spectrograms and autocorrelation tempograms extracted from depressed and non depressed speech. The results demonstrate the effectiveness of ToMe when integrated with attention mechanisms of audio spectrogram transformer (AST) models, such as AST and data efficient image transformer (DeiT) encoders. This underscores the importance of the token pruning mechanism utilized in the study. Additionally, a multimodal dual-channel architecture is introduced, featuring two distinct feature modalities extracted from speech: spectrograms and autocorrelation tempograms. The novel ToMe dual-channel AST and ToMe dual-channel AST with DeiT encoder models demonstrate remarkable performance on two different datasets, namely the EATD-Corpus (Chinese) and DAIC-WoZ (English), providing promising results for depression detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 7550
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 9437
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Al Hanai, T., Ghassemi, M.M., Glass, J.R.: Detecting depression with audio/text sequence modeling of interviews. In: Interspeech, pp. 1716–1720 (2018)

    Google Scholar 

  2. Alhanai, T., Ghassemi, M., Glass, J.: Detecting depression with audio/text sequence modeling of interviews. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2018-September, pp. 1716–1720 (2018). https://doi.org/10.21437/Interspeech.2018-2522

  3. Association, A.P., et al.: Diagnostic and statistical manual of mental disorders. Text revision (2000)

    Google Scholar 

  4. Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: your vit but faster (2023)

    Google Scholar 

  5. Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., Quatieri, T.F.: A review of depression and suicide risk assessment using speech analysis. Speech Commun. 71, 10–49 (2015)

    Article  Google Scholar 

  6. Dablain, D., Krawczyk, B., Chawla, N.V.: Deepsmote: fusing deep learning and smote for imbalanced data. IEEE Trans. Neural Netw. Learn. Syst. 34(9), 6390–6404 (2022)

    Article  Google Scholar 

  7. Dosovitskiy, A., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale (2021)

    Google Scholar 

  8. Elkin, I., et al.: National institute of mental health treatment of depression collaborative research program: general effectiveness of treatments. Arch. General Psychiat. 46(11), 971–982 (1989). https://doi.org/10.1001/archpsyc.1989.01810110013002

    Article  MATH  Google Scholar 

  9. France, D.J., Shiavi, R.G., Silverman, S., Silverman, M., Wilkes, M.: Acoustical properties of speech as indicators of depression and suicidal risk. IEEE Trans. Biomed. Eng. 47(7), 829–837 (2000)

    Article  MATH  Google Scholar 

  10. Fraser, G., Boashash, B.: Multiple window spectrogram and time-frequency distributions. In: Proceedings of ICASSP 1994. IEEE International Conference on Acoustics, Speech and Signal Processing. vol. iv, pp. IV/293–IV/296 (1994). https://doi.org/10.1109/ICASSP.1994.389818

  11. Fried, E.I., Nesse, R.M.: Depression sum-scores don’t add up: why analyzing specific depression symptoms is essential. BMC Med. 13(1), 1–11 (2015)

    Article  Google Scholar 

  12. Gong, Y., Chung, Y.A., Glass, J.: Ast: audio spectrogram transformer (2021)

    Google Scholar 

  13. Gratch, J., et al.: The distress analysis interview corpus of human and computer interviews. In: Calzolari, N. (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp. 3123–3128. European Language Resources Association (ELRA), Reykjavik (2014). http://www.lrec-conf.org/proceedings/lrec2014/pdf/508Paper.pdf

  14. Grosche, P., Müller, M., Kurth, F.: Cyclic tempogram-a mid-level tempo representation for musicsignals (2010). https://doi.org/10.1109/ICASSP.2010.5495219

  15. Hanai, T., Ghassemi, M., Glass, J.: Detecting depression with audio/text sequence modeling of interviews, pp. 1716–1720 (2018). https://doi.org/10.21437/Interspeech.2018-2522

  16. Iyortsuun, N.K., Kim, S.H., Yang, H.J., Kim, S.W., Jhon, M.: Additive cross-modal attention network (acma) for depression detection based on audio and textual features. IEEE Access 12, 20479–20489 (2024). https://doi.org/10.1109/ACCESS.2024.3362233

    Article  Google Scholar 

  17. Jiang, H., et al.: Detecting depression using an ensemble logistic regression model based on multiple speech features. Comput. Math. Methods Med. 2018 (2018)

    Google Scholar 

  18. Kaustubh, K., Gogoi, P., Prasanna, S.: Rhythm formant analysis for automatic depression classification. In: International Conference on Speech and Computer, pp. 94–106. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-48309-7_8

  19. Lam, G., Huang, D., Lin, W.: Context-aware deep learning for multi-modal depression detection. In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3946–3950 (2019). https://api.semanticscholar.org/CorpusID:145833193

  20. Lu, J., Liu, B., Lian, Z., Cai, C., Tao, J., Zhao, Z.: Prediction of depression severity based on transformer encoder and cnn model. In: 2022 13th International Symposium on Chinese Spoken Language Processing (ISCSLP), pp. 339–343. IEEE (2022)

    Google Scholar 

  21. Ma, X., Yang, H., Chen, Q., Huang, D., Wang, Y.: Depaudionet: an efficient deep model for audio based depression classification. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 35–42 (2016)

    Google Scholar 

  22. Ma, X., Yang, H., Chen, Q., Huang, D., Wang, Y.: Depaudionet: an efficient deep model for audio based depression classification. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge (2016). https://api.semanticscholar.org/CorpusID:2518379

  23. McFee, B., et al.: librosa: audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, vol. 8 (2015)

    Google Scholar 

  24. Othmani, A., Kadoch, D., Bentounes, K., Rejaibi, E., Alfred, R., Hadid, A.: Towards robust deep neural networks for affect and depression recognition from speech. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12662, pp. 5–19. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68790-8_1

    Chapter  Google Scholar 

  25. Park, D.S., et al.: Specaugment: a simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)

  26. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library (2019). https://arxiv.org/abs/1912.01703

  27. Sardari, S., Nakisa, B., Rastgoo, M.N., Eklund, P.: Audio based depression detection using convolutional autoencoder. Expert Syst. Appl. 189, 116076 (2022)

    Article  Google Scholar 

  28. Satt, A., Rozenberg, S., Hoory, R., et al.: Efficient emotion recognition from speech using deep learning on spectrograms. In: Interspeech, pp. 1089–1093 (2017)

    Google Scholar 

  29. Shen, Y., Yang, H., Lin, L.: Automatic depression detection: an emotional audio-textual corpus and a gru/bilstm-based model (2022)

    Google Scholar 

  30. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention (2021)

    Google Scholar 

  31. Valstar, M., et al.: Avec 2016 - depression, mood, and emotion recognition workshop and challenge (2016)

    Google Scholar 

  32. Verde, L., et al.: A lightweight machine learning approach to detect depression from speech analysis. In: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 330–335. IEEE (2021)

    Google Scholar 

  33. Wei, P.C., Peng, K., Roitberg, A., Yang, K., Zhang, J., Stiefelhagen, R.: Multi-modal depression estimation based on sub-attentional fusion (2022)

    Google Scholar 

  34. Wu, P., Wang, R., Lin, H., Zhang, F., Tu, J., Sun, M.: Automatic depression recognition by intelligent speech signal processing: a systematic survey. CAAI Trans. Intell. Technol. 8(3), 701–711 (2023)

    Article  Google Scholar 

  35. Zhao, Z., et al.: Automatic assessment of depression from speech via a hierarchical attention transfer network and attention autoencoders. IEEE J. Sel. Topics Signal Process. 14(2), 423–434 (2019)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lokesh Kumar .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kumar, L., Kaustubh, K., Prasanna, S.R.M. (2025). Depression Classification Using Token Merging-Based Speech Spectrotemporal Transformer. In: Karpov, A., Delić, V. (eds) Speech and Computer. SPECOM 2024. Lecture Notes in Computer Science(), vol 15299. Springer, Cham. https://doi.org/10.1007/978-3-031-77961-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-77961-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-77960-2

  • Online ISBN: 978-3-031-77961-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics