Abstract
Recent studies show that the spatial distribution of the sentence representations generated from pre-trained language models is highly anisotropic. This results in a degradation in the performance of the models on the downstream task. Most methods improve the isotropy of the sentence embeddings by refining the corresponding contextual word representations, then deriving the sentence embeddings from these refined representations. In this study, we propose to improve the quality of the sentence embeddings extracted from the [CLS] token of the pre-trained language models by improving the isotropy of the embeddings. We add one feed-forward layer between the model and the downstream task layers, and we train it using a novel joint loss function. The proposed approach results in embeddings with better isotropy, that generalize better on the downstream task. Experimental results on 3 GLUE datasets with classification as the downstream task show that our proposed method is on par with the state-of-the-art, as it achieves performance gains of around 2–3% on the downstream tasks compared to the baseline.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The labels of the test sets were not provided (they are only evaluated through the leaderboard at https://gluebenchmark.com/leaderboard) This setup also follows the work done by IsoBN [22].
- 2.
More information regarding hyper-parameter tuning is available in a technical report [1].
References
Attieh, J.: Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings using a Joint Loss Function. Master’s thesis, Aalto University. School of Science (2022). http://urn.fi/URN:NBN:fi:aalto-202209255727
Biś, D., Podkorytov, M., Liu, X.: Too much in common: Shifting of embeddings in transformer language models and its implications. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5117–5130. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.403, http://aclanthology.org/2021.naacl-main.403
Ethayarajh, K.: How contextual are contextualized word representations? comparing the geometry of BERT, ELMO, and GPT-2 embeddings. vol. abs/1909.00512 (2019). arXiv: abs/1909.00512
Gao, J., He, D., Tan, X., Qin, T., Wang, L., Liu, T.: Representation degeneration problem in training natural language generation models. In: International Conference on Learning Representations (2019). http://openreview.net/forum?id=SkEYojRqtm
Gong, C., He, D., Tan, X., Qin, T., Wang, L., Liu, T.Y.: Frage: Frequency-agnostic word representation. ArXiv arXiv:1809.06858 (2018)
Kalyan, K.S., Rajasekharan, A., Sangeetha, S.: Ammus : A survey of transformer-based pretrained models in natural language processing (2021). https://doi.org/10.48550/ARXIV.2108.05542,
Li, B., Zhou, H., He, J., Wang, M., Yang, Y., Li, L.: On the sentence embeddings from pre-trained language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.733
Liang, Y., Cao, R., Zheng, J., Ren, J., Gao, L.: Learning to remove: Towards isotropic pre-trained Bert embedding. In: Artificial Neural Networks and Machine Learning - ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part V, p. 448–459. Springer-Verlag, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-86383-8_36
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2017). https://doi.org/10.48550/ARXIV.1711.05101
Mu, J., Viswanath, P.: All-but-the-top: Simple and effective post-processing for word representations (2018), publisher Copyright: © Learning Representations, ICLR 2018 - Conference Track Proceedings. All right reserved.; 6th International Conference on Learning Representations, ICLR 2018; Conference date: 30–04-2018 Through 03–05-2018
Rajaee, S., Pilehvar, M.T.: A cluster-based approach for improving isotropy in contextual embedding space. In: ACL (2021)
Rajaee, S., Pilehvar, M.T.: How does fine-tuning affect the geometry of embedding space: a case study on isotropy. In: EMNLP (2021)
Rudman, W., Gillman, N., Rayne, T., Eickhoff, C.: IsoScore: Measuring the uniformity of embedding space utilization. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 3325–3339. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.findings-acl.262, http://aclanthology.org/2022.findings-acl.262
Su, J., Cao, J., Liu, W., Ou, Y.: Whitening sentence representations for better semantics and faster retrieval (2021). https://doi.org/10.48550/ARXIV.2103.15316, http://arxiv.org/abs/2103.15316
Timkey, W., van Schijndel, M.: All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4527–4546. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.372, http://aclanthology.org/2021.emnlp-main.372
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: A multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/W18-5446, http://aclanthology.org/W18-5446
Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., Gu, Q.: Improving neural language generation with spectrum control. In: International Conference on Learning Representations (2020). http://openreview.net/forum?id=ByxY8CNtvr
Wolf, T., et al.: Huggingface’s transformers: State-of-the-art natural language processing (2019). https://doi.org/10.48550/ARXIV.1910.03771
Zabihzadeh, D.: Ensemble of loss functions to improve generalizability of deep metric learning methods. arXiv:2107.01130 (2021)
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Predicting the Type and Target of Offensive Posts in Social Media. In: Proceedings of NAACL (2019)
Zhang, Z., Gao, C., Xu, C., Miao, R., Yang, Q., Shao, J.: Revisiting representation degeneration problem in language modeling. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 518–527. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.findings-emnlp.46, http://aclanthology.org/2020.findings-emnlp.46
Zhou, W., Lin, B.Y., Ren, X.: Isobn: Fine-tuning bert with isotropic batch normalization. Proc. AAAI Conf. Artif. Intell. 35(16), 14621–14629 (May 2021), http://ojs.aaai.org/index.php/AAAI/article/view/17718
Acknowledgments
This work was supported by the Terminal Cloud Service Competence Center of Huawei Technologies Oy. in Helsinki, Finland in the context of the master thesis of the first author.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Attieh, J., Woubie Zewoudie, A., Vlassov, V., Flanagan, A., Bäckström, T. (2023). Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14191. Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-41734-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41733-7
Online ISBN: 978-3-031-41734-4
eBook Packages: Computer ScienceComputer Science (R0)
