Abstract
Knowledge distillation, a popular model compression method, transfers knowledge from a large teacher model to a smaller student model. Self-distillation takes this a step further by having the model itself act as both teacher and student. However, existing self-distillation methods often focus on individual instance knowledge, such as logits and intermediate features, but overlook the structural information within each category’s representation. To address this gap, we propose Self-Distillation via Intra-Class Compactness (SDICC). Specifically, in SDICC, we use previous epoch models as teachers to guide training in the current epoch, while also emphasizing intra-class compactness as an additional training objective. This facilitates our model’s learning process in bringing intra-class features closer together, thereby promoting more discriminative representations across different categories. Moreover, to better combine both the knowledge from logits and the compactness of features, we adaptively perform self-distillation for progressive knowledge transfer. We extensively evaluate SDICC on popular image classification datasets like CIFAR-100 and Tiny ImageNet. Our results demonstrate that SDICC outperforms recent state-of-the-art self-distillation methods, showcasing its effectiveness in knowledge transfer and model compression.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bao, Z., Huang, Z., Gou, J., Du, L., Liu, K., Zhou, J., Chen, Y.: Teacher-student complementary sample contrastive distillation. Neural Netw. 170, 176–189 (2024)
Gou, J., Sun, L., Yu, B., Du, L., Ramamohanarao, K., Tao, D.: Collaborative knowledge distillation via multiknowledge transfer. IEEE Trans. Neural Netw. Learn. Syst. 35(5), 6718–6730 (2024)
Gou, J., Xiong, X., Yu, B., Du, L., Zhan, Y., Tao, D.: Multi-target knowledge distillation via student self-reflection. Int. J. Comput. Vision 131(7), 1857–1874 (2023)
Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vision 129, 1789–1819 (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). arXiv:1503.02531
Hou, Y., Ma, Z., Liu, C., Loy, C.C.: Learning lightweight lane detection cnns by self attention distillation. In: Proceedings of the IEEE/CVF conference on International Conference on Computer Vision (ICCV), pp. 1013–1021. IEEE (2019)
Kim, K., Ji, B., Yoon, D., Hwang, S.: Self-knowledge distillation with progressive refinement of targets. In: Proceedings of the IEEE/CVF conference on International Conference on Computer Vision (ICCV), pp. 6547–6556. IEEE (2021)
Li, C., Cheng, G., Han, J.: Boosting knowledge distillation via intra-class logit distribution smoothing. IEEE Trans. Circuits Syst. Video Technol. (2023)
Li, J., Zhou, P., Xiong, C., Hoi, S.C.: Prototypical contrastive learning of unsupervised representations (2020). arXiv:2005.04966
Li, L.: Self-regulated feature learning via teacher-free feature distillation. In: European Conference on Computer Vision, pp. 347–363. Springer (2022)
Liang, J., Li, L., Bing, Z., Zhao, B., Tang, Y., Lin, B., Fan, H.: Efficient one pass self-distillation with zipf’s label smoothing. In: Proceedings of the European Conference on Computer Vision (ECCV), vol. 13671, pp. 104–119. Springer (2022)
Lu, Y., Zhang, G., Sun, S., Guo, H., Yu, Y.: \( f \)-micl: Understanding and generalizing infonce-based contrastive learning (2024). arXiv:2402.10150
Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: practical guidelines for efficient cnn architecture design. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 116–131 (2018)
Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(11) (2008)
Mnih, A., Teh, Y.W.: A fast and simple algorithm for training neural probabilistic language models (2012). arXiv:1206.6426
Oord, A.V.D., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding (2018). arXiv:1807.03748
Sharma, S., Lodhi, S.S., Chandra, J.: SCL-IKD: intermediate knowledge distillation via supervised contrastive representation learning. Appl. Intell. 53(23), 28520–28541 (2023)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556
Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation (2019). arXiv:1910.10699
Wang, S., Yan, Z., Zhang, D., Wei, H., Li, Z., Li, R.: Prototype knowledge distillation for medical segmentation with missing modality. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE (2023)
Yang, C., An, Z., Zhou, H., Zhuang, F., Xu, Y., Zhang, Q.: Online knowledge distillation via mutual contrastive learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2023)
Yang, Z., Zeng, A., Li, Z., Zhang, T., Yuan, C., Li, Y.: From knowledge distillation to self-knowledge distillation: a unified approach with normalized loss and customized soft labels. In: Proceedings of the IEEE/CVF conference on International Conference on Computer Vision (ICCV), pp. 17139–17148. IEEE (2023)
Yun, S., Park, J., Lee, K., Shin, J.: Regularizing class-wise predictions via self-knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13873–13882. Computer Vision Foundation/IEEE (2020)
Yun, S., Park, J., Lee, K., Shin, J.: Revisiting knowledge distillation via label smoothing regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902–3910. Computer Vision Foundation/IEEE (2020)
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 1–15 (2016)
Zhang, H., Chen, D., Wang, C.: Confidence-aware multi-teacher knowledge distillation. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4498–4502. IEEE (2022)
Zhang, J., Tao, Z., Guo, K., Li, H., Zhang, S.: Hybrid mix-up contrastive knowledge distillation. Inf. Sci. 660, 120107 (2024)
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., Ma, K.: Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In: Proceedings of the IEEE/CVF Conference on International Conference on Computer Vision (ICCV), pp. 3712–3721. IEEE (2019)
Acknowledgement
This work was supported in part by National Natural Science Foundation of China (Grant Nos. 61976107, 61962010 and 62262005), the High-level Innovative Talents in Guizhou Province (No. GCC[2023]033), and the High Performance Computing (HPC) clusters at Southwest University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lin, J., Li, L., Yu, B., Ou, W., Gou, J. (2025). Self-Distillation via Intra-Class Compactness. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2024. Lecture Notes in Computer Science, vol 15031. Springer, Singapore. https://doi.org/10.1007/978-981-97-8487-5_10
Download citation
DOI: https://doi.org/10.1007/978-981-97-8487-5_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-8486-8
Online ISBN: 978-981-97-8487-5
eBook Packages: Computer ScienceComputer Science (R0)