Abstract
Although a plethora of architectural variants for deep classification has been introduced over time, recent works have found empirical evidence towards similarities in their training process. It haswrapfig been hypothesized that neural networks converge not only to similar representations, but also exhibit a notion of empirical agreement on which data instances are learned first. Following in the latter works’ footsteps, we define a metric to quantify the relationship between such classification agreement over time, and posit that the agreement phenomenon can be mapped to core statistics of the investigated dataset. We empirically corroborate this hypothesis across the CIFAR10, Pascal, ImageNet and KTH-TIPS2 datasets. Our findings indicate that agreement seems to be independent of specific architectures, training hyper-parameters or labels, albeit follows an ordering according to image statistics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahmed, N., Natarajan, T., Rao, K.R.: Discrete cosine transform. IEEE Trans. Comput. C-24(1), 90–93 (1974)
Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 34(11), 2189–2202 (2012)
Arpit, D., et al.: A closer look at memorization in deep networks. In: International Conference on Machine Learning (ICML) (2017)
Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: International Conference on Machine Learning (ICML) (2009)
Berg, A.C., et al.: Understanding and predicting importance in images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)
Byrt, T., Bishop, J., Carlin, J.B.: Bias, prevalence and kappa. J. Clin. Epidemiol. 46(5), 423–429 (1993)
Caelen, O.: A Bayesian interpretation of the confusion matrix. Ann. Math. Artif. Intell. 81(3–4), 429–450 (2017)
Caputo, B., Hayman, E., Mallikarjuna, P.: Class-specific material categorisation. In: International Conference on Computer Vision (ICCV) (2005)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
Everingham, M., Eslami, S.M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vision 111(1), 98–136 (2015)
Everingham, M., Van-Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)
Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient Graph-Based Image Segmentation. Int. J. Comput. Vision 59, 167–181 (2004)
Frieden, B.R.: Restoring with maximum likelihood and maximum entropy. J. Optical Soc. America (JOSA) 62(4), 511–518 (1972)
Geirhos, R., Michaelis, C., Wichmann, F.A., Rubisch, P., Bethge, M., Brendel, W.: Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations (ICLR) (2019)
Hacohen, G., Choshen, L., Weinshall, D.: Let’s agree to agree: neural networks share classification order on real datasets. In: International Conference on Learning Representations (ICLR) (2020)
Hacohen, G., Weinshall, D.: On the power of curriculum learning in training deep networks. In: International Conference on Machine Learning (ICML) (2019)
Hacohen, G., Weinshall, D.: Principal components bias in over-parameterizyed linear models, and its manifestation in deep neural networks. J. Mach. Learn. Res. 23, 1–46 (2022)
Hallgren, K.A.: Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 8(1), 23–34 (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Hermann, K.L., Chen, T., Kornblith, S.: The origins and prevalence of texture bias in convolutional neural networks. In: Neural Information Processing Systems (NeurIPS) 34 (2020)
Hoiem, D., Chodpathumwan, Y., Dai, Q.: Diagnosing error in object detectors. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 340–353. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_25
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017)
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: Neural Information Processing Systems (NeurIPS) (2019)
Ionescu, R.T., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D.P., Ferrari, V.: How hard can it be? estimating the difficulty of visual search in an image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2157–2166 (2016)
Isola, P., Xiao, J., Parikh, D., Torralba, A., Oliva, A.: What makes a photograph memorable? IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(7), 1469–1482 (2014)
Isola, P., Zoran, D., Krishnan, D., Adelson, E.H.: Crisp boundary detection using pointwise mutual information. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 799–814. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10578-9_52
Jiang, L., Meng, D., Zhao, Q., Shan, S., Hauptmann, A.G.: Self-paced curriculum learning. In: Proceedings of the National Conference on Artificial Intelligence, vol. 4, pp. 2694–2700 (2015)
Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Technical report Toronto (2009)
Kumar, M., Packer, B., Koller, D., Kumar, P., Packer, B., Koller, D.: Self-paced learning for latent variable models. In: Neural Information Processing Systems (NeurIPS) (2010)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2323 (1998)
Lee, Y.J., Grauman, K.: Learning the easy things first: Self-paced visual category discovery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1721–1728 (2011)
Li, Y., Yosinski, J., Clune, J., Lipson, H., Hopcroft, J.: Convergent Learning: do different neural networks learn the same representations? In: International Conference on Learning Representations (ICLR) (2016)
Liu, D., Xiong, Y., Pulli, K., Shapiro, L.: Estimating image segmentation difficulty. In: Perner, P. (ed.) MLDM 2011. LNCS (LNAI), vol. 6871, pp. 484–495. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23199-5_36
Maennel, H., et al.: What do neural networks learn when trained with random labels? In: Neural Information Processing Systems (NeurIPS) (2020)
Mangalam, K., Prabhu, V.: Do deep neural networks learn shallow learnable examples first? In: International Conference on Machine Learning (ICML), Deep Phenomena Workshop (2019)
Ortiz-Jiménez, G., Modas, A., Moosavi-Dezfooli, S.M., Frossard, P.: Hold me tight! Influence of discriminative features on deep network boundaries, Neural Information Processing Systems (NeurIPS) (2020)
Peterson, J.C., Battleday, R.M., Griffiths, T.L., Russakovsky, O.: Human uncertainty makes classification more robust. In: International Conference on Computer Vision (ICCV) (2019)
Pinto, N., Cox, D.D., DiCarlo, J.J.: Why is real-world visual object recognition hard? PLoS Comput. Biol. 4(1), 0151–0156 (2008)
Russakovsky, O., Deng, J., Huang, Z., Berg, A.C., Fei-Fei, L.: Detecting avocados to Zucchinis: what have we done, and where are we going? In: International Conference on Computer Vision (ICCV) (2013)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., Netrapalli, P.: The pitfalls of simplicity bias in neural networks. In: Neural Information Processing Systems (NeurIPS) (2020)
Shwartz-Ziv, R., Tishby, N.: Opening the Black Box of Deep Neural Networks via Information. Why & when Deep Learning works: looking inside Deep Learning ICRI-CI (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
Skilling, J., Bryan, R.: Maximum entropy image reconstruction: general algorithm. Mon. Not. R. Astron. Soc. 211, 111–124 (1984)
Spain, M., Perona, P.: Some objects are more equal than others: measuring and predicting importance. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 523–536. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88682-2_40
Vijayanarasimhan, S., Grauman, K.: What’s it going to cost you? Predicting effort vs. informativeness for multi-label image annotations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
Wang, L., Hu, L., Gu, J., Wu, Y., Hu, Z., He, K., Hopcroft, J.: Towards understanding learning representations: to what extent do different neural networks learn the same representation. In: Neural Information Processing Systems (NeurIPS) (2018)
Yang, K., Qinami, K., Fei-Fei, L., Deng, J., Russakovsky, O.: Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. In: Conference on Fairness, Accountability, and Transparency (FAT), pp. 547–558 (2020)
Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. In: International Conference on Learning Representations (ICLR) (2017)
Acknowledgements
This work was supported by the German Federal Ministry of Education and Research (BMBF) funded project 01IS19062 “AISEL" and the European Union’s Horizon 2020 project No. 769066 “RESIST".
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pliushch, I., Mundt, M., Lupp, N., Ramesh, V. (2022). When Deep Classifiers Agree: Analyzing Correlations Between Learning Order and Image Statistics. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13668. Springer, Cham. https://doi.org/10.1007/978-3-031-20074-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-20074-8_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20073-1
Online ISBN: 978-3-031-20074-8
eBook Packages: Computer ScienceComputer Science (R0)