Abstract
Hyperparameters (HPs) play a central role in the performance of machine learning (ML) models, governing model structure, regularization, and convergence properties. Understanding the intricate relationship between HP configurations and model performance is essential for ML practitioners, especially those with limited expertise, to develop effective models that produce satisfactory results. This paper introduces HyperParameter Explorer (HPExplorer), a semi-automated eXplainable AI (XAI) method, to support ML practitioners to explore this relationship. HPExplorer integrates an automated HP discovery algorithm with an interactive visual exploration component. The HP discovery algorithm identifies performance-consistent subspaces within the HP space, where models perform similarly despite minor variations in HP configurations. The interactive visual exploration component enables users to explore the discovered performance-consistent subspaces using an interactive 2-D projection called Star Coordinate. Users can also compare HP configurations from different subspaces to explore their impact on model performance. We developed HPExplorer in close collaboration with ML practitioners, particularly geoscientists, using ML in their research. Initial feedback from scientists using HPExplorer in real-world scenarios indicates that HPExploer enhances the transparency in configuring HPs and increases the confidence of users in their decisions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(10), 281–305 (2012)
Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, vol. 25, pp. 1–2. Curran Associates, Inc., Red Hook (2012)
Young, S.R., Rose, D.C., Karnowski, T.P., Lim, S.-H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, pp. 1–5. ACM, New York (2015)
Dhanorkar, S., Wolf, C.T., Qian, K., Xu, A., Popa, L., Li, Y.: Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In: Proceedings of the Conference, pp. 1591–1602. ACM, New York (2021)
Moosbauer, J., Herbinger, J., Casalicchio, G., Lindauer, M., Bischl, B.: Towards Explaining Hyperparameter Optimization via Partial Dependence Plots. In: Proceedings of the international workshop on Automated Machine Learning (AutoML) at ICML’21, pp. 1–2. Publisher, Location (2021)
Park, H., Nam, Y., Kim, J., Choo, J.: HyperTendril: visual analytics for user-driven hyperparameter optimization of deep neural networks. IEEE Trans. Visual Comput. Graphics 27(2), 1407–1416 (2021)
Park, H., et al.: VisualHyperTuner: visual analytics for user-driven hyperparameter tuning of deep neural networks. In: Proceedings of the Conference, pp. 1–2. Publisher, Location (2019)
Chatzimparmpas, A., Martins, R.M., Kucher, K., Kerren, A.: VisEvol: visual analytics to support hyperparameter search through evolutionary optimization. Comput. Graph. Forum 40(3), 69–91 (2021)
Chakraborty, T., Wirth, C., Seifert, C.: Post-hoc rule based explanations for black box bayesian optimization. In: Editors (eds.) Book Title, pp. 320-337. Springer, Location (2024)
Zöller, M.-A., Titov, W., Schlegel, T., Huber, M.F.: XAutoML: a visual analytics tool for understanding and validating automated machine learning. ACM Trans. Interactive Intell. Syst. 13(4), 1–39 (2023)
Kandogan, E.: Visualizing multi-dimensional clusters, trends, and outliers using Star Coordinates. In: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 107–116. ACM, New York (2001)
Hoffman, P., Grinstein, G., Marx, K., Grosse, I., Stanley, H.: DNA visual and analytic data mining. In: Proceedings of the Conference, pp. 437–442. Publisher, Location (1997)
Inselberg, A.: The plane with parallel coordinates. Vis. Comput. 1, 69–91 (1985)
Tufte, E.R.: The Visual Display of Quantitative Information. Graphics Press, USA (1986)
Etemadpour, R., Linsen, L., Paiva, J.G., Crick, C., Forbes, A.: Choosing Visualization Techniques for Multidimensional Data Projection Tasks: A Guideline with Examples. In: Editors (eds.) Book Title, vol. 598, pp. 166–186. Springer, Location (2016)
Dzemyda, G., Kurasova, O., Zilinskas, J.: Multidimensional data visualization. Methods and applications series: Springer optimization and its applications 75(122), 10–5555 (2013)
Asimov, D.: The grand tour: a tool for viewing multidimensional data. SIAM J. Sci. Stat. Comput. 6(1), 128–143 (1985)
Renardy, M., Joslyn, L.R., Millar, J.A., Kirschner, D.E.: To sobol or not to sobol? the effects of sampling schemes in systems biology applications. Math. Biosci. 337, 108593 (2021)
Burhenne, S., Jacob, D., Henze, G.P.: Sampling based on Sobol’ sequences for Monte Carlo techniques applied to building simulations. In: Proceedings of the Conference, pp. 1816–1823. Publisher, Location (2011)
HILDA ’23: Proceedings of the Workshop on Human-In-the-Loop Data Analytics. ACM, New York, NY, USA (2023)
Inselberg, A.: Parallel Coordinates: Visual Multidimensional Geometry and Its Applications. Springer, Heidelberg (2017)
Decker, T., Gross, R., Koebler, A., Lebacher, M., Schnitzer, R., Weber, S.H.: The thousand faces of explainable AI along the machine learning life cycle: industrial reality and current state of research. In: AI-HCI 2023: Artificial Intelligence in HCI, pp. 184-208. Springer, Heidelberg (2023)
Weidele, D.K.I., Weisz, J.D., Oduor, E., Muller, M., Andres, J., Gray, A., Wang, D.: AutoAIViz: opening the blackbox of automated artificial intelligence with conditional parallel coordinates. In: Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 308–312. ACM, New York (2020)
Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: 31st International Conference on Machine Learning, ICML 2014, pp. 1130–1144 (2014)
Sheikholeslami, S., Meister, M., Wang, T., Payberah, A. H., Vlassov, V., Dowling, J.: AutoAblation: automated parallel ablation studies for deep learning. In: Proceedings of the 1st Workshop on Machine Learning and Systems, EuroMLSys ’21, pp. 55-61. ACM, New York (2021)
Bansal, N., Agarwal, C., Nguyen, A.: SAM: the sensitivity of attribution methods to hyperparameters. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8670–8680. IEEE Computer Society, Los Alamitos (2020)
Mishra, S., Dutta, S., Long, J., Magazzeni, D.: A survey on the robustness of feature importance and counterfactual explanations. arXiv preprint arXiv:2111.00358 (2021)
Hohman, F., Kahng, M., Pienta, R., Chau, D.H.: Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. IEEE Trans. Visual Comput. Graphics 25(8), 2674–2693 (2019)
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. German National Research Center for Information Technology GMD, vol. 148, no. 34, pp. 13, Bonn, Germany (2001)
Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11), 2531–2560 (2002)
Sánchez-Benítez, A., Gößling, H., Pithan, F., Semmler, T., Jung, T.: The July 2019 european heat wave in a warmer climate: storyline scenarios with a coupled model using spectral nudging. J. Clim. 35(8), 2373–2390 (2022)
Montgomery, D.C.: Design and analysis of experiments. John Wiley & Sons, Location (2017)
Acknowledgments
We would like to express our gratitude to Daniel Eggert and Peter Morstein for their assistance in implementing HPExplorer. Aviad Etzion helps us in integrating HPExplorer into the real-world scenarios.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Grushetskaya, Y., Sips, M., Schachtschneider, R., Saberioon, M., Mahan, A. (2024). HPExplorer: XAI Method to Explore the Relationship Between Hyperparameters and Model Performance. In: Bifet, A., Krilavičius, T., Miliou, I., Nowaczyk, S. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14949. Springer, Cham. https://doi.org/10.1007/978-3-031-70378-2_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-70378-2_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70377-5
Online ISBN: 978-3-031-70378-2
eBook Packages: Computer ScienceComputer Science (R0)
