Abstract
High performance computing (HPC) plays an essential role in enabling first-principles calculations based on the Kohn–Sham density functional theory (KS-DFT) for investigating quantum structural and electronic properties of large-scale molecules and solids in condensed matter physics, quantum chemistry and materials science. This review focuses on recent advances for HPC software development in large-scale KS-DFT calculations containing tens of thousands of atoms on modern heterogeneous supercomputers, especially for the HPC software with independent intellectual property rights supported on the Chinese domestic exascale supercomputers. We first introduce three various types of DFT software developed on modern heterogeneous supercomputers, involving PWDFT (Plane-Wave Density Functional Theory), HONPAS (Hefei Order-N Packages for Ab initio Simulations) and DGDFT (Discontinuous Galerkin Density Functional Theory), respectively based on three different types of basis sets (plane waves, numerical atomic orbitals and adaptive local basis functions). Then, we describe the theoretical algorithms and parallel implementation of these three software on modern heterogeneous supercomputers in detail. Finally, we conclude this review and propose several promising research fields for future large-scale KS-DFT calculations towards exascale supercomputers.
Similar content being viewed by others
References
Adamo, C., Barone, V.: Toward reliable density functional methods without adjustable parameters: the PBE0 model. J. Chem. Phys. 110(13), 6158–6170 (1999)
Apra, E., Bylaska, E.J., De Jong, W.A., Govind, N., Kowalski, K., Straatsma, T.P., Valiev, M., van Dam, H.J., Alexeev, Y., Anchell, J., et al.: Nwchem: Past, present, and future. J. Chem. Phys. 152(18), 184102 (2020)
Arita, M., Arapan, S., Bowler, D.R., Miyazaki, T.: Large-scale dft simulations with a linear-scaling dft code conquest on k-computer. J. Adv. Simulat. Sci. Eng. 1(1), 87–97 (2014)
Asadchev, A., Gordon, M.S.: Mixed-precision evaluation of two-electron integrals by rys quadrature. Comput. Phys. Commun. 183(8), 1563–1567 (2012)
Banerjee, A.S., Lin, L., Hu, W., et al.: Chebyshev polynomial filtered subspace iteration in the discontinuous galerkin method for large-scale electronic structure calculations. J. Chem. Phys. 145, 154101 (2016)
Banerjee, A.S., Lin, L., Suryanarayana, P., Yang, C., Pask, J.E.: Two-level chebyshev filter based complementary subspace method: Pushing the envelope of large-scale electronic structure calculations. J. Chem. Theory Comput. 14(6), 2930–2946 (2018)
Becke, A.D.: Density-functional exchange-energy approximation with correct asymptotic behavior. Phys. Rev. A 38(6), 3098 (1988)
Blum, V., Gehrke, R., Hanke, F., Havu, P., Havu, V., Ren, X., Reuter, K., Scheffler, M.: Ab initio molecular simulations with numeric atom-centered orbitals. Comput. Phys. Commun. 180(11), 2175–2196 (2009)
Bottin, F., Leroux, S., Knyazev, A., Zérah, G.: Large-scale ab initio calculations based on three levels of parallelization. Comp. Mater. Sci. 42(2), 329–336 (2008)
Bowler, D., Choudhury, R., Gillan, M., Miyazaki, T.: Recent progress with large-scale ab initio calculations: the conquest code. Phys. Status. Solidi. B. 243(5), 989–1000 (2006)
Bowler, D.R., Miyazaki, T.: Calculations for millions of atoms with density functional theory: linear scaling shows its potential. J. Phys. 22(7), 074207 (2010)
Bowler, D.R., Miyazaki, T.: \(\cal{O}\)(n) methods in electronic structure calculations. Rep. Prog. Phys 75(3), 036503 (2012)
Buluç, A., Gilbert, J.R.: Parallel sparse matrix-matrix multiplication and indexing: Implementation and experiments. SIAM. J. Sci. Comput. 34(4), 170–191 (2012)
Cao, Y., Romero, J., Olson, J.P., Degroote, M., Johnson, P.D., Kieferova, M., Kivlichan, I.D., Menke, T., Peropadre, B., Sawaya, N.P.D., Sim, S., Veis, L., Aspuru-Guzik, A.: Quantum chemistry in the age of quantum computing. Chem. Rev. 119, 10856–10915 (2019)
Ceperley, D.M., Alder, B.J.: Ground state of the electron gas by a stochastic method. Phys. Rev. Lett. 45(7), 566 (1980)
Das, S., Motamarri, P., Gavini, V., Turcksin, B., Li, Y.W., Leback, B.: Fast, scalable and accurate finite-element based ab initio calculations using mixed precision computing: 46 pflops simulation of a metallic dislocation system. Association for Computing Machinery, New York, NY, USA (2019)
Davidson, E.R.: The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices. J. Comput. Phys. 17, 87–94 (1975)
Ehrenreich, H., Cohen, M.H.: Self-consistent field approach to the many-electron problem. Phys. Rev. 115(4), 786 (1959)
Ernzerhof, M., Scuseria, G.E.: Assessment of the perdew–burke–ernzerhof exchange-correlation functional. J. Chem. Phys. 110(11), 5029–5036 (1999)
Fu, H., Liao, J., Yang, J., Wang, L., Song, Z., Huang, X., Yang, C., Xue, W., Liu, F., Qiao, F., et al.: The sunway taihulight supercomputer: system and applications. Sci. China Inf. Sci. 59(7), 1–16 (2016)
Giannozzi, P., Andreussi, O., Brumme, T., Bunau, O., Nardelli, M.B., Calandra, M., Car, R., Cavazzoni, C., Ceresoli, D., Cococcioni, M., et al.: Advanced capabilities for materials modelling with quantum espresso. J. Phys. 29(46), 465901 (2017)
Giannozzi, P., Baroni, S., Bonini, N., et al.: QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. 21(39), 395502 (2009)
Goedecker, S.: Linear scaling electronic structure methods. Rev. Mod. Phys. 71, 1085–1123 (1999)
Gonze, X., Amadon, B., Anglade, P.-M., Beuken, J.-M., Bottin, F., Boulanger, P., Bruneval, F., Caliste, D., Caracas, R., Côté, M., Deutsch, T., Genovese, L., Ghosez, P., Giantomassi, M., Goedecker, S., Hamann, D.R., Hermet, P., Jollet, F., Jomard, G., Leroux, S., Mancini, M., Mazevet, S., Oliveira, M.J.T., Onida, G., Pouillon, Y., Rangel, T., Rignanese, G.-M., Sangalli, D., Shaltaf, R., Torrent, M., Verstraete, M.J., Zerah, G., Zwanziger, J.W.: Abinit: First-principles approach to material and nanosystem properties. Comput. Phys. Commun. 180(12), 2582–2615 (2009)
Gygi, F., Draeger, E.W., Schulz, M., de Supinski, B.R., Gunnels, J.A., Austel, V., Sexton, J.C., Franchetti, F., Kral, S., Ueberhuber, C.W., Lorenz, J.: Large-scale electronic structure calculations of high-z metals on the BlueGene/L platform. Association for Computing Machinery, New York, NY, USA (2006)
Hasegawa, Y., Iwata, J.-I., Tsuji, M., Takahashi, D., Oshiyama, A., Minami, K., Boku, T., Shoji, F., Uno, A., Kurokawa, M., Inoue, H., Miyoshi, I., Yokokawa, M.: First-principles calculations of electron states of a silicon nanowire with 100,000 atoms on the k computer. Association for Computing Machinery, New York, NY, USA (2011)
Heyd, J., Scuseria, G.E., Ernzerhof, M.: Hybrid functionals based on a screened coulomb potential. J. Chem. Phys. 118(18), 8207–8215 (2003)
Hohenberg, P., Kohn, W.: Inhomogeneous electron gas. Phys. Rev. 136, 864–871 (1964)
Hu, W., Lin, L., Banerjee, A.S., et al.: Adaptively compressed exchange operator for large-scale hybrid density functional calculations with applications to the adsorption of water on silicene. J. Chem. Theory Comput. 13(3), 1188–1198 (2017)
Hu, W., Lin, L., Yang, C.: DGDFT: A massively parallel method for large scale density functional theory calculations. J. Chem. Phys. 143(12), 124110 (2015)
Hu, W., Lin, L., Yang, C.: Edge reconstruction in armchair phosphorene nanoribbons revealed by discontinuous galerkin density functional theory. Phys. Chem. Chem. Phys. 17(47), 31397–31404 (2015)
Hu, W., Lin, L., Yang, C.: Interpolative separable density fitting decomposition for accelerating hybrid density functional calculations with applications to defects in silicon. J. Chem. Theory Comput. 13(11), 5420–5431 (2017)
Hu, W., Lin, L., Yang, C.: Projected commutator diis method for accelerating hybrid functional electronic structure calculations. J. Chem. Theory Comput. 13, 5458–5467 (2017)
Hu, W., Qin, X., Jiang, Q., Chen, J., An, H., Jia, W., Li, F., Liu, X., Chen, D., Liu, F., et al.: High performance computing of DGDFT for tens of thousands of atoms using millions of cores on Sunway TaihuLight. Sci. Bull. 66(2), 111–119 (2021)
Igram, D., Bhattarai, B., Biswas, P., Drabold, D.A.: Large and realistic models of amorphous silicon. J. Non. Cryst. Solids. 492, 27–32 (2018)
Jia, W., Cao, Z., Wang, L., Fu, J., Chi, X., Gao, W., Wang, L.-W.: The analysis of a plane wave pseudopotential density functional theory code on a gpu machine. Comput. Phys. Commun. 184(1), 9–18 (2013)
Knyazev, A.V.: Toward the optimal preconditioned eigensolver: locally optimal block preconditioned conjugate gradient method. SIAM J Sci. Comput. 23(2), 517–541 (2001)
Kohn, W.: Density functional and density matrix method scaling linearly with the number of atoms. Phys. Rev. Lett. 76, 3168–3171 (1996)
Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140(4A), 1133 (1965)
Kresse, G., Hafner, J.: Ab initio molecular dynamics for liquid metals. Phys Rev B 47, 558 (1993)
Kühne, T.D., Iannuzzi, M., Del Ben, M., Rybkin, V.V., Seewald, P., Stein, F., Laino, T., Khaliullin, R.Z., Schütt, O., Schiffmann, F., et al.: CP2K: An electronic structure and molecular dynamics software package-quickstep: Efficient and accurate electronic structure calculations. J. Chem. Phys. 152(19), 194103 (2020)
Lanczos, C.: Applied Analysis. Dover, New York (1988)
Li, X.S., Demmel, J.W.: Superlu_dist: a scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Trans. Math.AD Softw. (TOMS) 29(2), 110–140 (2003)
Li, P., Liu, X., Chen, M., Lin, P., Ren, X., Lin, L., Yang, C., He, L.: Large-scale ab initio simulations based on systematically improvable atomic basis. Comput. Mater. Sci. 112, 503–517 (2016)
Lin, L.: Adaptively compressed exchange operator. J. Chem. Theory Comput. 12(5), 2242–2249 (2016)
Lin, L., Chen, M., Yang, C., He, L.: Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion. J. Phys. 25(29), 295501 (2013)
Lin, L., García, A., Huhs, G., et al.: SIESTA-PEXSI: Massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization. J. Phys.A 26, 305503 (2014)
Lin, L., García, A., Huhs, G., Yang, C.: SIESTA-PEXSI: massively parallel method for efficient and accurateab initiomaterials simulation without matrix diagonalization. J. Phys. 26(30), 305503 (2014)
Lin, L., Lu, J., Ying, L.E.W.: Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation. J. Comput. Phys 231(4), 2140–2154 (2012)
Lin, L., Lu, J., Ying, L., Car, R.E.W.: Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems. Comm. Math. Sci 7(3), 755–777 (2009)
Lin, P., Ren, X., He, L.: Efficient hybrid density functional calculations for large periodic systems using numerical atomic orbitals. J. Chem. Theory Comput. 17(1), 222–239 (2021)
Lucignano, P., Alfè, D., Cataudella, V., Ninno, D., Cantele, G.: Crucial role of atomic corrugation on the flat bands and energy gaps of twisted bilayer graphene at the magic angle \(\theta \sim 1.{08}^{\circ }\). Phys. Rev. B 99(19), 195419 (2019)
Ma, H., Wang, L., Wan, L., Li, J., Qin, X., Liu, J., Hu, W., Lin, L., Yang, C., Yang, J.: Realizing effective cubic-scaling coulomb hole plus screened exchange approximation in periodic systems via interpolative separable density fitting with a plane-wave basis set. J. Phys. Chem. A 125(34), 7545–7557 (2021)
Paier, J., Ren, X., Rinke, P., Scuseria, G.E., Grüneis, A., Kresse, G., Scheffler, M.: Assessment of correlation energies based on the random-phase approximation. New J. Phys. 14(4), 043002 (2012)
Perdew, J.P., Burke, K., Ernzerhof, M.: Generalized gradient approximation made simple. Phys. Rev. Lett. 77(18), 3865 (1996)
Perdew, J.P., Schmidt, K.: Jacob’s ladder of density functional approximations for the exchange-correlation energy. AIP Conf. Proc. 577(1), 1–20 (2001)
Qin, X., Shang, H., Xiang, H., Li, Z., Yang, J.: HONPAS: A linear scaling open-source solution for large system simulations. Int. J. Quantum. Chem. 115(10), 647–655 (2015)
Ratcliff, L.E., Dawson, W., Fisicaro, G., Caliste, D., Mohr, S., Degomme, A., Videau, B., Cristiglio, V., Stella, M., D’Alessandro, M., et al.: Flexibilities of wavelets as a computational basis set for large-scale electronic structure calculations. J. Chem. Phys. 152(19), 194110 (2020)
Shang, H., Li, Z., Yang, J.: Implementation of screened hybrid density functional for periodic systems with numerical atomic orbitals: Basis function fitting and integral screening. J. Chem. Phys. 135(3), 034110 (2011)
Shang, H., Li, F., Zhang, Y., Zhang, L., Fu, Y., Gao, Y., Wu, Y., Duan, X., Lin, R., Liu,X., Liu, Y., Chen, D.: Extreme-scale ab initio quantum raman spectra simulations on the leadership HPC system in China. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.
Shang, H., Xu, L., Wu, B., Qin, X., Zhang, Y., Yang, J.: The dynamic parallel distribution algorithm for hybrid density-functional calculations in HONPAS package. Comput. Phys. Commun. 254, 107204 (2020)
Soler, J.M., Artacho, E., Gale, J.D., García, A., Junquera, J., Ordejón, P., Sánchez-Portal, D.: The SIESTA method for ab initio order-N materials simulation. J. Phys. Condens. Matter. 14(11), 2745 (2002)
Stephens, P.J., Devlin, F.J., Chabalowski, C.F., Frisch, M.J.: Ab initio calculation of vibrational absorption and circular dichroism spectra using density functional force fields. J. Phys. Chem. 98(45), 11623–11627 (1994)
Strohmaier, E., Dongarra, J., Simon, H., Meuer, M., Meuer, H.: Top500 list. https://www.top500.org (2021)
Sun, J., Ruzsinszky, A., Perdew, J.P.: Strongly constrained and appropriately normed semilocal density functional. Phys. Rev. Lett. 115(3), 036402 (2015)
Tao, J., Perdew, J.P., Staroverov, V.N., Scuseria, G.E.: Climbing the density functional ladder: Nonempirical meta-generalized gradient approximation designed for molecules and solids. Phys. Rev. Lett. 91(14), 146401 (2003)
Tian, Y., Suo, B., Ma, Y., Jin, Z.: Optimizing two-electron repulsion integral calculations with McMurchie-Davidson method on graphic processing unit. J. Chem. Phys. 155(3), 034112 (2021)
Tsuchida, E., Choe, Y.-K.: Iterative diagonalization of symmetric matrices in mixed precision and its application to electronic structure calculations. Comput. Phys. Commun. 183(4), 980–985 (2012)
Valeev, E.F.: Libint: A library for the evaluation of molecular integrals of many-body operators over Gaussian functions. http://libint.valeyev.net/. version 2.7.1 (2021)
VandeVondele, J., Borštnik, U., Hutter, J.: Linear scaling self-consistent field calculations with millions of atoms in the condensed phase. J. Chem. Theory Comput. 8(10), 3565–3573 (2012)
Vecharynski, E., Yang, C., Pask, J.E.: A projected preconditioned conjugate gradient algorithm for computing many extreme eigenpairs of a Hermitian matrix. J. Comput. Phys. 290(1), 73–89 (2015)
Vinson, J.: Faster exact exchange in periodic systems using single-precision arithmetic. J. Chem. Phys. 153(20), 204106 (2020)
Wan, L., Liu, X., Liu, J., Qin, X., Hu, W., Yang, J.: Hybrid mpi and openmp parallel implementation of large-scale linear-response time-dependent density functional theory with plane-wave basis set. Electron. Struct. 3, 024004 (2021)
Wang, L.: Divide-and-conquer quantum mechanical material simulations with exascale supercomputers. Natl. Sci. Rev. 1(4), 604–617 (2014)
Wang, L.-W., Lee, B., Shan, H., Zhao, Z., Meza, J., Strohmaier, E., Bailey, D.H.: Linearly scaling 3D fragment method for large-scale electronic structure calculations. IEEE Press, New York, NY, USA (2008)
Yang, W.: Direct calculation of electron density in density-functional theory. Phys. Rev. Lett. 66, 1438–1441 (1991)
Zhang, G., Lin, L., Hu, W., et al.: Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework II: Force, vibration, and molecular dynamics calculations. J. Comput. Phys. 335, 426–443 (2017)
Zhou, Y., Saad, Y., Tiago, M.L., et al.: Self-consistent-field calculations using chebyshev-filtered subspace iteration. J. Comput. Phys. 219, 172–184 (2006)
Acknowledgements
This work is partly supported by the National Natural Science Foundation of China (21688102, 21803066, 22003061, 22173093), by the Hefei National Laboratory for Physical Sciences at the Microscale (KF2020003), the Chinese Academy of Sciences Pioneer Hundred Talents Program (KJ2340000031), the Anhui Initiative in Quantum Information Technologies (AHY090400), the CAS Project for Young Scientists in Basic Research (YSBR-005), the Strategic Priority Research Program of Chinese Academy of Sciences (XDC01040100), the Fundamental Research Funds for the Central Universities (WK2340000091, WK2060000018), the Hefei National Laboratory for Physical Sciences at the Microscale (SK2340002001), the Research Start-Up Grants (KY2340000094) and the Academic Leading Talents Training Program (KY2340000103) from University of Science and Technology of China. The authors thank the Hefei Advanced Computing Center, the Supercomputing Center of Chinese Academy of Sciences (SunRising-1), the Supercomputing Center of USTC, the National Supercomputing Center in Wuxi, Tianjin, Shanghai, and Guangzhou for the computational resources.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Qin, X., Chen, J., Luo, Z. et al. High performance computing for first-principles Kohn-Sham density functional theory towards exascale supercomputers. CCF Trans. HPC 5, 26–42 (2023). https://doi.org/10.1007/s42514-022-00120-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42514-022-00120-0