Abstract
In this paper we consider finding an approximate second-order stationary point (SOSP) of general nonconvex conic optimization that minimizes a twice differentiable function subject to nonlinear equality constraints and also a convex conic constraint. In particular, we propose a Newton-conjugate gradient (Newton-CG) based barrier-augmented Lagrangian method for finding an approximate SOSP of this problem. Under some mild assumptions, we show that our method enjoys a total inner iteration complexity of \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-11/2})\) and an operation complexity of \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-11/2}\min \{n,\epsilon ^{-5/4}\})\) for finding an \((\epsilon ,\sqrt{\epsilon })\)-SOSP of general nonconvex conic optimization with high probability. Moreover, under a constraint qualification, these complexity bounds are improved to \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-7/2})\) and \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-7/2}\min \{n,\epsilon ^{-3/4}\})\), respectively. To the best of our knowledge, this is the first study on the complexity of finding an approximate SOSP of general nonconvex conic optimization. Preliminary numerical results are presented to demonstrate superiority of the proposed method over first-order methods in terms of solution quality.








Similar content being viewed by others
Data availibility
The codes for generating the random data and implementing the algorithms in the numerical section are available from the first author upon request.
Notes
In fact, a total inner iteration complexity of \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-7})\) and an operation complexity \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-7}\min \{n,\epsilon ^{-1}\})\) were established in [70] for finding an \((\epsilon ,\epsilon )\)-SOSP of problem (1) with high probability; see [70, Theorem 4(ii), Corollary 3(ii), Theorem 5] Nevertheless, they can be easily modified to obtain the aforementioned complexity for finding an \((\epsilon ,\sqrt{\epsilon })\)-SOSP of (1) with high probability.
The operation complexity of the barrier method [43] is measured by the amount of fundamental operations consisting of matrix–vector products, matrix multiplications, Cholesky factorizations, and backward or forward substitutions to a triangular linear system.
The arithmetic complexity of Cholesky factorizations for a positive definite matrix is \({{\,\mathrm{\mathcal {O}}\,}}(n^3)\) in general, while the arithmetic complexity of matrix–vector products and backward or forward substitutions is at most \({{\,\mathrm{\mathcal {O}}\,}}(n^2)\), where n is the number of rows of the matrix.
It shall be mentioned that the total numbers of Cholesky factorizations are only \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-7/2})\) and \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\epsilon ^{-11/2})\) respectively for the case where constraint qualification holds or not. See Subsections 5.3 and 5.4 for details.
The vectorization of a matrix is the column vector obtained by stacking the columns of the matrix on top of one another.
References
Agarwal, N., Allen-Zhu, Z., Bullins, B., Hazan, E., Ma, T.: Finding approximate local minima faster than gradient descent. In: Proceedings of the 49th annual ACM SIGACT symposium on theory of computing, pp. 1195–1199 (2017)
Allen-Zhu, Z., Li, Y.: Neon2: finding local minima via first-order oracles. Adv. Neural Info. Process. Syst. 31, 3716–3726 (2018)
Andreani, R., Birgin, E.G., Martínez, J.M., Schuverdt, M.L.: On augmented Lagrangian methods with general lower-level constraints. SIAM J. Optim. 18(4), 1286–1309 (2008)
Andreani, R., Haeser, G., Ramos, A., Silva, P.J.: A second-order sequential optimality condition associated to the convergence of optimization algorithms. IMA J. Numer. Anal. 37(4), 1902–1929 (2017)
Argáez, M., Tapia, R.: On the global convergence of a modified augmented Lagrangian linesearch interior-point Newton method for nonlinear programming. J. Optim. Theory Appl 114(1), 1–25 (2002)
Armand, P., Omheni, R.: A mixed logarithmic barrier-augmented Lagrangian method for nonlinear optimization. J. Optim. Theory Appl. 173(2), 523–547 (2017)
Armand, P., Tran, N.N.: Rapid infeasibility detection in a mixed logarithmic barrier-augmented Lagrangian method for nonlinear optimization. Optim. Methods Softw. 34(5), 991–1013 (2019)
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific. (1995)
Bhojanapalli, S., Neyshabur, B., Srebro, N.: Global optimality of local search for low rank matrix recovery. Adv. Neural Info. Process. Syst. 29, 3873–3881 (2016)
Bian, W., Chen, X., Ye, Y.: Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization. Math. Program. 149(1), 301–327 (2015)
Birgin, E.G., Gardenghi, J., Martínez, J.M., Santos, S.A., Toint, P.L.: Evaluation complexity for nonlinear constrained optimization using unscaled KKT conditions and high-order models. SIAM J. Optim. 26(2), 951–967 (2016)
Birgin, E.G., Haeser, G., Ramos, A.: Augmented Lagrangians with constrained subproblems and convergence to second-order stationary points. Comput. Optim. and Appl. 69(1), 51–75 (2018)
Birgin, E. G., Martínez, J.M.: Practical augmented Lagrangian methods for constrained optimization. SIAM (2014)
Birgin, E.G., Martínez, J.M.: The use of quadratic regularization with a cubic descent condition for unconstrained optimization. SIAM J. Optim. 27(2), 1049–1074 (2017)
Bonnans, J.F., Launay, G.: Sequential quadratic programming with penalization of the displacement. SIAM J. Optim. 5(4), 792–812 (1995)
Bueno, L.F., Martínez, J.M.: On the complexity of an inexact restoration method for constrained optimization. SIAM J. Optim. 30(1), 80–101 (2020)
Byrd, R.H., Schnabel, R.B., Shultz, G.A.: A trust region algorithm for nonlinearly constrained optimization. SIAM J. Numer. Anal. 24(5), 1152–1170 (1987)
Carmon, Y., Duchi, J.: Gradient descent finds the cubic-regularized nonconvex newton step. SIAM J. Optim. 29(3), 2146–2178 (2019)
Carmon,Y., Duchi, J.C., Hinder, O., Sidford, A.:“Convex until proven guilty": dimension-free acceleration of gradient descent on non-convex functions. In: International conference on machine learning, pp. 654–663. PMLR, (2017)
Carmon, Y., Duchi, J.C., Hinder, O., Sidford, A.: Accelerated methods for nonconvex optimization. SIAM J. Optim. 28(2), 1751–1772 (2018)
Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Math. Program. 127(2), 245–295 (2011)
Cartis, C., Gould, N.I., Toint, P.L.: On the evaluation complexity of cubic regularization methods for potentially rank-deficient nonlinear least-squares problems and its relevance to constrained nonlinear optimization. SIAM J. Optim. 23(3), 1553–1574 (2013)
Cartis, C., Gould, N.I., Toint, P.L.: On the complexity of finding first-order critical points in constrained nonlinear optimization. Math. Program. 144(1), 93–106 (2014)
Cartis, C., Gould, N.I., Toint, P.L.: On the complexity of finding first-order critical points in constrained nonlinear optimization. Math. Program. 144(1), 93–106 (2014)
Cartis, C., Gould, N.I., Toint, P.L.: On the evaluation complexity of constrained nonlinear least-squares and general constrained nonlinear optimization using second-order methods. SIAM J. Numer. Anal. 53(2), 836–851 (2015)
Cartis, C., Gould, N.I., Toint, P.L.: Evaluation complexity bounds for smooth constrained nonlinear optimization using scaled KKT conditions, high-order models and the criticality measure \(\chi \). In: Demetriou, I., Pardalos, P. (eds.) Approximation and Optimization: Algorithms, Complexity and Applications, pp. 5–26. Springer, NY (2019)
Cartis, C., Gould, N.I., Toint, P.L.: Optimality of orders one to three and beyond: characterization and evaluation complexity in constrained nonconvex optimization. J. Complex. 53, 68–94 (2019)
Chen, X., Guo, L., Lu, Z., Ye, J.J.: An augmented Lagrangian method for non-Lipschitz nonconvex programming. SIAM J. Numer. Anal. 55(1), 168–193 (2017)
Chi, Y., Lu, Y.M., Chen, Y.: Nonconvex optimization meets low-rank matrix factorization: an overview. IEEE Trans. Signal Process. 67(20), 5239–5269 (2019)
Cifuentes, D., Moitra, A.: Polynomial time guarantees for the Burer-Monteiro method. (2019) arXiv:1912.01745
Coleman, T.F., Liu, J., Yuan, W.: A new trust-region algorithm for equality constrained optimization. Comput. Optim. Appl. 21(2), 177–199 (2002)
Conn, A.R., Gould, G., Toint, P.L.: LANCELOT: a Fortran package for large-scale nonlinear optimization. Springer Science & Business Media, NY (2013)
Conn, A.R., Gould, N.I., Toint, P.L.: A globally convergent Lagrangian barrier algorithm for optimization with general inequality constraints and simple bounds. Math. Comput. 66(217), 261–288 (1997)
Curtis, F.E., Robinson, D.P., Royer, C.W., Wright, S.J.: Trust-region Newton-CG with strong second-order complexity guarantees for nonconvex optimization. SIAM J. Optim. 31(1), 518–544 (2021)
Curtis, F.E., Robinson, D.P., Samadi, M.: A trust region algorithm with a worst-case iteration complexity of \(\cal{O} (\epsilon ^{-3/2})\) for nonconvex optimization. Math. Program. 1(162), 1–32 (2016)
Curtis, F.E., Robinson, D.P., Samadi, M.: Complexity analysis of a trust funnel algorithm for equality constrained optimization. SIAM J. Optim. 28(2), 1533–1563 (2018)
De Carvalho, E.P., dos Santos Júnior, A., Ma, T.F.: Reduced gradient method combined with augmented Lagrangian and barrier for the optimal power flow problem. Appl. Math. Comput. 200(2), 529–536 (2008)
P. Dvurechensky and M. Staudigl. Hessian barrier algorithms for non-convex conic optimization. (2021) arXiv:2111.00100
Goldfarb, D., Polyak, R., Scheinberg, K., Yuzefovich, I.: A modified barrier-augmented Lagrangian method for constrained minimization. Comput. Optim. Appl. 14(1), 55–74 (1999)
F. Goyens, A. Eftekhari, and N. Boumal. Computing second-order points under equality constraints: revisiting Fletcher’s augmented Lagrangian. (2022) arXiv:2204.01448
Grapiglia, G.N., Yuan, Y.-X.: On the complexity of an augmented Lagrangian method for nonconvex optimization. IMA J. Numer. Anal. 41(2), 1546–1568 (2021)
Haeser, G., Liu, H., Ye, Y.: Optimality condition and complexity analysis for linearly-constrained optimization without differentiability on the boundary. Math. Program. 178(1), 263–299 (2019)
He, C., Lu, Z.: A Newton-CG based barrier method for finding a second-order stationary point of nonconvex conic optimization with complexity guarantees. SIAM J. Optim. 33(2), 1191–1222 (2023)
He, C., Lu, Z., Pong, T.K.: A Newton-CG based augmented Lagrangian method for finding a second-order stationary point of nonconvex equality constrained optimization with complexity guarantees. SIAM J. Optim. 33(3), 1734–1766 (2023)
Huck, A., Guillaume, M., Blanc-Talon, J.: Minimum dispersion constrained nonnegative matrix factorization to unmix hyperspectral data. IEEE Trans. Geosci. Remote Sens. 48(6), 2590–2602 (2010)
C. Jin, P. Netrapalli, and M. I. Jordan. Accelerated gradient descent escapes saddle points faster than gradient descent. In: Conference on learning theory, pp. 1042–1085. (2018)
Kanzow, C., Steck, D.: An example comparing the standard and safeguarded augmented Lagrangian methods. Oper. Res. Lett. 45(6), 598–603 (2017)
Kuczyński, J., Woźniakowski, H.: Estimating the largest eigenvalue by the power and Lanczos algorithms with a random start. SIAM J. Matrix Anal. Appl. 13(4), 1094–1122 (1992)
Kuhlmann, R., Büskens, C.: A primal-dual augmented Lagrangian penalty-interior-point filter line search algorithm. Math. Method Oper. Res. 87(3), 451–483 (2018)
Liu, X., Xia, W., Wang, B., Zhang, L.: An approach based on constrained nonnegative matrix factorization to unmix hyperspectral data. IEEE Trans. Geosci. Remote Sens. 49(2), 757–772 (2010)
Lu, S., Razaviyayn, M., Yang, B., Huang, K., Hong, M.: Finding second-order stationary points efficiently in smooth nonconvex linearly constrained optimization problems. Adv. Neural Info. Process. Syst. 33, 2811–2822 (2020)
Lu, Z., Zhang, Y.: An augmented Lagrangian approach for sparse principal component analysis. Math. Program. 135(1), 149–193 (2012)
Martínez, J.M., Raydan, M.: Cubic-regularization counterpart of a variable-norm trust-region method for unconstrained minimization. J. Glob. Optim. 68(2), 367–385 (2017)
Miao, L., Qi, H.: Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization. IEEE Trans. Geosci. Remote Sens. 45(3), 765–777 (2007)
Moguerza, J.M., Prieto, F.J.: An augmented Lagrangian interior-point method using directions of negative curvature. Math. Program. 95(3), 573–616 (2003)
Nesterov, Y., Nemirovskii, A.: Interior-point polynomial algorithms in convex programming. SIAM, Philadelphia (1994)
Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. 108(1), 177–205 (2006)
O’Neill, M., Wright, S.J.: A log-barrier Newton-CG method for bound constrained optimization with complexity guarantees. IMA J. Numer. Anal. 41(1), 84–121 (2021)
Park, D., Kyrillidis, A., Carmanis, C., Sanghavi, S.: Non-square matrix sensing without spurious local minima via the burer-monteiro approach. In Artificial intelligence and statistics, pp. 65–74. PMLR, (2017)
Royer, C.W., O’Neill, M., Wright, S.J.: A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization. Math. Program. 180(1), 451–488 (2020)
Royer, C.W., Wright, S.J.: Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim. 28(2), 1448–1477 (2018)
Ruszczynski, A.: Nonlinear optimization. Princeton University Press, NJ (2011)
M. F. Sahin, A. Eftekhari, A. Alacaoglu, F. Latorre, and V. Cevher. An inexact augmented Lagrangian framework for nonconvex optimization with nonlinear constraints. Adv. Neural Info. Process. Syst. 32, 632–650 (2019)
Thanh, O. V., Gillis, N., Lecron, F.: Bounded simplex-structured matrix factorization. In ICASSP 2022-2022 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp. 9062–9066. IEEE, (2022)
Vanderbei, R.J., Shanno, D.F.: An interior-point algorithm for nonconvex nonlinear programming. Comput. Optim. Appl. 13(1), 231–252 (1999)
Wächter, A., Biegler, L.T.: On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006)
Waltz, R.A., Morales, J.L., Nocedal, J., Orban, D.: An interior algorithm for nonlinear optimization that combines line search and trust region steps. Math. Program. 107(3), 391–408 (2006)
Wright, S.J., Nowak, R.D., Figueiredo, M.A.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)
Xie, P., Wright, S.J.: Complexity of projected Newton methods for bound-constrained optimization. (2021) arXiv:2103.15989
Xie, Y., Wright, S.J.: Complexity of proximal augmented Lagrangian for nonconvex optimization with nonlinear equality constraints. J. Sci. Comput. 86(3), 1–30 (2021)
Xu, Y., Jin, R., Yang, Y.: NEON+: Accelerated gradient methods for extracting negative curvature for non-convex optimization. (2017) arXiv:1712.01033
Yang, L., Sun, D., Toh, K.C.: SDPNAL+: a majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints. Math. Program. Comput. 7(3), 331–366 (2015)
Zhao, X., Sun, D., Toh, K.C.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20(4), 1737–1765 (2010)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The third author is an editorial board member of this journal.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The work of this author was partially supported by NSF Award IIS-2211492
The work of this author was partially supported by NSF Award IIS-2211491.
Appendices
Appendix
A capped conjugate gradient method
In this part we present the capped CG method proposed in [60, Algorithm 1] for solving a possibly indefinite linear system (15). As briefly discussed in Sect. 4, the capped CG method finds either an approximate solution to (15) or a sufficiently negative curvature direction of the associated matrix H. More details about this method can be found in [60, Section 3.1].
The following theorem presents the iteration complexity of Algorithm 3, whose proof can be found in [44, Theorem A.1], and thus omitted here.
Theorem A.1
(iteration complexity of Algorithm 3) Consider applying Algorithm 3 with the optional input \(U=0\) to the linear system (15) with \(g\ne 0\), \(\varepsilon >0\), and H being an \(n\times n\) symmetric matrix. Then the number of iterations of Algorithm 3 is \({\widetilde{{{\,\mathrm{\mathcal {O}}\,}}}}(\min \{n,\sqrt{\Vert H\Vert /\varepsilon }\})\).
A randomized Lanczos based minimum eigenvalue oracle
In this part we present the randomized Lanczos method proposed in [60, Section 3.2], which can be used as a minimum eigenvalue oracle for Algorithm 1. As mentioned in Sect. 4, this oracle either outputs a sufficiently negative curvature direction of H or certifies that H is nearly positive semidefinite with high probability. More details about it can be found in [60, Section 3.2].
The following theorem justifies that Algorithm 4 is a suitable minimum eigenvalue oracle for Algorithm 1. Its proof is identical to that of [60, Lemma 2] and thus omitted.
Theorem B.1
[Iteration complexity of Algorithm 4] Consider Algorithm 4 with tolerance \(\varepsilon >0\), probability parameter \(\delta \in (0,1)\), and symmetric matrix \(H\in {{\,\mathrm{\mathbb {R}}\,}}^{n\times n}\) as its input. Then it either finds a sufficiently negative curvature direction v satisfying \(v^THv\le -\varepsilon /2\) and \(\Vert v\Vert =1\) or certifies that \(\lambda _{\min }(H)\ge -\varepsilon \) holds with probability at least \(1-\delta \) in at most \(N(\varepsilon ,\delta )\) iterations, where \(N(\varepsilon ,\delta )\) is defined in (111).
Notice that generally, computing \(\Vert H\Vert \) in Algorithm 4 may not be cheap when n is large. Nevertheless, \(\Vert H\Vert \) can be efficiently estimated via a randomization scheme with high confidence (e.g., see the discussion in [60, Appendix B3]).
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
He, C., Huang, H. & Lu, Z. A Newton-CG based barrier-augmented Lagrangian method for general nonconvex conic optimization. Comput Optim Appl 89, 843–894 (2024). https://doi.org/10.1007/s10589-024-00603-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-024-00603-6
Keywords
- Nonconvex conic optimization
- Second-order stationary point
- Augmented Lagrangian method
- Barrier method
- Newton-conjugate gradient method
- Iteration complexity
- Operation complexity