Fast Convergence of Dynamical ADMM via Time Scaling of Damped Inertial Dynamics | Journal of Optimization Theory and Applications Skip to main content
Springer Nature Link
Log in

Fast Convergence of Dynamical ADMM via Time Scaling of Damped Inertial Dynamics

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose in a Hilbertian setting a second-order time-continuous dynamic system with fast convergence guarantees to solve structured convex minimization problems with an affine constraint. The system is associated with the augmented Lagrangian formulation of the minimization problem. The corresponding dynamics brings into play three general time-varying parameters, each with specific properties, and which are, respectively, associated with viscous damping, extrapolation and temporal scaling. By appropriately adjusting these parameters, we develop a Lyapunov analysis which provides fast convergence properties of the values and of the feasibility gap. These results will naturally pave the way for developing corresponding accelerated ADMM algorithms, obtained by temporal discretization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Actually, convexity is not needed here.

  2. Again, convexity is superfluous here.

References

  1. Alvarez, F.: On the minimizing property of a second order dissipative system in Hilbert spaces. SIAM J. Control Optim. 38(4), 1102–1119 (2000). https://doi.org/10.1137/S0363012998335802

    Article  MathSciNet  MATH  Google Scholar 

  2. Alvarez, F., Attouch, H., Bolte, J., Redont, P.: A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics. J. Math. Pures Appl. (9) 81(8), 747–779 (2002). https://doi.org/10.1016/S0021-7824(01)01253-3

    Article  MathSciNet  MATH  Google Scholar 

  3. Apidopoulos, V., Aujol, J.F., ossal, C.: Convergence rate of inertial forward-backward algorithm beyond Nesterov’s rule. Math. Program. 180(1–2, Ser. A), 137–156 (2020). https://doi.org/10.1007/s10107-018-1350-9

  4. Attouch, H.: Variational convergence for functions and operators. Applicable mathematics series. Pitman Advanced Publishing Program (1984). https://books.google.fr/books?id=oxGoAAAAIAAJ

  5. Attouch, H.: Fast inertial proximal ADMM algorithms for convex structured optimization with linear constraint. Minimax Theory Appl. 6(1), 1–24 (2021)

    MathSciNet  MATH  Google Scholar 

  6. Attouch, H., Balhag, A., Chbani, Z., Riahi, H.: Fast convex optimization via inertial dynamics combining viscous and hessian-driven damping with time rescaling. Evolution Equations & Control Theory (2021). http://aimsciences.org//article/id/92767e45-ae6f-4ff2-9c62-d7ead2d77844

  7. Attouch, H., Cabot, A.: Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity. J. Differ Equ. 263(9), 5412–5458 (2017). https://doi.org/10.1016/j.jde.2017.06.024

    Article  MathSciNet  MATH  Google Scholar 

  8. Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018). https://doi.org/10.1137/17M1114739

    Article  MathSciNet  MATH  Google Scholar 

  9. Attouch, H., Cabot, A., Chbani, Z., Riahi, H.: Rate of convergence of inertial gradient dynamics with time-dependent viscous damping coefficient. Evol. Equ. Control Theory 7(3), 353–371 (2018). https://doi.org/10.3934/eect.2018018

    Article  MathSciNet  MATH  Google Scholar 

  10. Attouch, H., Chbani, Z., Fadili, J., Riahi, H.: First-order optimization algorithms via inertial systems with hessian driven damping. Math. Program. (2020). https://doi.org/10.1007/s10107-020-01591-1

    Article  Google Scholar 

  11. Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. 168(1–2, Ser. B), 123–175 (2018). https://doi.org/10.1007/s10107-016-0992-8

    Article  MathSciNet  MATH  Google Scholar 

  12. Attouch, H., Chbani, Z., Riahi, H.: Fast proximal methods via time scaling of damped inertial dynamics. SIAM J. Optim. 29(3), 2227–2256 (2019). https://doi.org/10.1137/18M1230207

    Article  MathSciNet  MATH  Google Scholar 

  13. Attouch, H., Chbani, Z., Riahi, H.: Rate of convergence of the Nesterov accelerated gradient method in the subcritical case \(\alpha \le 3\). ESAIM Control Optim. Calc. Var. 25(2), 34 (2019). https://doi.org/10.1051/cocv/2017083

  14. Attouch, H., Chbani, Z., Riahi, H.: Fast convex optimization via a third-order in time evolution equation. Optimization (2020). Preprint available at hal-02432351. https://doi.org/10.1080/02331934.2020.1764953

  15. Attouch, H., Chbani, Z., Riahi, H.: Fast convex optimization via time scaling of damped inertial gradient dynamics. Pure and Applied Functional Analysis (2020). To appear

  16. Attouch, H., Czarnecki, M.O., Peypouquet, J.: Coupling forward-backward with penalty schemes and parallel splitting for constrained variational inequalities. SIAM J. Optim. 21(4), 1251–1274 (2011). https://doi.org/10.1137/110820300

    Article  MathSciNet  MATH  Google Scholar 

  17. Attouch, H., Goudou, X., Redont, P.: The heavy ball with friction method. I. The continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system. Commun. Contemp. Math. 2(1), 1–34 (2000). https://doi.org/10.1142/S0219199700000025

    Article  MathSciNet  MATH  Google Scholar 

  18. Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than \(1/k^2\). SIAM J. Optim. 26(3), 1824–1834 (2016). https://doi.org/10.1137/15M1046095

    Article  MathSciNet  MATH  Google Scholar 

  19. Attouch, H., Peypouquet, J.: Convergence of inertial dynamics and proximal algorithms governed by maximally monotone operators. Math. Program. 174(1–2, Ser. B), 391–432 (2019). https://doi.org/10.1007/s10107-018-1252-x

    Article  MathSciNet  MATH  Google Scholar 

  20. Attouch, H., Peypouquet, J., Redont, P.: A dynamical approach to an inertial forward-backward algorithm for convex minimization. SIAM J. Optim. 24(1), 232–256 (2014). https://doi.org/10.1137/130910294

    Article  MathSciNet  MATH  Google Scholar 

  21. Attouch, H., Peypouquet, J., Redont, P.: Fast convex optimization via inertial dynamics with Hessian driven damping. J. Differ. Equ. 261(10), 5734–5783 (2016). https://doi.org/10.1016/j.jde.2016.08.020

    Article  MathSciNet  MATH  Google Scholar 

  22. Attouch, H., Soueycatt, M.: Augmented Lagrangian and proximal alternating direction methods of multipliers in Hilbert spaces. Applications to game, PDE’s and control. Pac. J. Optim. 5(1), 17–37 (2009)

    MathSciNet  MATH  Google Scholar 

  23. Aujol, J.F., Dossal, C.: Stability of over-relaxations for the forward-backward algorithm, application to FISTA. SIAM J. Optim. 25(4), 2408–2433 (2015). https://doi.org/10.1137/140994964

    Article  MathSciNet  MATH  Google Scholar 

  24. Bauschke, H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd edn. CMS Books in Mathematics. Springer, New York (2017)

  25. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009). https://doi.org/10.1137/080716542

    Article  MathSciNet  MATH  Google Scholar 

  26. Boţ, R.I., Csetnek, E.R.: An inertial alternating direction method of multipliers. Minimax Theory Appl. 1(1), 29–49 (2016)

    MathSciNet  MATH  Google Scholar 

  27. Boţ, R.I., Csetnek, E.R.: An inertial Tseng’s type proximal algorithm for nonsmooth and nonconvex optimization problems. J. Optim. Theory Appl. 171(2), 600–616 (2016). https://doi.org/10.1007/s10957-015-0730-z

    Article  MathSciNet  MATH  Google Scholar 

  28. Boţ, R.I., Csetnek, E.R., Hendrich, C.: Inertial Douglas-Rachford splitting for monotone inclusion problems. Appl. Math. Comput. 256, 472–487 (2015). https://doi.org/10.1016/j.amc.2015.01.017

    Article  MathSciNet  MATH  Google Scholar 

  29. Boţ, R.I., Csetnek, E.R., László, S.C.: Second-order dynamical systems with penalty terms associated to monotone inclusions. Anal. Appl. (Singap.) 16(5), 601–622 (2018). https://doi.org/10.1142/S0219530518500021

    Article  MathSciNet  MATH  Google Scholar 

  30. Brézis, H.: Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert. North-Holland Publishing Co., Amsterdam-London; American Elsevier Publishing Co., Inc., New York (1973). North-Holland Mathematics Studies, No. 5. Notas de Matemática (50)

  31. Chambolle, A., Dossal, C.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015). https://doi.org/10.1007/s10957-015-0746-4

    Article  MathSciNet  MATH  Google Scholar 

  32. Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. In: Splitting methods in communication, imaging, science, and engineering, Sci. Comput., pp. 115–163. Springer, Cham (2016)

  33. Davis, D., Yin, W.: Faster convergence rates of relaxed Peaceman-Rachford and ADMM under regularity assumptions. Math. Oper. Res. 42(3), 783–805 (2017). https://doi.org/10.1287/moor.2016.0827

    Article  MathSciNet  MATH  Google Scholar 

  34. Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci. 7(3), 1588–1623 (2014). https://doi.org/10.1137/120896219

    Article  MathSciNet  MATH  Google Scholar 

  35. Haraux, A.: Systèmes dynamiques dissipatifs et applications, Recherches en Mathématiques Appliquées [Research in Applied Mathematics], vol. 17. Masson, Paris (1991)

    Google Scholar 

  36. He, X., Hu, R., Fang, Y.: Convergence rates of inertial primal-dual dynamical methods for separable convex optimization problems. arXiv:2007.12428 (2020)

  37. Kang, M., Kang, M., Jung, M.: Inexact accelerated augmented Lagrangian methods. Comput. Optim. Appl. 62(2), 373–404 (2015). https://doi.org/10.1007/s10589-015-9742-8

    Article  MathSciNet  MATH  Google Scholar 

  38. Kang, M., Yun, S., Woo, H., Kang, M.: Accelerated Bregman method for linearly constrained \(\ell _1\)-\(\ell _2\) minimization. J. Sci. Comput. 56(3), 515–534 (2013). https://doi.org/10.1007/s10915-013-9686-z

    Article  MathSciNet  MATH  Google Scholar 

  39. May, R.: Asymptotic for a second-order evolution equation with convex potential and vanishing damping term. Turkish J. Math. 41(3), 681–685 (2017). https://doi.org/10.3906/mat-1512-28

    Article  MathSciNet  MATH  Google Scholar 

  40. Michiels, W., Vyhlídal, T., Huijberts, H., Nijmeijer, H.: Stabilizability and stability robustness of state derivative feedback controllers. SIAM J. Control Optim. 47(6), 3100–3117 (2009). https://doi.org/10.1137/070697136

    Article  MathSciNet  MATH  Google Scholar 

  41. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1, Ser. B), 125–161 (2013). https://doi.org/10.1007/s10107-012-0629-5

    Article  MathSciNet  MATH  Google Scholar 

  42. Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate \(O(1/k^{2})\). Dokl. Akad. Nauk SSSR 269(3), 543–547 (1983)

    MathSciNet  Google Scholar 

  43. Patrinos, P., Stella, L., Bemporad, A.: Douglas-Rachford splitting: Complexity estimates and accelerated variants. In: 53rd IEEE Conference on Decision and Control, pp. 4234–4239 (2014). https://doi.org/10.1109/CDC.2014.7040049

  44. Pejcic, I., Jones, C.N.: Accelerated ADMM based on accelerated Douglas-Rachford splitting. In: European Control Conference (ECC), pp. 1952–1957. Aalborg, Denmark (2016)

  45. Polyak, B.T.: Some methods of speeding up the convergence of iterative methods. Ž. Vyčisl. Mat i Mat. Fiz. 4, 791–803 (1964)

    MathSciNet  Google Scholar 

  46. Polyak, B.T.: Introduction to optimization. Translations Series in Mathematics and Engineering. Optimization Software Inc, Publications Division, New York, : Translated from the Russian. With a foreword by Dimitri P, Bertsekas (1987)

  47. Poon, C., Liang, J.: Trajectory of alternating direction method of multipliers and adaptive acceleration. In: 33rd Conference on Neural Information Processing Systems (NeurIPS). Vancouver, Canada (2019)

  48. Rockafellar, R.T.: Monotone operators associated with saddle-functions and minimax problems. In: Nonlinear Functional Analysis (Proc. Sympos. Pure Math., Vol. XVIII, Part 1, Chicago, Ill., 1968), pp. 241–250. Amer. Math. Soc., Providence, R.I. (1970)

  49. Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1(2), 97–116 (1976). https://doi.org/10.1287/moor.1.2.97

    Article  MathSciNet  MATH  Google Scholar 

  50. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976). https://doi.org/10.1137/0314056

    Article  MathSciNet  MATH  Google Scholar 

  51. Shi, B., Du, S.S., Jordan, M.I., Su, W.J.: Understanding the acceleration phenomenon via high-resolution differential equations. arXiv:2440124 (2018)

  52. Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. J. Mach. Learn. Res. 17(153), 43 (2016)

    MathSciNet  MATH  Google Scholar 

  53. Wilson, A.C., Recht, B., Jordan, M.I.: A lyapunov analysis of momentum methods in optimization. arXiv:1611.02635 (2016)

  54. Zeng, X., Lei, J., Chen, J.: Dynamical primal-dual accelerated method with applications to network optimization. arXiv:1912.03690 (2019)

Download references

Author information

Authors and Affiliations

  1. IMAG, CNRS, Université Montpellier, Montpellier, France

    Hedy Attouch

  2. Faculty of Sciences Semlalia, Mathematics, Cadi Ayyad University, 40000, Marrakech, Morocco

    Zaki Chbani & Hassan Riahi

  3. Normandie Université, ENSICAEN, UNICAEN, CNRS, GREYC, Caen, France

    Jalal Fadili

Authors
  1. Hedy Attouch
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zaki Chbani
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jalal Fadili
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hassan Riahi
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jalal Fadili.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Attouch, H., Chbani, Z., Fadili, J. et al. Fast Convergence of Dynamical ADMM via Time Scaling of Damped Inertial Dynamics. J Optim Theory Appl 193, 704–736 (2022). https://doi.org/10.1007/s10957-021-01859-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-021-01859-2

Keywords

JEL Classification