Hostname: page-component-848d4c4894-8kt4b Total loading time: 0 Render date: 2024-06-16T09:43:26.977Z Has data issue: false hasContentIssue false
  • English
  • Français

Bregman distance regularization for nonsmooth and nonconvex optimization

Published online by Cambridge University Press:  26 October 2023

Zeinab Mashreghi  and
Mostafa Nasri
Zeinab Mashreghi
Affiliation:
Department of Mathematics and Statistics, University of Winnipeg, Winnipeg, MB R3B 2E9, Canada e-mail: z.mashreghi@uwinnipeg.ca
Mostafa Nasri*
Affiliation:
Department of Mathematics and Statistics, University of Winnipeg, Winnipeg, MB R3B 2E9, Canada e-mail: z.mashreghi@uwinnipeg.ca
*
e-mail: m.nasri@uwinnipeg.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

Solving a nonsmooth and nonconvex minimization problem can be approached as finding a zero of a set-valued operator. With this perspective, we propose a novel Majorizer–Minimizer technique to find a local minimizer of a nonsmooth and nonconvex function and establish its convergence. Our approach leverages Bregman distances to generalize the classical quadratic regularization. By doing so, we generate a family of regularized problems that encompasses quadratic regularization as a special case. To further demonstrate the effectiveness of our method, we apply it on a lasso regression model, showcasing its performance.

Keywords

Bregman distancelasso regression modellocal minimizernonconvex optimizationnonsmooth optimization

MSC classification

Primary: 90C26: Nonconvex programming, global optimization
Type
Article
Information
Canadian Mathematical Bulletin , Volume 67 , Issue 2 , June 2024 , pp. 415 - 424
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of The Canadian Mathematical Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bolte, J., Daniilidis, A., Lewis, A., and Shiota, M., Clarke subgradients of stratifiable functions . SIAM J. Optim. 18(2007), 556–572.CrossRefGoogle Scholar
Bregman, L., The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming . USSR Comput. Math. Math. Phys. 7(1967), 200–217.CrossRefGoogle Scholar
Burke, J., Lewis, A., and Overton, M., A robust gradient sampling algorithm for nonsmooth, nonconvex optimization . SIAM J. Optim. 15(2005), 751–779.CrossRefGoogle Scholar
Butnariu, D. and Iusem, A. N., Totally convex functions for fixed points computation and infinite dimensional optimization, Kluwer, Dordrecht, 2000.CrossRefGoogle Scholar
Censor, Y. and Zenios, S., Proximal minimization with d-functions . J. Optim. Theory Appl. 73(1992), 451–464.CrossRefGoogle Scholar
Chen, X. and Zhou, W., Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization . SIAM J. Imaging Sci. 3(2010), 765–790.CrossRefGoogle Scholar
Dang, C. and Lan, G., Stochastic block mirror descent methods for nonsmooth and stochastic optimization . SIAM J. Optim. 25(2015), 856–881.CrossRefGoogle Scholar
Hastie, T., Tibshirani, R., and Wainwright, M., Statistical learning with sparsity: the lasso and generalizations, Chapman & Hall/CRC Press, New York, 2015.CrossRefGoogle Scholar
Hunter, D. and Lange, K., A tutorial on MM algorithms . Am. Stat. 58(2004), 30–37.CrossRefGoogle Scholar
Iusem, A., Augmented Lagrangian methods and proximal point methods for convex optimization . Investigación Oper. 8(1999), 11–49.Google Scholar
Iusem, A., Svaiter, B., and Teboulle, M., Entropy-like proximal methods in convex programming . Math. Oper. Res. 19(1994), 790–814.CrossRefGoogle Scholar
James, G., Witten, D., Hastie, T., and Tibshirani, R., An introduction to statistical learning with applications in R, Springer, New York, 2017.Google Scholar
Mairal, J., Optimization with first-order surrogate functions . In: ICML 2013 – international conference on machine learning. Vol. 28, PMLR, 2013, pp. 783–791.Google Scholar
Mayeli, A., Non-convex optimization via strongly convex majoirziation–minimization . Canad. Math. Bull. 63(2020), 726–737.CrossRefGoogle Scholar
Ruszczyński, A., Convergence of a stochastic subgradient method with averaging for nonsmooth nonconvex constrained optimization . Optim. Lett. 14(2020), 1615–1625.CrossRefGoogle Scholar
Selesnick, I., Sparse regularization via convex analysis . IEEE Trans. Signal Process. 65(2017), 4481–4494.CrossRefGoogle Scholar
Sun, Y., Babu, P., and Palomar, D., Majorization–minimization algorithms in signal processing, communications, and machine learning . IEEE Trans. Signal Process. 65(2017), 794–816.CrossRefGoogle Scholar