Hostname: page-component-77c89778f8-m42fx Total loading time: 0 Render date: 2024-07-16T21:07:35.195Z Has data issue: false hasContentIssue false
  • English
  • Français

Gradient descent and fast artificial time integration

Published online by Cambridge University Press:  08 July 2009

Uri M. Ascher ,
Kees van den Doel ,
Hui Huang  and
Benar F. Svaiter
Uri M. Ascher
Affiliation:
Department of Computer Science, University of British Columbia, Vancouver, Canada. ascher@cs.ubc.ca kvdoel@cs.ubc.ca
Kees van den Doel
Affiliation:
Department of Computer Science, University of British Columbia, Vancouver, Canada. ascher@cs.ubc.ca kvdoel@cs.ubc.ca
Hui Huang
Affiliation:
Department of Mathematics, University of British Columbia, Vancouver, Canada. hhzhiyan@math.ubc.ca
Benar F. Svaiter
Affiliation:
Institute of Pure and Applied Mathematics (IMPA), Rio de Janeiro, Brazil. benar@impa.br
Get access

Abstract

The integration to steady state of many initial value ODEs and PDEs using the forward Euler method can alternatively be considered as gradient descent for an associated minimization problem. Greedy algorithms such as steepest descent for determining the step size are as slow to reach steady state as is forward Euler integration with the best uniform step size. But other, much faster methods using bolder step size selection exist. Various alternatives are investigated from both theoretical and practical points of view. The steepest descent method is also known for the regularizing or smoothing effect that the first few steps have for certain inverse problems, amounting to a finite time regularization. We further investigate the retention of this property using the faster gradient descent variants in the context of two applications. When the combination of regularization and accuracy demands more than a dozen or so steepest descent steps, the alternatives offer an advantage, even though (indeed because) the absolute stability limit of forward Euler is carefully yet severely violated.

Keywords

Steady stateartificial timegradient descentforward Eulerlagged steepest descentregularization.
Type
Research Article
Information
ESAIM: Mathematical Modelling and Numerical Analysis , Volume 43 , Issue 4: Special issue on Numerical ODEs today , July 2009 , pp. 689 - 708
Copyright
© EDP Sciences, SMAI, 2009

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akaike, H., On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Stat. Math. Tokyo 11 (1959) 1–16. CrossRef
U. Ascher, Numerical Methods for Evolutionary Differential Equations. SIAM, Philadelphia, USA (2008).
Ascher, U., Haber, E. and Huang, H., On effective methods for implicit piecewise smooth surface recovery. SIAM J. Sci. Comput. 28 (2006) 339–358. CrossRef
Ascher, U., Huang, H. and van den Doel, K., Artificial time integration. BIT 47 (2007) 3–25. CrossRef
Barzilai, J. and Borwein, J., Two point step size gradient methods. IMA J. Num. Anal. 8 (1988) 141–148. CrossRef
Cheney, M., Isaacson, D. and Newell, J.C., Electrical impedance tomography. SIAM Review 41 (1999) 85–101. CrossRef
Chung, E., Chan, T. and Tai, X., Electrical impedance tomography using level set representations and total variation regularization. J. Comp. Phys. 205 (2005) 357–372. CrossRef
Dai, Y. and Fletcher, R., Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100 (2005) 21–47. CrossRef
Dai, Y., Hager, W., Schittkowsky, K. and Zhang, H., A cyclic Barzilai-Borwein method for unconstrained optimization. IMA J. Num. Anal. 26 (2006) 604–627. CrossRef
H.W. Engl, M. Hanke and A. Neubauer, Regularization of Inverse Problems. Kluwer (1996).
Figueiredo, M., Nowak, R. and Wright, S., Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1 (2007) 586–598. CrossRef
Forsythe, G.E., On the asymptotic directions of the s-dimensional optimum gradient method. Numer. Math. 11 (1968) 57–76. CrossRef
Friedlander, A., Martinez, J., Molina, B. and Raydan, M., Gradient method with retard and generalizations. SIAM J. Num. Anal. 36 (1999) 275–289. CrossRef
Golub, G. and Inexact, Q. Ye preconditioned conjugate gradient method with inner-outer iteration. SIAM J. Sci. Comp. 21 (2000) 1305–1320. CrossRef
A. Greenbaum, Iterative Methods for Solving Linear Systems. SIAM, Philadelphia, USA (1997).
Haber, E. and Ascher, U., Preconditioned all-at-one methods for large, sparse parameter estimation problems. Inverse Problems 17 (2001) 1847–1864. CrossRef
E. Hairer and G. Wanner, Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems. Second Edition, Springer (1996).
H. Huang, Efficient Reconstruction of 2D Images and 3D Surfaces. Ph.D. Thesis, University of BC, Vancouver, Canada (2008).
W. Hundsdorfer and J.G. Verwer, Numerical Solution of Time-Dependent Advection-Diffusion-Reaction Equations. Springer (2003).
Li, Y. and Oldenburg, D.W., Inversion of 3-D DC resistivity data using an approximate inverse mapping. Geophys. J. Int. 116 (1994) 557–569.
Nagy, J. and Palmer, K., Steepest descent, CG and iterative regularization of ill-posed problems. BIT 43 (2003) 1003–1017. CrossRef
J. Nocedal and S. Wright, Numerical Optimization. Springer, New York (1999).
Nocedal, J., Sartenar, A. and Zhu, C., On the behavior of the gradient norm in the steepest descent method. Comput. Optim. Appl. 22 (2002) 5–35. CrossRef
S. Osher and R. Fedkiw, Level Set Methods and Dynamic Implicit Surfaces. Springer (2003).
Perona, P. and Malik, J., Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 629–639. CrossRef
L. Pronzato, H. Wynn and A. Zhigljavsky, Dynamical Search: Applications of Dynamical Systems in Search and Optimization. Chapman & Hall/CRC, Boca Raton (2000).
Raydan, M. and Svaiter, B., Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Comput. Optim. Appl. 21 (2002) 155–167. CrossRef
Sincovec, R. and Madsen, N., Software for nonlinear partial differential equations. ACM Trans. Math. Software 1 (1975) 232–260. CrossRef
Smith, N.C. and Vozoff, K., Two dimensional DC resistivity inversion for dipole dipole data. IEEE Trans. Geosci. Remote Sens. 22 (1984) 21–28. CrossRef
G. Strang and G. Fix, An Analysis of the Finite Element Method. Prentice-Hall, Engelwood Cliffs, NJ (1973).
Tadmor, E., Nezzar, S. and Vese, L., A multiscale image representation using hierarchical (BV, L 2) decompositions. SIAM J. Multiscale Model. Simul. 2 (2004) 554–579. CrossRef
van den Berg, E. and Friedlander, M., Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31 (2008) 840–912.
van den Doel, K. and Ascher, U., On level set regularization for highly ill-posed distributed parameter estimation problems. J. Comp. Phys. 216 (2006) 707–723. CrossRef
van den Doel, K. and Ascher, U., Dynamic level set regularization for large distributed parameter estimation problems. Inverse Problems 23 (2007) 1271–1288. CrossRef
C. Vogel, Computational methods for inverse problem. SIAM, Philadelphia, USA (2002).
J. Weickert, Anisotropic Diffusion in Image Processing. B.G. Teubner, Stuttgart (1998).