Skip to main content Accessibility help
Hostname: page-component-cd4964975-xtmlv Total loading time: 0 Render date: 2023-03-28T13:36:41.960Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

8 - Introduction to Optimization and Nonlinear Equations

Published online by Cambridge University Press:  01 June 2011

John F. Monahan
North Carolina State University
Get access



This chapter serves as an appetizer to the main course, maximum likelihood and nonlinear least squares. This is stated so boldly because many statistical problems of this type originate in estimation problems with maximum likelihood (or a similar criterion) as the goal. Our discussion begins with some of the background calculus and definitions. Next, the discussion turns to the safe and slow methods for optimization in a single variable, for which the statistical term “nonparametric” has the correct connotations. Next, the root-finding problem is addressed with the standard techniques, Newton and secant methods, followed by a brief presentation of convergence rates. After a short digression on stopping and condition, the multivariate problem is first approached with Newton's methods. After a second digression on numerical differentiation, quasi-Newton methods are discussed for optimization and nonlinear equations. Discussions of condition, scaling, and implementation conclude the chapter.

Some topics are not addressed in this discussion. One problem is the solution of polynomial equations, which arise rarely in an isolated form in statistics. Constrained optimization can often be avoided through reparameterization. The specialized problem of nonlinear regression is postponed until the next chapter, to be treated as a special topic in maximum likelihood.

Before attacking the problems at hand, it is wise to review some foundations to gain a clearer perspective of the situation. The cornerstone for everything are the first results of calculus, the primary tools in applied mathematics.

Publisher: Cambridge University Press
Print publication year: 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Bodily, Christopher H. (2002), “Numerical Differentiation Using Statistical Design,” unpublished Ph.D. thesis, North Carolina State University.
Brent, R. P. (1973), Algorithms for Minimization without Derivatives. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Cox, D. R. (1970), Analysis of Binary Data. London: Methuen.Google Scholar
Dennis, J. E. Jr., (1984), “A User's Guide to Nonlinear Optimization Algorithms,” Proceedings of the IEEE 72: 1765–76.CrossRefGoogle Scholar
Dennis, J. E. Jr., and Schnabel, R. B. (1983), Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Dowell, M. and Jarratt, P. (1971), “A Modified Regula Falsi Method for Computing the Root of an Equation,” BIT 11: 168–74.CrossRefGoogle Scholar
Gay, D. M. (1983), “Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust Region Approach,” ACM Transactions on Mathematical Software 9: 503–24.CrossRefGoogle Scholar
Gill, P. E., Murray, W., and Wright, M. H. (1981), Practical Optimization. London: Academic Press.Google Scholar
Goldfarb, D. (1976), “Factorized Variable Metric Methods for Unconstrained Optimization,” Mathematics of Computation 30: 796–811.CrossRefGoogle Scholar
Griewank, Andreas and Walther, Andrea (2008), Evaluating Derivatives: Principles and Techniques for Algorithmic Differentiation. 2nd edition, Philadelphia: SIAM.CrossRefGoogle Scholar
Kelley, C. T. (1999), Iterative Methods for Optimization. Philadelphia: SIAM.CrossRefGoogle Scholar
Luenberger, D. G. (1984), Linear and Nonlinear Programming, 2nd ed. Reading, MA: Addison-Wesley.Google Scholar
McCullough, B. D. and Vinod, H. D. (2003), “Verifying the Solution from a Nonlinear Solver: A Case Study,” The American Economic Review 93: 873–91.CrossRefGoogle Scholar
Nelder, J. A. and Mead, R. (1965), “A Simplex Method for Function Minimization,” The Computer Journal 7: 308–13.CrossRefGoogle Scholar
Powell, M. J. D. (1970), “A New Algorithm for Unconstrained Optimization,” in Rosen, J. B., Mangasarian, O. L., and Ritter, K. (Eds.), Nonlinear Programming, pp. 31–65. New York: Academic Press.CrossRefGoogle Scholar
Rosenbrock, H. (1960), “An Automatic Method for Finding the Greatest or Least Value of a Function,” Computer Journal 3: 175–84.CrossRefGoogle Scholar
Spendley, W., Hext, G. R., and Himsworth, F. R. (1962), “Sequential Application of Simplex Designs in Optimization and Evolutionary Operation,” Technometrics 4: 441–61.CrossRefGoogle Scholar
Vrahatis, Michael N. (1988), “Solving Systems of Nonlinear Equations Using the Nonzero Value of the Topological Degree,” ACM Transactions on Mathematical Software 14: 312–29.CrossRefGoogle Scholar
Wilde, D. J. (1964), Optimum Seeking Methods. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats