Hostname: page-component-cd4964975-xtmlv Total loading time: 0 Render date: 2023-03-28T13:36:41.960Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

# 8 - Introduction to Optimization and Nonlinear Equations

Published online by Cambridge University Press:  01 June 2011

## Summary

Introduction

This chapter serves as an appetizer to the main course, maximum likelihood and nonlinear least squares. This is stated so boldly because many statistical problems of this type originate in estimation problems with maximum likelihood (or a similar criterion) as the goal. Our discussion begins with some of the background calculus and definitions. Next, the discussion turns to the safe and slow methods for optimization in a single variable, for which the statistical term “nonparametric” has the correct connotations. Next, the root-finding problem is addressed with the standard techniques, Newton and secant methods, followed by a brief presentation of convergence rates. After a short digression on stopping and condition, the multivariate problem is first approached with Newton's methods. After a second digression on numerical differentiation, quasi-Newton methods are discussed for optimization and nonlinear equations. Discussions of condition, scaling, and implementation conclude the chapter.

Some topics are not addressed in this discussion. One problem is the solution of polynomial equations, which arise rarely in an isolated form in statistics. Constrained optimization can often be avoided through reparameterization. The specialized problem of nonlinear regression is postponed until the next chapter, to be treated as a special topic in maximum likelihood.

Before attacking the problems at hand, it is wise to review some foundations to gain a clearer perspective of the situation. The cornerstone for everything are the first results of calculus, the primary tools in applied mathematics.

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2011

## Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

### Purchase

Buy print or eBook[Opens in a new window]

## References

(2002), “Numerical Differentiation Using Statistical Design,” unpublished Ph.D. thesis, North Carolina State University.
(1973), Algorithms for Minimization without Derivatives. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
(1970), Analysis of Binary Data. London: Methuen.Google Scholar
, (1984), “A User's Guide to Nonlinear Optimization Algorithms,” Proceedings of the IEEE 72: 1765–76.CrossRefGoogle Scholar
, and (1983), Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
and (1971), “A Modified Regula Falsi Method for Computing the Root of an Equation,” BIT 11: 168–74.CrossRefGoogle Scholar
(1983), “Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust Region Approach,” ACM Transactions on Mathematical Software 9: 503–24.CrossRefGoogle Scholar
(1976), “Factorized Variable Metric Methods for Unconstrained Optimization,” Mathematics of Computation 30: 796–811.CrossRefGoogle Scholar
and (2008), Evaluating Derivatives: Principles and Techniques for Algorithmic Differentiation. 2nd edition, Philadelphia: SIAM.CrossRefGoogle Scholar
and (2003), “Verifying the Solution from a Nonlinear Solver: A Case Study,” The American Economic Review 93: 873–91.CrossRefGoogle Scholar
and (1965), “A Simplex Method for Function Minimization,” The Computer Journal 7: 308–13.CrossRefGoogle Scholar
(1970), “A New Algorithm for Unconstrained Optimization,” in , , and (Eds.), Nonlinear Programming, pp. 31–65. New York: Academic Press.CrossRefGoogle Scholar
(1960), “An Automatic Method for Finding the Greatest or Least Value of a Function,” Computer Journal 3: 175–84.CrossRefGoogle Scholar
, , and (1962), “Sequential Application of Simplex Designs in Optimization and Evolutionary Operation,” Technometrics 4: 441–61.CrossRefGoogle Scholar
(1988), “Solving Systems of Nonlinear Equations Using the Nonzero Value of the Topological Degree,” ACM Transactions on Mathematical Software 14: 312–29.CrossRefGoogle Scholar
(1964), Optimum Seeking Methods. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar

# Save book to Kindle

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

# Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

# Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×