Stochastic approximation in parameter estimation

Remarks

Consider the general linear model

y = XT β + e

where e is N (0, σ2). The parameter estimate may be obtained using the least squares approach. Toward that end, we minimize the functional Λ = (y–XT β)T (y–XTβ) for every unknown parameter (β1, β2, … βp) of the vector β. The solution of the normal equation ∂Λ/∂βj = 0 yields the LSE estimator. Under the Gaussian assumption, this leads to the usual F-statistic. This approach is still valid if there are small to moderate deviations from the Gaussian distribution. This can be verified by extensive simulations. Many of them were performed by the authors and some of the senior author's doctoral students. Also, one should consult reference.

In the presence of impulsive noise, the LS approach is no longer applicable, and alternate techniques must be used. In this case, we consider two different situations. In the first case noise contamination is not severe, in which case the parameters of the linear model are estimated using the nonrobustized version of the Robbins–Monro stochastic approximation (RMSA) algorithm. Second, if the background noise is severe, we use the robustized version of the RMSA estimator.