In statistics, Fisher's scoring algorithm is a form of Newton's method used to solve maximum likelihood equations numerically.
Sketch of Derivation
Let
be random variables, independent and identically distributed with twice differentiable p.d.f.
, and we wish to calculate the maximum likelihood estimator (M.L.E.)
of
. First, suppose we have a starting point for our algorithm
, and consider a Taylor expansion of the score function,
, about
:
![{\displaystyle V(\theta )\approx V(\theta _{0})-{\mathcal {J}}(\theta _{0})(\theta -\theta _{0}),\,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9479eb6e09c3297fc8c5133be66c925613eddde2)
where
![{\displaystyle {\mathcal {J}}(\theta _{0})=-\sum _{i=1}^{n}\left.\nabla \nabla ^{\top }\right|_{\theta =\theta _{0}}\log f(Y_{i};\theta )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a7f40eb3711baab4e7d5ae1b84f550fa596b6d73)
is the observed information matrix at
. Now, setting
, using that
and rearranging gives us:
![{\displaystyle \theta ^{*}\approx \theta _{0}+{\mathcal {J}}^{-1}(\theta _{0})V(\theta _{0}).\,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fb7d11d24799b68c87849ec12a41271b925f13a)
We therefore use the algorithm
![{\displaystyle \theta _{m+1}=\theta _{m}+{\mathcal {J}}^{-1}(\theta _{m})V(\theta _{m}),\,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a678930028b7896bf356d259f8daae7c45d66748)
and under certain regularity conditions, it can be shown that
.
Fisher scoring
In practice,
is usually replaced by
, the Fisher information, thus giving us the Fisher Scoring Algorithm:
.
See also
References
Jennrich, R. I., & Sampson, P. F. (1976). Newton-Raphson and related algorithms for maximum likelihood variance component estimation. Technometrics, 18, 11-17.