2003:63:2420:8691:DC42:37E7:5240:CDB3: /* Theory */

2014-11-23T18:57:12Z

Theory

en>Nsda: link -> mean squared displacement

2012-08-14T08:10:24Z

New page

In [[statistics]], '''efficiency''' is a term used in the comparison of various statistical procedures and, in particular, it refers to a measure of the optimality of an [[estimator]], of an [[experimental design]]{{sfn|Everitt|2002|loc=p. 128}} or of a hypothesis testing procedure.<ref>{{SpringerEOM | title= Efficiency of a statistical procedure| id=E/e035080 | last=Nikulin | first=M.S.}}</ref> Essentially, a more efficient estimator, experiment or test needs fewer samples than a less efficient one to achieve a given performance. This article primarily deals with efficiency of estimators.

The '''relative efficiency''' of two procedures is the ratio of their efficiencies, although often this term is used where the comparison is made between a given procedure and a notional "best possible" procedure. The efficiencies and the relative efficiency of two procedures theoretically depend on the sample size available for the given procedure, but it is often possible to use the '''asymptotic relative efficiency''' (defined as the limit of the relative efficiencies as the sample size grows) as the principal comparison measure.

Efficiencies are often defined using the [[variance]] or [[mean square error]] as the measure of desirability.{{sfn|Everitt|2002|loc=p. 128}}

==Estimators==

The efficiency of an [[bias of an estimator|unbiased]] [[estimator]], ''T'', of a [[statistical parameter|parameter]] ''θ'' is defined as{{cn|date=April 2013}}

:<math>
e(T)
=
\frac{1/\mathcal{I}(\theta)}{\mathrm{var}(T)}
</math>

where <math>\mathcal{I}(\theta)</math> is the [[Fisher information]] of the sample.
Thus ''e''(''T'') is the minimum possible variance for an unbiased estimator divided by its actual variance. The [[Cramér–Rao bound]] can be used to prove that ''e''(''T'') ≤ 1.

===Efficient estimators===
{{Main|efficient estimator}}
If an [[estimator bias|unbiased]] [[estimator]] of a parameter ''θ'' attains <math>e(T) = 1</math> for all values of the parameter, then the estimator is called efficient.{{Citation needed|date=January 2012}}

Equivalently, the estimator achieves equality in the [[Cramér–Rao inequality]] for all ''θ''.

An efficient estimator is also the [[minimum variance unbiased estimator]] (MVUE).
This is because an efficient estimator maintains equality on the Cramér–Rao inequality for all parameter values, which means it attains the minimum variance for all parameters (the definition of the MVUE). The MVUE estimator, even if it exists, is not necessarily efficient, because "minimum" does not mean equality holds on the Cramér–Rao inequality.

Thus an efficient estimator need not exist, but if it does, it is the MVUE.

===Asymptotic efficiency===
For some [[estimator]]s, they can attain efficiency [[asymptotically]] and are thus called asymptotically efficient estimators.
This can be the case for some [[maximum likelihood]] estimators or for any estimators that attain equality of the Cramér–Rao bound asymptotically.

====Example====
Consider a sample of size <math>N</math> drawn from a [[normal distribution]] of mean <math>\mu</math> and unit [[variance]], i.e., <math>X_n \sim \mathcal{N}(\mu, 1).</math>

The [[sample mean]], <math>\overline{X}</math>, of the sample <math>X_1, X_2, \ldots, X_N</math>, defined as

:<math>
\overline{X} = \frac{1}{N} \sum_{n=1}^{N} X_n \sim \mathcal{N}\left(\mu, \frac{1}{N}\right).
</math>

The variance of the mean, 1/''N'' (the square of the [[standard error]]) is equal to the reciprocal of the [[Fisher information]] from the sample and thus, by the [[Cramér–Rao inequality]], the sample mean is efficient in the sense that its efficiency is unity (100%).

Now consider the [[sample median]], <math>\widetilde{X}</math>.
This is an [[estimator bias|unbiased]] and [[Consistent estimator|consistent]] estimator for <math>\mu</math>.
For large <math>N</math> the sample median is approximately [[normal distribution|normally distributed]] with mean <math>\mu</math> and variance <math>{\pi}/{2N},</math> i.e.,<ref>Williams, D. (2001) ''Weighing the Odds'', CUP. ISBN 052100618X (p.165) </ref>
:<math>\widetilde{X} \sim \mathcal{N}\left(\mu, \frac{\pi}{2N}\right).</math>
The efficiency for large <math>N</math> is thus
:<math>
e\left(\widetilde{X}\right) = \left(\frac{1}{N}\right) \left(\frac{\pi}{2N}\right)^{-1} = 2/\pi \approx 64%.
</math>
Note that this is the [[asymptote|asymptotic]] efficiency — that is, the efficiency in the limit as sample size <math>N</math> tends to infinity. For finite values of <math>N,</math> the efficiency is higher than this (for example, a sample size of 3 gives an efficiency of about 74%).{{cn|date=April 2013}}

The sample mean is thus more efficient than the sample median in this example. However, there may be measures by which the median performs better. For example, the median is far more robust to [[outlier]]s, so that if the Gaussian model is questionable or approximate, there may advantages to using the median (see [[Robust statistics]]).

===Dominant estimators===
If <math>T_1</math> and <math>T_2</math> are estimators for the parameter <math>\theta</math>, then <math>T_1</math> is said to '''[[dominating decision rule|dominate]]''' <math>T_2</math> if:
# its [[mean squared error]] (MSE) is smaller for at least some value of <math>\theta</math>
# the MSE does not exceed that of <math>T_2</math> for any value of θ.

Formally, <math>T_1</math> dominates <math>T_2</math> if
:<math>
\mathrm{E}
\left[
(T_1 - \theta)^2
\right]
\leq
\mathrm{E}
\left[
(T_2-\theta)^2
\right]
</math>

holds for all <math>\theta</math>, with strict inequality holding somewhere.

===Relative efficiency===
The relative efficiency of two estimators is defined as

:<math>
e(T_1,T_2)
=
\frac
{\mathrm{E} \left[ (T_2-\theta)^2 \right]}
{\mathrm{E} \left[ (T_1-\theta)^2 \right]}
</math>

Although <math>e</math> is in general a function of <math>\theta</math>, in many cases the dependence drops out; if this is so, <math>e</math> being greater than one would indicate that <math>T_1</math> is preferable, whatever the true value of <math>\theta</math>.

An alternative to relative efficiency for comparing estimators, is the [[Pitman closeness criterion]]. This replaces the comparison of mean-squared-errors with comparing how often one estimator produces estimates closer to the true value than another estimator.

===Robustness===
Efficiency of an estimator may change significantly if the distribution changes, often dropping. This is one of the motivations of [[robust statistics]] – an estimator such as the sample mean is an efficient estimator of the population mean of a normal distribution, for example, but can be an inefficient estimator of a [[mixture distribution]] of two normal distributions with the same mean and different variances. For example, if a distribution is a combination of 98% ''N''(''μ,'' ''σ'') and 2% ''N''(''μ,'' 10''σ''), the presence of extreme values from the latter distribution (often "contaminating outliers") significantly reduces the efficiency of the sample mean as an estimator of ''μ.'' By contrast, the trimmed mean is less efficient for a normal distribution, but is more robust (less affected) by changes in distribution, and thus may be more efficient for a mixture distribution. Similarly, the shape of a distribution, such as [[skewness]] or heavy tails, can significantly reduce the efficiency of estimators that assume a symmetric distribution or thin tails.

===Uses of inefficient estimators===
{{see|L-estimator#Applications}}

While efficiency is a desirable quality of an estimator, it must be weighed against other desiderata, and an estimator that is efficient for certain distributions may well be inefficient for other distributions. Most significantly, estimators that are efficient for clean data from a simple distribution, such as the normal distribution (which is symmetric, unimodal, and has thin tails) may not be robust to contamination by outliers, and may be inefficient for more complicated distributions. In [[robust statistics]], more importance is placed on robustness and applicability to a wide variety of distributions, rather than efficiency on a single distribution. [[M-estimator]]s are a general class of solutions motivated by these concerns, yielding both robustness and high relative efficiency, though possibly lower efficiency than traditional estimators for some cases. These are potentially very computationally complicated, however.

A more traditional alternative are [[L-estimator]]s, which are very simple statistics that are easy to compute and interpret, in many cases robust, and often sufficiently efficient for initial estimates. See [[L-estimator#Applications|applications of L-estimators]] for further discussion.

==Hypothesis tests==
For comparing [[significance test]]s, a meaningful measure of efficiency can be defined based on the sample size required for the test to achieve a given [[statistical power|power]].{{sfn|Everitt|2002|loc=p. 321}}

[[Pitman efficiency]]<ref>{{SpringerEOM | title=Efficiency, asymptotic | id=E/e035070 | last=Nikitin | first=Ya.Yu.}}</ref> and [http://www.encyclopediaofmath.org/index.php/Bahadur_efficiency Bahadur efficiency] (or [[Hodges–Lehmann efficiency]] )<ref>Arcones M.A. [http://www.math.binghamton.edu/arcones/prep/pv.pdf "Bahadur efficiency of the likelihood ratio test"] preprint</ref><ref>Canay I.A. & Otsu, T. [http://faculty.wcas.northwestern.edu/~iac879/wp/HL.pdf "Hodges-Lehmann Optimality for Testing Moment Condition Models"]</ref> relate to the comparison of the performance of [[Statistical hypothesis testing]] procedures. The Encyclopedia of Mathematics provides a [http://www.encyclopediaofmath.org/index.php/Efficiency,_asymptotic brief exposition] of these three criteria.

==Experimental design==
{{see|Optimal design}}

For experimental designs, efficiency relates to the ability of a design to achieve the objective of the study with minimal expenditure of resources such as time and money. In simple cases, the relative efficiency of designs can be expressed as the ratio of the sample sizes required to achieve a given objective.<ref>Dodge, Y. (2006) ''The Oxford Dictionary of Statistical Terms'', OUP. ISBN 0-19-920613-9</ref>

See [[optimal design]] for further discussion.

==Notes==
{{reflist}}

==References==
*{{cite isbn|052181099X}}
*{{cite isbn|9780387985954}}
* {{SpringerEOM
| title=Efficiency, asymptotic
| id=E/e035070
| last=Nikitin
| first=Ya.Yu.
}}
{{refend}}

[[Category:Statistical theory]]
[[Category:Statistical terminology]]

[[de:Effizienz (Statistik)]]
[[es:Eficiencia (estadística)]]
[[it:Efficienza (statistica)]]
[[pt:Eficiência (estatística)]]

Anisotropic Network Model - Revision history

2003:63:2420:8691:DC42:37E7:5240:CDB3: /* Theory */

en>Nsda: link -> mean squared displacement