en>GregorB: /* References */ WL

2014-12-23T09:37:00Z

References: WL

en>BG19bot: WP:CHECKWIKI error fix for #99. Broken sup tag. Do general fixes if a problem exists. -, replaced: ^{op → ^op using AWB (9957)}

2014-03-01T08:57:22Z

WP:CHECKWIKI error fix for #99. Broken sup tag. Do general fixes if a problem exists. -, replaced: op → op using AWB (9957)

Show changes

en>Jakob.scholbach: /* References */ add a ref

2012-08-01T13:33:43Z

References: add a ref

New page

{{Bayesian statistics}}

In [[statistics]], the use of '''Bayes factors''' is a [[Bayesian probability|Bayesian]] alternative to classical [[hypothesis testing]].<ref name=Goodman1999a>{{cite journal | author = Goodman S | title = Toward evidence-based medical statistics. 1: The P value fallacy | journal = Ann Intern Med | volume = 130 | issue = 12 | pages = 995–1004 | year = 1999 | pmid = 10383371 | url= http://www.annals.org/cgi/reprint/130/12/995.pdf | format = PDF | doi=10.7326/0003-4819-130-12-199906150-00008}}</ref><ref name=Goodman1999b>{{cite journal | author = Goodman S | title = Toward evidence-based medical statistics. 2: The Bayes factor | journal = Ann Intern Med | volume = 130 | issue = 12 | pages = 1005–13 | year = 1999 | pmid = 10383350 | url=http://www.annals.org/cgi/reprint/130/12/1005.pdf | format = PDF}}</ref> '''Bayesian model comparison''' is a method of [[model selection]] based on Bayes factors.

==Definition==
The [[posterior probability]] Pr(''M''|''D'') of a model ''M'' given data ''D'' is given by [[Bayes' theorem]]:

:<math>\Pr(M|D) = \frac{\Pr(D|M)\Pr(M)}{\Pr(D)}.</math>

The key data-dependent term Pr(''D''|''M'') is a [[likelihood function|likelihood]], and represents the probability that some data is produced under the assumption of this model, ''M''; evaluating it correctly is the key to Bayesian model comparison.

Given a [[model selection]] problem in which we have to choose between two models, on the basis of observed data ''D'', the plausibility of the two different models ''M''1 and ''M''2, parametrised by model parameter vectors <math> \theta_1 </math> and <math> \theta_2 </math> is assessed by the '''Bayes factor''' ''K'' given by

:<math> K = \frac{\Pr(D|M_1)}{\Pr(D|M_2)}
= \frac{\int \Pr(\theta_1|M_1)\Pr(D|\theta_1,M_1)\,d\theta_1}
{\int \Pr(\theta_2|M_2)\Pr(D|\theta_2,M_2)\,d\theta_2} .
</math>

If instead of the Bayes factor integral, the likelihood corresponding to the [[Maximum likelihood|maximum likelihood estimate]] of the parameter for each model is used, then the test becomes a classical [[likelihood-ratio test]].{{Citation needed|date=February 2011}}
Unlike a likelihood-ratio test, this Bayesian model comparison does not depend on any single set of parameters, as it integrates over all parameters in each model (with respect to the respective priors). However, an advantage of the use of Bayes factors is that it automatically, and quite naturally, includes a penalty for including too much model structure.<ref name=kassraftery1995>{{Cite journal | author = Robert E. Kass and Adrian E. Raftery |year=1995|title=Bayes Factors|url=http://www.andrew.cmu.edu/user/kk3n/simplicity/KassRaftery1995.pdf|journal=Journal of the American Statistical Association|volume= 90 |number= 430|p= 791}}</ref> It thus guards against [[overfitting]]. For models where an explicit version of the likelihood is not available or too costly to evaluate numerically, [[approximate Bayesian computation]] can be used for model selection in a Bayesian framework,<ref name= Toni2009b>{{cite journal |author = Toni, T.; Stumpf, M.P.H. |year = 2009 |title = Simulation-based model selection for dynamical systems in systems and population biology | journal = Bioinformatics |volume = 26|pages = 104–10 |doi = 10.1093/bioinformatics/btp619 |url=http://bioinformatics.oxfordjournals.org/cgi/reprint/26/1/104.pdf|format=PDF |pmid = 19880371 |issue = 1 |pmc = 2796821 }}</ref>
with the caveat that approximate-Bayesian estimates of Bayes factors are often biased.<ref name=Robert2011>{{cite journal | author = Robert, C.P., J. Cornuet, J. Marin and N.S. Pillai | year = 2011 | title = Lack of confidence in approximate Bayesian computation model choice | journal = Proceedings of the National Academy of Sciences | volume = 108 | issue = 37 | pages = 15112--15117 | doi = 10.1073/pnas.1102900108 | url = http://www.pnas.org/content/early/2011/08/25/1102900108.short | pmid = 21876135}}</ref>

Other approaches are:
* to treat model comparison as a [[Decision theory#Choice under uncertainty|decision problem]], computing the expected value or cost of each model choice;
* to use [[minimum message length]] (MML).

==Interpretation==
A value of ''K'' > 1 means that ''M''1 is more strongly supported by the data under consideration than ''M''2. Note that classical [[hypothesis testing]] gives one hypothesis (or model) preferred status (the 'null hypothesis'), and only considers evidence ''against'' it. [[Harold Jeffreys]] gave a scale for interpretation of ''K'':<ref>{{Cite book | url = http://books.google.ca/books?id=vh9Act9rtzQC&printsec=frontcover#v=onepage&q&f=false | author=H. Jeffreys|title=The Theory of Probability|edition=3|location= Oxford|year=1961}} p. 432</ref>

:{|
! K !! dB !! bits !! Strength of evidence
|-
! < 1:1
| <center> < 0 </center>
|
| Negative (supports M2)
|-
!1:1 to 3:1
| <center>0 to 5</center>
| <center>0 to 1.6</center>
| Barely worth mentioning
|-
!3:1 to 10:1
| <center>5 to 10</center>
| <center>1.6 to 3.3</center>
| Substantial
|-
!10:1 to 30:1
| <center>    10 to 15    </center>
| <center>    3.3 to 5.0    </center>
| Strong
|-
!30:1 to 100:1
| <center>15 to 20</center>
| <center>5.0 to 6.6</center>
| Very strong
|-
!> 100:1
| <center>> 20</center>
| <center> > 6.6 </center>
| Decisive
|}

The second column gives the corresponding weights of evidence in [[deciban]]s (tenths of a power of 10); [[bit]]s are added in the third column for clarity. According to [[I. J. Good]] a change in a weight of evidence of 1 deciban or 1/3 of a bit (i.e. a change in an odds ratio from evens to about 5:4) is about as finely as [[human]]s can reasonably perceive their [[Bayesian probability|degree of belief]] in a hypothesis in everyday use.<ref>{{cite journal|last=Good|first=I.J.|authorlink=I. J. Good|title=Studies in the History of Probability and Statistics. XXXVII A. M. Turing's statistical work in World War II|journal=[[Biometrika]]|year=1979|volume=66|issue=2|pages=393–396|doi=10.1093/biomet/66.2.393|mr=82c:01049}}</ref>

An alternative table, widely cited, is provided by Kass and Raftery (1995):<ref name=kassraftery1995/>

:{|
! 2 ln K !! K !! Strength of evidence
|-
! 0 to 2
| <center> 1 to 3 </center>
| Not worth more than a bare mention
|-
! 2 to 6
| <center>3 to 20</center>
| Positive
|-
! 6 to 10
| <center>20 to 150</center>
| Strong
|-
! >10
| <center>>150</center>
| Very strong
|}

The use of Bayes factors or classical hypothesis testing takes place in the context of [[inference]] rather than [[decision theory|decision-making under uncertainty]]. That is, we merely wish to find out which hypothesis is true, rather than actually making a decision on the basis of this information. [[Frequentist statistics]] draws a strong distinction between these two because classical hypothesis tests are not [[Coherence (philosophical gambling strategy)|coherent]] in the Bayesian sense. Bayesian procedures, including Bayes factors, are coherent, so there is no need to draw such a distinction. Inference is then simply regarded as a special case of decision-making under uncertainty in which the resulting action is to report a value. For decision-making, Bayesian statisticians might use a Bayes factor combined with a [[prior distribution]] and a [[loss function]] associated with making the wrong choice. In an inference context the loss function would take the form of a [[scoring rule]]. Use of a [[scoring rule|logarithmic score function]] for example, leads to the expected [[utility]] taking the form of the [[Kullback–Leibler divergence]].

==Example==
Suppose we have a [[random variable]] that produces either a success or a failure. We want to compare a model ''M''1 where the probability of success is ''q'' = ½, and another model ''M''2 where ''q'' is completely unknown and we take a [[prior distribution]] for ''q'' which is [[uniform distribution (continuous)|uniform]] on [0,1]. We take a sample of 200, and find 115 successes and 85 failures. The likelihood can be calculated according to the [[binomial distribution]]:

:<math>{{200 \choose 115}q^{115}(1-q)^{85}}.</math>

So we have

:<math>P(X=115|M_1)={200 \choose 115}\left({1 \over 2}\right)^{200}=0.005956...,\,</math>

but

:<math>P(X=115|M_2)=\int_{0}^1{200 \choose 115}q^{115}(1-q)^{85}dq = {1 \over 201} = 0.004975...\,.</math>

The ratio is then 1.197..., which is "barely worth mentioning" even if it points very slightly towards ''M''1.

This is not the same as a classical likelihood ratio test, which would have found the [[maximum likelihood]] estimate for ''q'', namely 115⁄200 = 0.575, and used that to get a ratio of 0.1045... (rather than averaging over all possible ''q''), and so pointing towards ''M''2. Alternatively, [[A. W. F. Edwards|Edwards]]'s "exchange rate"{{citation needed|date=December 2010}} of two units of likelihood per degree of freedom suggests that <Math>M_2</math> is preferable (just) to <math>M_1</math>, as <math>0.1045\ldots = e^{-2.25\ldots}</math> and <math>2.25>2</math>: the extra likelihood compensates for the unknown parameter in <math>M_2</math>.

A [[frequentist]] [[Statistical hypothesis testing|hypothesis test]] of <math>M_1</math> (here considered as a [[null hypothesis]]) would have produced a more dramatic result, saying that ''M''1 could be rejected at the 5% significance level, since the probability of getting 115 or more successes from a sample of 200 if ''q'' = ½ is 0.0200..., and as a two-tailed test of getting a figure as extreme as or more extreme than 115 is 0.0400... Note that 115 is more than two standard deviations away from 100.

''M''2 is a more complex model than ''M''1 because it has a free parameter which allows it to model the data more closely. The ability of Bayes factors to take this into account is a reason why [[Bayesian inference]] has been put forward as a theoretical justification for and generalisation of [[Occam's razor]], reducing [[Type I error]]s.<ref>[http://www.stat.duke.edu/~berger/papers/ockham.html Sharpening Ockham's Razor On a Bayesian Strop]</ref>

==See also==
* [[Akaike information criterion]]
* [[Approximate Bayesian Computation]]
* [[Deviance information criterion]]
* [[Model selection]]
* Schwarz's [[Bayesian information criterion]]
* [[Chris Wallace (computer scientist)|Wallace]]'s [[Minimum Message Length]] (MML)
;Statistical ratios
* [[Odds ratio]]
* [[Relative risk]]

==References==
{{reflist}}
* Gelman, A., Carlin, J., Stern, H. and Rubin, D. Bayesian Data Analysis. Chapman and Hall/CRC.(1995)
* Bernardo, J. and Smith, A.F.M., Bayesian Theory. John Wiley. (1994)
* Lee, P.M. Bayesian Statistics. Arnold.(1989).
* Denison, D.G.T., Holmes, C.C., Mallick, B.K., Smith, A.F.M., Bayesian Methods for Nonlinear Classification and Regression. John Wiley. (2002).
* Richard O. Duda, Peter E. Hart, David G. Stork (2000) ''Pattern classification'' (2nd edition), Section 9.6.5, p. 487-489, Wiley, ISBN 0-471-05669-3
* Chapter 24 in [http://omega.math.albany.edu:8008/JaynesBook.html Probability Theory - The logic of science] by [[Edwin Thompson Jaynes|E. T. Jaynes]], 1994.
* [[David J.C. MacKay]] (2003) Information theory, inference and learning algorithms, CUP, ISBN 0-521-64298-1, (also [http://www.inference.phy.cam.ac.uk/mackay/itila/book.html available online])
* Winkler, Robert, ''Introduction to Bayesian Inference and Decision, 2nd Edition'' (2003), Probabilistic. ISBN 0-9647938-4-9.

== External links ==
* [http://www.cs.ucsd.edu/users/goguen/courses/275f00/stat.html Bayesian critique of classical hypothesis testing]
* [http://pcl.missouri.edu/bayesfactor Web-based Bayes-factor calculator for t-tests, regression designs, and binomially distributed data]
* [http://bayesfactorpcl.r-forge.r-project.org/ BayesFactor, an R package for computing Bayes factors in common research designs]
* [http://www.inference.phy.cam.ac.uk/mackay/itila/ The on-line textbook: Information Theory, Inference, and Learning Algorithms], by [[David J.C. MacKay]], discusses Bayesian model comparison in Chapter 28, p343.

{{DEFAULTSORT:Bayes Factor}}
[[Category:Bayesian inference|Factor]]
[[Category:Model selection]]
[[Category:Statistical ratios]]

Inverse system - Revision history

en>GregorB: /* References */ WL

en>BG19bot: WP:CHECKWIKI error fix for #99. Broken sup tag. Do general fixes if a problem exists. -, replaced: op → op using AWB (9957)

en>Jakob.scholbach: /* References */ add a ref

en>BG19bot: WP:CHECKWIKI error fix for #99. Broken sup tag. Do general fixes if a problem exists. -, replaced: ^{op → ^op using AWB (9957)}