|
|
| Line 1: |
Line 1: |
| The '''Brier score''' is a [[scoring rule|proper score function]] that measures the accuracy of probabilistic predictions. It is applicable to tasks in which predictions must assign probabilities to a set of [[mutually exclusive]] discrete outcomes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1). It was proposed by Glenn W. Brier in 1950.<ref name="Brier">
| | Oscar is how he's known as and he completely enjoys this title. California is where I've usually been living and I love each day residing here. To play baseball is the hobby he will by no means quit doing. She is a librarian but she's usually needed her personal business.<br><br>Also visit my blog; home std test kit ([http://www.hard-ass-porn.com/blog/111007 just click the following page]) |
| | |
| {{Cite journal | author = Brier
| |
| | year = 1950
| |
| | month = | title = Verification of Forecasts Expressed in Terms of Probability
| |
| | journal = Monthly Weather Review
| |
| | volume = 78
| |
| | issue =
| |
| | pages = 1–3
| |
| | id =
| |
| | url = http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf
| |
| }}</ref>
| |
| | |
| The Brier score can be thought of as either a measure of the "calibration" of a set of probabilistic predictions, or as a "[[Loss function|cost function]]". More precisely, across all items <math>i\in{1...N}</math> in a set ''N'' predictions, the Brier score measures the mean squared difference between:
| |
| * The predicted probability assigned to the possible outcomes for item ''i''
| |
| * The actual outcome <math>o_i</math>
| |
| | |
| Therefore, the lower the Brier score is for a set of predictions, the better the predictions are calibrated. Note that the Brier score, in its most common formulation, takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). The original (1950) formulation of the Brier score, the range is double, from zero to two.
| |
| | |
| The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently "distant" from one another).
| |
| | |
| == Definition of the Brier score ==
| |
| The most common formulation of the Brier score is
| |
| :<math>BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2 \,\!</math>
| |
| | |
| In which <math>f_t</math> is the probability that was forecast, <math>o_t</math> the actual outcome of the event at instance t (0 if it does not happen and 1 if it does happen) and N is the number of forecasting instances. This formulation is mostly used for binary events (for example "rain" or "no rain"). The above equation is only a proper scoring rule for binary events; if a multi-category forecast is to be evaluated, then the original definition given by Brier below should be used.
| |
| | |
| ===Example===
| |
| Suppose it is required to give a probability P forecast of a binary event—such as a forecast of rain. The forecast issued says that there is a probability P that the event will occur. Let ''X'' = 1 if the event occurs and X = 0 if it doesn’t.
| |
| | |
| Then the Brier score is given by: .
| |
| * If you forecast 100% (P = 1) and there is at least 1 mm of rain in the bucket, your Brier Score is 0, the best score achievable.
| |
| * If you forecast 100% P and there is no rain in the bucket, your Brier Score is 1, the worst score achievable.
| |
| * If you forecast 70% P and there is at least 1 mm of rain in the bucket, your Brier Score is <math>(0.70-1)^2 = 0.09</math>.
| |
| * If you forecast 30% P and there is at least 1 mm of rain in the bucket, your Brier Score is <math>(0.30-1)^2 = 0.49</math>.
| |
| * If you hedge your forecast with a 50% P and whether or not there is at least 1 mm of rain in the bucket, your Brier score is 0.25.
| |
| | |
| In weather forecasting, a trace (< 0.01) is considered "0.0"
| |
| | |
| ===Original definition by Brier===
| |
| Although the above formulation is the most widely used, the original definition by Brier <ref name="Brier" /> is applicable to multi-category forecasts as well as it remains a proper scoring rule, while the binary form is only proper for binary events. For binary forecasts the original formulation of Brier's "probability score" has twice the value of the score currently known as the Brier score.
| |
| | |
| :<math>BS = \frac{1}{N}\sum\limits _{t=1}^{N}\sum\limits _{i=1}^{R}(f_{ti}-o_{ti})^2 \,\!</math>
| |
| | |
| In which R is the number of possible classes in which the event can fall. For the case Rain / No rain, R=2, while for the forecast Cold / Normal / Warm, R=3.
| |
| | |
| ==Decompositions==
| |
| There are several decompositions of the Brier score which provide a deeper insight on the behavior of a binary classifier.
| |
| | |
| === 3-component decomposition ===
| |
| The Brier score can be decomposed into 3 additive components: Uncertainty, Reliability, and Resolution. (Murphy 1973)<ref name="Murphy1973">
| |
| {{Cite journal
| |
| | author = Murphy, A. H.
| |
| | year = 1973
| |
| | month =
| |
| | title = '''A new vector partition of the probability score'''
| |
| | journal = Journal of Applied Meteorology
| |
| | volume = 12
| |
| | issue = 4
| |
| | pages = 595–600
| |
| | id =
| |
| | url = http://journals.ametsoc.org/doi/pdf/10.1175/1520-0450(1973)012%3C0595%3AANVPOT%3E2.0.CO%3B2
| |
| | doi=10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
| |
| }}</ref>
| |
| | |
| :<math> BS=REL-RES+UNC</math>
| |
| | |
| :<math> BS=\frac{1}{N}\sum\limits _{k=1}^{K}{n_{k}(\mathbf{f_{k}}-\mathbf{\bar{o}}_{\mathbf{k}})}^{2}-\frac{1}{N}\sum\limits _{k=1}^{K}{n_{k}(\mathbf{\bar{o}_{k}}-\bar{\mathbf{o}})}^{2}+\mathbf{\bar{o}}\left({1-\mathbf{\bar{o}}}\right)</math>
| |
| | |
| With <math>\textstyle N</math> being the total number of forecasts issued, <math>\textstyle K</math> the number of unique forecasts issued, <math>\mathbf{\bar{o}}={\sum_{t=1}^{N}}\mathbf{{o_t}}/N</math> the observed climatological base rate for the event to occur, <math> n_{k}</math> the number of forecasts with the same probability category and <math>\mathbf{\overline{o}}_{\mathbf{k}}</math> the observed frequency, given forecasts of probability <math>\mathbf{f_{k}}</math>. The bold notation is in the above formula indicates vectors, which is another way of denoting the original definition of the score. For example, a 70% chance of rain and an occurrence of no rain are denoted as <math>\mathbf{{f}}=(0.3,0.7)</math> and <math>\mathbf{{o}}=(1,0)</math> respectively.
| |
| | |
| ====Uncertainty====
| |
| The uncertainty term measures the inherent uncertainty in the event. For binary events, it is at a maximum when the event occurs 50% of the time and the uncertainty is zero if the event always occurs.
| |
| | |
| ====Reliability====
| |
| The reliability term measures how close the forecast probabilities are to the true probabilities, given that forecast. Strangely enough, the reliability is defined in the contrary direction compared to [[English language]]. If the reliability is 0, the forecast is perfectly reliable. For example, if we group all forecast instances where 80% chance of rain was forecast, we get a perfect reliability only if it rained 4 out of 5 times after such a forecast was issued.
| |
| | |
| ====Resolution====
| |
| The resolution term measures how much the conditional probabilities given the different forecasts differ from the climatic average. The higher this term is the better. In the worst case, when the climatic probability is always forecast, the resolution is zero. In the best case, when the conditional probabilities are zero and one, the resolution is equal to the uncertainty.
| |
| | |
| === Two-component decomposition ===
| |
| An alternative (and related) decomposition generates two terms instead of three.
| |
| | |
| :<math> BS=CAL + REF</math>
| |
| | |
| :<math> BS=\frac{1}{N}\sum\limits _{k=1}^{K}{n_{k}(\mathbf{f_{k}}-\mathbf{\bar{o}}_{\mathbf{k}})}^{2}+\frac{1}{N}\sum\limits _{k=1}^{K}{ n_{k}(\mathbf{\bar{o}_{k}} (1 - \mathbf{\bar{o}_{k}} } ) )</math>
| |
| | |
| The first term is known as calibration (and can be used as a measure of calibration, see [[Calibration (statistics)|statistical calibration]]), as is equal to reliability. The second term is known as refinement, and it is an aggregation of resolution and uncertainty, and is related to the area under the ROC Curve.
| |
| | |
| The Brier Score, and the CAL + REF decomposition, can be represented graphically through the so-called Brier Curves,<ref name="hernandez2011brier">{{cite conference |first1=J.|last1=Hernandez-Orallo | first2=P.A.| last2=Flach |first3=C. | last3=Ferri |year=2011 |title=Brier curves: a new cost-based visualisation of classifier performance|booktitle=Proceedings of the 28th International Conference on Machine Learning (ICML-11)|pages=585–592|url=http://www.icml-2011.org/papers/366_icmlpaper.pdf.}}</ref> where the expected loss is shown for each operating condition. This makes the Brier Score a measure of aggregated performance under a uniform distribution of class asymmetries.<ref name="hernandez2012unified ">{{cite journal |first1=J.| last1= Hernandez-Orallo| first2=P.A.| last2=Flach | first3=C. | last3=Ferri |year=2012 |title=A unified view of performance metrics: translating threshold choice into expected classification loss|journal=Journal of Machine Learning Research|volume=13 |pages=2813–2869 |url=http://jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf}}</ref>
| |
| | |
| == See also ==
| |
| *[[Mean squared error]]
| |
| *[[Scoring rule]]
| |
| | |
| == References ==
| |
| {{Reflist}}
| |
| * [http://armstrong.wharton.upenn.edu/dictionary/definitions/brier%20score.html J. Scott Armstrong, ''Principles of Forecasting''].
| |
| * [http://amsglossary.allenpress.com/glossary/browse?s=b&p=43 AMS Glossary of Meteorology]
| |
| | |
| [[Category:Probability assessment]]
| |
Oscar is how he's known as and he completely enjoys this title. California is where I've usually been living and I love each day residing here. To play baseball is the hobby he will by no means quit doing. She is a librarian but she's usually needed her personal business.
Also visit my blog; home std test kit (just click the following page)