|
|
| Line 1: |
Line 1: |
| {{context|date=February 2012}}
| | Hi there. Allow [http://netwk.hannam.ac.kr/xe/data_2/85669 psychic readers] me begin by introducing the author, her title is Sophia. Ohio is where my home is but my husband desires us to transfer. What me and my family members adore is to climb but I'm thinking on starting some thing new. Office supervising is where her primary income arrives from but she's already utilized for another 1.<br><br>Also visit my web page [http://www.article-galaxy.com/profile.php?a=143251 online psychic] reader [[http://ltreme.com/index.php?do=/profile-127790/info/ http://ltreme.com/]] |
| '''Pointwise mutual information''' ('''PMI'''), or '''point mutual information''', is a measure of association used in [[information theory]] and [[statistics]].
| |
| | |
| ==Definition==
| |
| The PMI of a pair of [[probability space|outcomes]] ''x'' and ''y'' belonging to [[discrete random variable]]s ''X'' and ''Y'' quantifies the discrepancy between the probability of their coincidence given their joint distribution and their individual distributions, assuming independence. Mathematically:
| |
| | |
| : <math>
| |
| \operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}.
| |
| </math>
| |
| | |
| The [[mutual information]] (MI) of the random variables ''X'' and ''Y'' is the expected value of the PMI over all possible outcomes (with respect to the joint distribution <math>p(x,y)</math>).
| |
| | |
| The measure is symmetric (<math>\operatorname{pmi}(x;y)=\operatorname{pmi}(y;x)</math>). It can take positive or negative values, but is zero if ''X'' and ''Y'' are [[statistical independence|independent]]. PMI maximizes when ''X'' and ''Y'' are [[perfectly associated]], yielding the following bounds:
| |
| | |
| :<math>
| |
| -\infty \leq \operatorname{pmi}(x;y) \leq \min\left[ -\log p(x), -\log p(y) \right] .
| |
| </math>
| |
| | |
| Finally, <math>\operatorname{pmi}(x;y)</math> will increase if <math>p(x|y)</math> is fixed but <math>p(x)</math>decreases.
| |
| | |
| Here is an example to illustrate:
| |
| {| border="1" cellpadding="2" class="wikitable"
| |
| !''x''!!''y''!!''p''(''x'', ''y'')
| |
| |-
| |
| |0||0||0.1
| |
| |-
| |
| |0||1||0.7
| |
| |-
| |
| |1||0||0.15
| |
| |-
| |
| |1||1||0.05
| |
| |}
| |
| Using this table we can marginalize to get the following additional table for the individual distributions:
| |
| {| border="1" cellpadding="2" class="wikitable"
| |
| ! !!''p''(''x'')!!''p''(''y'')
| |
| |-
| |
| |0||.8||0.25
| |
| |-
| |
| |1||.2||0.75
| |
| |}
| |
| With this example, we can compute four values for <math>pmi(x;y)</math>. Using base-2 logarithms:
| |
| {| cellpadding="2"
| |
| |-
| |
| |pmi(x=0;y=0)||−1
| |
| |-
| |
| |pmi(x=0;y=1)||0.222392421
| |
| |-
| |
| |pmi(x=1;y=0)||1.584962501
| |
| |-
| |
| |pmi(x=1;y=1)||−1.584962501
| |
| |-
| |
| |}
| |
| | |
| (For reference, the [[mutual information]] <math>\operatorname{I}(X;Y)</math> would then be 0.214170945)
| |
| | |
| ==Similarities to mutual information== | |
| Pointwise Mutual Information has many of the same relationships as the mutual information. In particular,
| |
| | |
| <math>
| |
| \begin{align}
| |
| \operatorname{pmi}(x;y) &=& h(x) + h(y) - h(x,y) \\
| |
| &=& h(x) - h(x|y) \\
| |
| &=& h(y) - h(y|x)
| |
| \end{align}
| |
| </math>
| |
| | |
| Where <math>h(x)</math> is the [[self-information]], or <math>-\log_2 p(X=x)</math>.
| |
| | |
| ==Normalized pointwise mutual information (npmi)==
| |
| Pointwise mutual information can be normalized between [-1,+1] resulting in -1 (in the limit) for never occurring together, 0 for independence, and +1 for complete [[co-occurrence]].
| |
| | |
| <math>
| |
| | |
| \operatorname{npmi}(x;y) = \frac{\operatorname{pmi}(x;y)}{-\log \left[ p(x, y) \right] }
| |
| | |
| </math>
| |
| | |
| ==Chain-rule for pmi==
| |
| Pointwise mutual information follows the [[Chain_rule_%28disambiguation%29|chain rule]], that is,
| |
| :<math>\operatorname{pmi}(x;yz) = \operatorname{pmi}(x;y) + \operatorname{pmi}(x;z|y)</math> | |
| | |
| This is easily proven by:
| |
| :<math>
| |
| \begin{align}
| |
| \operatorname{pmi}(x;y) + \operatorname{pmi}(x;z|y) & {} = \log\frac{p(x,y)}{p(x)p(y)} + \log\frac{p(x,z|y)}{p(x|y)p(z|y)} \\
| |
| & {} = \log \left[ \frac{p(x,y)}{p(x)p(y)} \frac{p(x,z|y)}{p(x|y)p(z|y)} \right] \\
| |
| & {} = \log \frac{p(x|y)p(y)p(x,z|y)}{p(x)p(y)p(x|y)p(z|y)} \\
| |
| & {} = \log \frac{p(x,yz)}{p(x)p(yz)} \\
| |
| & {} = \operatorname{pmi}(x;yz)
| |
| \end{align}
| |
| </math>
| |
| | |
| {{inline|date=February 2012}}
| |
| | |
| ==References==
| |
| * {{cite web|title=Normalized (Pointwise) Mutual Information in Collocation Extraction|url=https://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdf|last1=Bouma|first1=Gerloff|year=2009|publisher=Proceedings of the Biennial GSCL Conference}}
| |
| * {{cite book|last1=Fano|first1=R M|year=1961|title=Transmission of Information: A Statistical Theory of Communications|publisher=MIT Press, Cambridge, MA|chapter=chapter 2}}
| |
| | |
| ==External links==
| |
| * [http://cwl-projects.cogsci.rpi.edu/msr/ Demo at Rensselaer MSR Server] (PMI values normalized to be between 0 and 1)
| |
| | |
| | |
| [[Category:Information theory]]
| |
| [[Category:Statistical dependence]]
| |
| [[Category:Entropy and information]]
| |
Hi there. Allow psychic readers me begin by introducing the author, her title is Sophia. Ohio is where my home is but my husband desires us to transfer. What me and my family members adore is to climb but I'm thinking on starting some thing new. Office supervising is where her primary income arrives from but she's already utilized for another 1.
Also visit my web page online psychic reader [http://ltreme.com/]