|
|
| Line 1: |
Line 1: |
| The chain rule for [[Kolmogorov complexity]] is an analogue of the chain rule for [[information entropy]], which states:
| | They call me Emilia. Bookkeeping is her day occupation now. To do aerobics is a factor that I'm completely addicted to. For years he's been living in North Dakota and his family enjoys it.<br><br>my page: [http://www.teenvalley.net/blog/17832 http://www.teenvalley.net/] |
| | |
| :<math>
| |
| H(X,Y) = H(X) + H(Y|X)
| |
| </math>
| |
| | |
| That is, the combined [[randomness]] of two sequences ''X'' and ''Y'' is the sum of the randomness of ''X'' plus whatever randomness is left in ''Y'' once we know ''X''.
| |
| This follows immediately from the definitions of [[conditional entropy|conditional]] and [[joint entropy]] fact from [[probability theory]] that the [[joint probability]] is the product of the [[marginal probability|marginal]] and [[conditional probability]]:
| |
| | |
| :<math>
| |
| P(X,Y) = P(X) P(Y|X) \,
| |
| </math>
| |
| | |
| The equivalent statement for Kolmogorov complexity does not hold exactly; it is true only up to a [[logarithm]]ic term:
| |
| | |
| :<math>
| |
| K(x,y) = K(x) + K(y|x) + O(\log(K(x,y)))
| |
| </math>
| |
| | |
| (An exact version, ''KP''(''x'', ''y'') = ''KP''(''x'') + ''KP''(''y''|''x''*) + O(1),
| |
| holds for the prefix complexity ''KP'', where ''x*'' is a shortest program for ''x''.)
| |
| | |
| It states that the shortest program printing ''X'' and ''Y'' is obtained by concatenating a shortest program printing ''X'' with a program printing ''Y'' given ''X'', plus [[Big-O notation|at most]] a logarithmic factor. The results implies that [[Mutual information#Absolute mutual information|algorithmic mutual information]], an analogue of mutual information for Kolmogorov complexity is symmetric: ''I(x:y) = I(y:x) + O(log K(x,y))'' for all ''x,y''.
| |
| | |
| ==Proof==
| |
| | |
| The ≤ direction is obvious: we can write a program to produce ''x'' and ''y'' by concatenating a program to produce ''x'', a program to produce ''y'' given
| |
| access to ''x'', and (whence the log term) the length of one of the programs, so
| |
| that we know where to separate the two programs for ''x'' and ''y''|''x'' (log(''K''(''x'', ''y'')) upper-bounds this length).
| |
| | |
| For the ≥ direction, it suffices to show that for all k,l such that k+l = K(x,y) we have that either
| |
| | |
| ''K(x|k,l) ≤ k + O(1)'' or ''K(y|x,k,l) ≤ l + O(1)''.
| |
| | |
| Consider the list ''(a<sub>1</sub>,b<sub>1</sub>), (a<sub>2</sub>,b<sub>2</sub>), ..., (a<sub>e</sub>,b<sub>e</sub>)'' of all pairs ''(a,b)'' produced by programs of length exactly ''K(x,y)'' [hence K(a,b) ≤ K(x,y)]. Note that this list
| |
| * contains the pair ''(x,y)'',
| |
| * can be [[recursively enumerable|enumerated]] given ''k'' and ''l'' (by running all programs of length ''K(x,y)'' in parallel),
| |
| * has at most ''2<sup>K(x,y)</sup>'' elements (because there are at most 2<sup>n</sup> programs of length n).
| |
| | |
| First, suppose that ''x'' appears less than ''2<sup>l</sup>'' times as first element. We can specify ''y'' given ''x,k,l'' by enumerating ''(a<sub>1</sub>,b<sub>1</sub>), (a<sub>2</sub>,b<sub>2</sub>), ...'' and then selecting ''(x,y)'' in the sub-list of pairs ''(x,b)''. By assumption, the index of ''(x,y)'' in this sub-list is less than ''2<sup>l</sup>'' and hence, there is a program for ''y'' given ''x,k,l'' of length ''l + O(1)''.
| |
| Now, suppose that ''x'' appears at least ''2<sup>l</sup>'' times as first element. This can happen for at most ''2<sup>K(x,y)-l</sup> = 2<sup>k</sup>'' different strings. These strings can be enumerated given ''k,l'' and hence ''x'' can be specified by its index in this enumeration. The corresponding program for ''x'' has size ''k + O(1)''. Theorem proved.
| |
| | |
| | |
| ==References==
| |
| * {{cite book
| |
| | last = Li
| |
| | first = Ming
| |
| | coauthors = Vitányi, Paul
| |
| | title = An introduction to Kolmogorov complexity and its applications
| |
| | location = New York
| |
| | publisher = [[Springer-Verlag]]
| |
| |date=February 1997
| |
| | isbn = 0-387-94868-6 }}
| |
| | |
| * {{cite article
| |
| | last = Kolmogorov
| |
| | first = A.N.
| |
| | title = Logical basis for information theory and probability theory
| |
| | journal = IEEE Transactions on Information Theory
| |
| | number = 5
| |
| | volume = 14
| |
| | pages = 662-664
| |
| | year = 1968 }}
| |
| | |
| * {{cite article
| |
| | first = A.
| |
| | last = Zvonkin
| |
| | coauthors = L. Levin
| |
| | title = The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms.
| |
| | journal = Russian Mathematical Surveys
| |
| | volume = 25
| |
| | number = 6
| |
| | pages = 83-124
| |
| | year = 1970}}
| |
| | |
| | |
| [[Category:Algorithmic information theory|*]]
| |
| [[Category:Information theory|*]]
| |
| [[Category:Computability theory]]
| |
| [[Category:Theory of computation]]
| |
| [[Category:Articles containing proofs]]
| |
They call me Emilia. Bookkeeping is her day occupation now. To do aerobics is a factor that I'm completely addicted to. For years he's been living in North Dakota and his family enjoys it.
my page: http://www.teenvalley.net/