|
|
| Line 1: |
Line 1: |
| In [[physics]], '''maximum entropy thermodynamics''' (colloquially, ''MaxEnt'' [[thermodynamics]]) views [[equilibrium thermodynamics]] and [[statistical mechanics]] as [[Inference#Inference and uncertainty|inference]] processes. More specifically, MaxEnt applies inference techniques rooted in [[Shannon information theory]], [[Bayesian probability]], and the [[principle of maximum entropy]]. These techniques are relevant to any situation requiring prediction from incomplete or insufficient data (e.g., [[image processing|image reconstruction]], [[signal processing]], [[spectral density estimation|spectral analysis]], and [[inverse problem]]s). MaxEnt thermodynamics began with two papers [[Edwin Thompson Jaynes|Edwin T. Jaynes]] published in the 1957 ''Physical Review''.
| | Andera is what you can contact her but she by no means truly favored that name. Mississippi is the only place I've been residing in but I will have to transfer in a year or two. As a woman what she really likes is fashion [http://165.132.39.93/xe/visitors/372912 psychic readings] and she's been [http://breenq.com/index.php?do=/profile-1144/info/ best psychics] doing it for fairly a while. For many years she's been operating as a journey agent.<br><br>Feel free to surf to my web site - [http://www.weddingwall.com.au/groups/easy-advice-for-successful-personal-development-today/ online psychic readings] |
| | |
| == Maximum Shannon entropy ==
| |
| Central to the MaxEnt thesis is the [[principle of maximum entropy]]. It demands as given some partly specified model and some specified data related to the model. It selects a preferred probability distribution to represent the model. The given data state "testable information"<ref>Jaynes, E.T. (1968), p. 229.</ref><ref>Jaynes, E.T. (1979), pp. 30, 31, 40.</ref> about the [[probability distribution]], for example particular [[Expected value|expectation]] values, but are not in themselves sufficient to uniquely determine it. The principle states that one should prefer the distribution which maximizes the [[Shannon entropy|Shannon information entropy]].
| |
| | |
| :<math>S_I = - \sum p_i \ln p_i</math>
| |
| | |
| This is known as the [[Gibbs algorithm]], having been introduced by [[J. Willard Gibbs]] in 1878, to set up [[statistical ensemble]]s to predict the properties of thermodynamic systems at equilibrium. It is the cornerstone of the statistical mechanical analysis of the thermodynamic properties of equilibrium systems (see [[partition function (statistical mechanics)|partition function]]).
| |
| | |
| A direct connection is thus made between the equilibrium [[thermodynamic entropy]] ''S<sub>Th</sub>'', a [[state function]] of pressure, volume, temperature, etc., and the [[information entropy]] for the predicted distribution with maximum uncertainty conditioned only on the expectation values of those variables:
| |
| | |
| :<math>S_{Th}(P,V,T,...)_{(eqm)} = k_B \, S_I(P,V,T,...)</math>
| |
| | |
| ''k<sub>B</sub>'', [[Boltzmann's constant]], has no fundamental physical significance here, but is necessary to retain consistency with the previous historical definition of entropy by [[Clausius]] (1865) (see [[Boltzmann's constant]]).
| |
| | |
| However, the MaxEnt school argue that the MaxEnt approach is a general technique of statistical inference, with applications far beyond this. It can therefore also be used to predict a distribution for "trajectories" Γ "over a period of time" by maximising:
| |
| | |
| :<math>S_I = - \sum p_{\Gamma} \ln p_{\Gamma}</math>
| |
| | |
| This "information entropy" does ''not'' necessarily have a simple correspondence with thermodynamic entropy. But it can be used to predict features of [[non-equilibrium thermodynamics|nonequilibrium thermodynamic]] systems as they evolve over time.
| |
| | |
| In the field of [[near-equilibrium thermodynamics]], the [[Onsager reciprocal relations]] and the [[Green-Kubo relations]] fall out very directly. The approach also creates a solid theoretical framework for the study of [[far-from-equilibrium thermodynamics]], making the derivation of the [[fluctuation theorem|entropy production fluctuation theorem]] particularly straightforward. Practical calculations for most far-from-equilibrium systems remain very challenging, however.
| |
| | |
| ''Technical note'': For the reasons discussed in the article [[differential entropy]], the simple definition of Shannon entropy ceases to be directly applicable for [[random variable]]s with continuous [[probability distribution function]]s. Instead the appropriate quantity to maximise is the "relative information entropy,"
| |
| | |
| :<math>H_c=-\int p(x)\log\frac{p(x)}{m(x)}\,dx.</math>
| |
| | |
| ''H<sub>c</sub>'' is the negative of the [[Kullback-Leibler divergence]], or discrimination information, of ''m''(''x'') from ''p''(''x''), where ''m''(''x'') is a prior [[invariant measure]] for the variable(s). The relative entropy ''H<sub>c</sub>'' is always less than zero, and can be thought of as (the negative of) the number of [[bit]]s of uncertainty lost by fixing on ''p''(''x'') rather than ''m''(''x''). Unlike the Shannon entropy, the relative entropy ''H<sub>c</sub>'' has the advantage of remaining finite and well-defined for continuous ''x'', and invariant under 1-to-1 coordinate transformations. The two expressions coincide for [[discrete probability distribution]]s, if one can make the assumption that ''m''(''x''<sub>i</sub>) is uniform - i.e. the [[principle of equal a-priori probability]], which underlies statistical thermodynamics.
| |
| | |
| == Philosophical Implications ==
| |
| | |
| Adherents to the MaxEnt viewpoint take a clear position on some of the [[philosophy of thermal and statistical physics|conceptual/philosophical questions]] in thermodynamics. This position is sketched below.
| |
| | |
| === The nature of the probabilities in statistical mechanics ===
| |
| | |
| Jaynes (1985,<ref name="Jaynes 1985">Jaynes, E.T. (1985).</ref> 2003,<ref name="Jaynes 2003">Jaynes, E.T. (2003).</ref> ''et passim'') discussed the concept of probability. According to the MaxEnt viewpoint, the probabilities in statistical mechanics are determined jointly by two factors: by respectively specified particular models for the underlying state space (e.g. Liouvillian [[phase space]]); and by respectively specified particular partial descriptions of the system (the macroscopic description of the system used to constrain the MaxEnt probability assignment). The probabilities are [[Objectivity (science)|objective]] in the sense that, given these inputs, a uniquely defined probability distribution will result, the same for every rational investigator, independent of the subjectivity or arbitrary opinion of particular persons. The probabilities are epistemic in the sense that they are defined in terms of specified data and derived from those data by definite and objective rules of inference, the same for every rational investigator.<ref>Jaynes, E.T. (1979), p. 28.</ref> Here the word epistemic, which refers to objective and impersonal scientific knowledge, the same for every rational investigator, is used in the sense that contrasts it with opiniative, which refers to the subjective or arbitrary beliefs of particular persons; this contrast was used by [[Plato]] and [[Aristotle]], and stands reliable today.
| |
| | |
| Jaynes also used the word 'subjective' in this context because others have used it in this context. He accepted that in a sense, a state of knowledge has a subjective aspect, simply because it refers to thought, which is a mental process. But he emphasized that the principle of maximum entropy refers only to thought which is rational and objective, independent of the personality of the thinker. In general, from a philosophical viewpoint, the words 'subjective' and 'objective' are not contradictory; often an entity has both subjective and objective aspects. Jaynes explicitly rejected the criticism of some writers that, just because one can say that thought has a subjective aspect, thought is automatically non-objective. He explicitly rejected subjectivity as a basis for scientific reasoning, the epistemology of science; he required that scientific reasoning have a fully and strictly objective basis.<ref>Jaynes, E.T. (1968), p. 228.</ref> Nevertheless, critics continue to attack Jaynes, alleging that his ideas are "subjective". One writer even goes so far as to label Jaynes' approach as "ultrasubjectivist",<ref>Guttmann, Y.M. (1999), pp. 28, 36, 38, 57, 61.</ref> and to mention "the panic that the term subjectivism created amongst physicists".<ref>Guttmann, Y.M. (1999), p. 29.</ref>
| |
| | |
| The probabilities represent both the degree of knowledge and lack of information in the data and the model used in the analyst's macroscopic description of the system, and also what those data say about the nature of the underlying reality.
| |
| | |
| The fitness of the probabilities depends on whether the constraints of the specified macroscopic model are a sufficiently accurate and/or complete description of the system to capture all of the experimentally reproducible behaviour. This cannot be guaranteed, ''a priori''. For this reason MaxEnt proponents also call the method '''predictive statistical mechanics'''. The predictions can fail. But if they do, this is informative, because it signals the presence of new constraints needed to capture reproducible behaviour in the system, which had not been taken into account.
| |
| | |
| === Is entropy "real" ? ===
| |
| | |
| The thermodynamic entropy (at equilibrium) is a function of the state variables of the model description. It is therefore as "real" as the other variables in the model description. If the model constraints in the probability assignment are a "good" description, containing all the information needed to predict reproducible experimental results, then that includes all of the results one could predict using the formulae involving entropy from classical thermodynamics. To that extent, the MaxEnt ''S<sub>Th</sub>'' is as "real" as the entropy in classical thermodynamics.
| |
| | |
| Of course, in reality there is only one real state of the system. The entropy is not a direct function of that state. It is a function of the real state only through the (subjectively chosen) macroscopic model description.
| |
| | |
| === Is ergodic theory relevant ? ===
| |
| | |
| The Gibbsian [[statistical ensemble|ensemble]] idealises the notion of repeating an experiment again and again on ''different'' systems, not again and again on the ''same'' system. So long-term time averages and the [[ergodic hypothesis]], despite the intense interest in them in the first part of the twentieth century, strictly speaking are not relevant to the probability assignment for the state one might find the system in.
| |
| | |
| However, this changes if there is additional knowledge that the system is being prepared in a particular way some time before the measurement. One must then consider whether this gives further information which is still relevant at the time of measurement. The question of how 'rapidly mixing' different properties of the system are then becomes very much of interest. Information about some degrees of freedom of the combined system may become unusable very quickly; information about other properties of the system may go on being relevant for a considerable time.
| |
| | |
| If nothing else, the medium and long-run time correlation properties of the system are interesting subjects for experimentation in themselves. Failure to accurately predict them is a good indicator that relevant macroscopically determinable physics may be missing from the model.
| |
| | |
| === The Second Law ===
| |
| | |
| According to [[Liouville's theorem (Hamiltonian)|Liouville's theorem]] for [[Hamiltonian dynamics]], the hyper-volume of a cloud of points in [[phase space]] remains constant as the system evolves. Therefore, the information entropy must also remain constant, if we condition on the original information, and then follow each of those microstates forward in time:
| |
| | |
| :<math>\Delta S_I = 0 \,</math>
| |
| | |
| However, as time evolves, that initial information we had becomes less directly accessible. Instead of being easily summarisable in the macroscopic description of the system, it increasingly relates to very subtle correlations between the positions and momenta of individual molecules. (Compare to Boltzmann's [[H-theorem]].) Equivalently, it means that the probability distribution for the whole system, in 6N-dimensional phase space, becomes increasingly irregular, spreading out into long thin fingers rather than the initial tightly defined volume of possibilities.
| |
| | |
| Classical thermodynamics is built on the assumption that entropy is a [[state function]] of the [[macroscopic variable]]s—i.e., that none of the history of the system matters, so that it can all be ignored.
| |
| | |
| The extended, wispy, evolved probability distribution, which still has the initial Shannon entropy ''S<sub>Th</sub><sup>(1)</sup>'', should reproduce the expectation values of the observed macroscopic variables at time ''t<sub>2</sub>''. However it will no longer necessarily be a maximum entropy distribution for that new macroscopic description. On the other hand, the new thermodynamic entropy ''S<sub>Th</sub><sup>(2)</sup>'' assuredly ''will'' measure the maximum entropy distribution, by construction. Therefore, we expect:
| |
| | |
| :<math>{S_{Th}}^{(2)} \geq {S_{Th}}^{(1)} </math>
| |
| | |
| At an abstract level, this result simply means that some of the information we originally had about the system has become "no longer useful" at a macroscopic level. At the level of the 6''N''-dimensional probability distribution, this result represents [[coarse graining]]—i.e., information loss by smoothing out very fine-scale detail.
| |
| | |
| === Caveats with the argument ===
| |
| | |
| Some caveats should be considered with the above.
| |
| | |
| 1. Like all statistical mechanical results according to the MaxEnt school, this increase in thermodynamic entropy is only a ''prediction''. It assumes in particular that the initial macroscopic description contains all of the information relevant to predicting the later macroscopic state. This may not be the case, for example if the initial description fails to reflect some aspect of the preparation of the system which later becomes relevant. In that case the "failure" of a MaxEnt prediction tells us that there is something more which is relevant that we may have overlooked in the physics of the system.
| |
| | |
| It is also sometimes suggested that [[quantum measurement]], especially in the [[decoherence]] interpretation, may give an apparently unexpected reduction in entropy per this argument, as it appears to involve macroscopic information becoming available which was previously inaccessible. (However, the entropy accounting of quantum measurement is tricky, because to get full decoherence one may be assuming an infinite environment, with an infinite entropy).
| |
| | |
| 2. The argument so far has glossed over the question of ''fluctuations''. It has also implicitly assumed that the uncertainty predicted at time ''t<sub>1</sub>'' for the variables at time ''t<sub>2</sub>'' will be much smaller than the measurement error. But if the measurements do meaningfully update our knowledge of the system, our uncertainty as to its state is reduced, giving a new ''S<sub>'''I'''</sub><sup>(2)</sup>'' which is ''less'' than ''S<sub>'''I'''</sub><sup>(1)</sup>''. (Note that if we allow ourselves the abilities of [[Laplace's demon]], the consequences of this new information can also be mapped backwards, so our uncertainty about the dynamical state at time ''t<sub>1</sub>'' is now ''also'' reduced from ''S<sub>'''I'''</sub><sup>(1)</sup>'' to ''S<sub>'''I'''</sub><sup>(2)</sup>'' ).
| |
| | |
| We know that ''S<sub>Th</sub><sup>(2)</sup> > S<sub>'''I'''</sub><sup>(2)</sup>''; but we can now no longer be certain that it is greater than ''S<sub>Th</sub><sup>(1)</sup> = S<sub>'''I'''</sub><sup>(1)</sup>''. This then leaves open the possibility for fluctuations in ''S<sub>Th</sub>''. The thermodynamic entropy may go "down" as well as up. A more sophisticated analysis is given by the entropy [[Fluctuation Theorem]], which can be established as a consequence of the time-dependent MaxEnt picture.
| |
| | |
| 3. As just indicated, the MaxEnt inference runs equally well in reverse. So given a particular final state, we can ask, what can we "retrodict" to improve our knowledge about earlier states? However the Second Law argument above also runs in reverse: given macroscopic information at time ''t<sub>2</sub>'', we should expect it too to become less useful. The two procedures are time-symmetric. But now the information will become less and less useful at earlier and earlier times. (Compare with [[Loschmidt's paradox]].) The MaxEnt inference would predict that the most probable origin of a currently low-entropy state would be as a spontaneous fluctuation from an earlier high entropy state. But this conflicts with what we know to have happened, namely that entropy has been increasing steadily, even back in the past.
| |
| | |
| The MaxEnt proponents' response to this would be that such a systematic failing in the prediction of a MaxEnt inference is a "good" thing.<ref name="Jaynes 1979">Jaynes, E.T. (1979).</ref> It means that there is thus clear evidence that some important physical information has been missed in the specification the problem. If it is correct that the dynamics "are" [[T-symmetry|time-symmetric]], it appears that we need to put in by hand a [[prior probability]] that initial configurations with a low thermodynamic entropy are more likely than initial configurations with a high thermodynamic entropy. This cannot be explained by the immediate dynamics. Quite possibly, it arises as a reflection of the evident time-asymmetric evolution of the universe on a cosmological scale (see [[arrow of time]]).
| |
| | |
| == Criticisms ==
| |
| | |
| Maximum Entropy thermodynamics has generally failed to be accepted by the majority of scientists, with mainstream thermodynamicists considering Jaynes' work as an unfounded mathematical contrivance. This is in part because of the relative paucity of published results from the MaxEnt school, especially with regard to new testable predictions far-from-equilibrium.<ref>Kleidon, A., Lorenz, R.D. (2005).</ref>
| |
| | |
| The theory has also been criticized in the grounds of internal consistency. For instance, [[Radu Balescu]] provides a concise but strong criticism of the MaxEnt School and of Jaynes' work. Balescu states how Jaynes' and coworkers theory is based on a non-transitive evolution law that produces ambiguous results. Although some difficulties of the theory can be cured, the theory "lacks a solid foundation" and "has not led to any new concrete result".<ref>Balescu, R. (1997).</ref>
| |
| | |
| ==See also==
| |
| *[[First Law of Thermodynamics]]
| |
| *[[Second Law of Thermodynamics]]
| |
| *[[Principle of maximum entropy]]
| |
| *[[Principle of Minimum Discrimination Information]]
| |
| *[[Kullback-Leibler divergence]]
| |
| *[[Quantum relative entropy]]
| |
| *[[Information theory and measure theory]]
| |
| *[[Entropy power inequality]]
| |
| | |
| == References ==
| |
| {{reflist|colwidth=25em}}
| |
| | |
| ===Bibliography of cited references===
| |
| *{{cite book |author=Balescu, Radu |title=Statistical Dynamics: Matter out of equilibrium |publisher=Imperial College Press |location=London |year=1997 }}
| |
| *{{cite journal
| |
| |last1=Jaynes |first1=E.T.
| |
| |author1-link=Edwin Thompson Jaynes
| |
| |date=Sept 1968
| |
| |doi=10.1109/TSSC.1968.300117
| |
| |title=Prior Probabilities
| |
| |url=http://bayes.wustl.edu/etj/articles/prior.pdf
| |
| |journal=IEEE Transactions on Systems Science and Cybernetics
| |
| |volume=SSC–4 |pages=227–241
| |
| |ref=harv
| |
| }}
| |
| *Guttmann, Y.M. (1999). ''The Concept of Probability in Statistical Physics'', Cambridge University Press, Cambridge UK, ISBN 978-0-521-62128-1.
| |
| *{{cite book |last=Jaynes |first=E.T. |authorlink=Edwin Thompson Jaynes |chapter=Where do we stand on maximum entropy? |chapterurl=http://bayes.wustl.edu/etj/articles/stand.on.entropy.pdf |editor=Levine, R., Tribus M. |title=The Maximum Entropy Formalism |publisher=MIT Press |year=1979 |isbn=0-262-12080-1 |ref={{harvid|Jaynes|1979}}}}
| |
| *{{cite journal |author=Jaynes, E.T. |authorlink=Edwin Thompson Jaynes |title=Some random observations |journal=Synthese |volume=63 |pages=115–138 |year=1985 |doi=10.1007/BF00485957 }}
| |
| *{{cite book |author=Jaynes, E.T. |title=Probability Theory: The Logic of Science |editor=Bretthorst, G.L. |publisher=Cambridge University Press |location=Cambridge |year=2003 |isbn=0-521-59271-2 }}
| |
| *{{cite book |first1=Axel |last1=Kleidon |first2=Ralph D. |last2=Lorenz |title=Non-equilibrium thermodynamics and the production of entropy: life, earth, and beyond |url=http://books.google.com/books?id=YRjfuEP_QycC&pg=PA42 |year=2005 |publisher=Springer |isbn=978-3-540-22495-2|pages=42–}}
| |
| | |
| == Further reading ==
| |
| {{refbegin}}
| |
| * {{cite journal |author=Bajkova, A.T. |title=The generalization of maximum entropy method for reconstruction of complex functions |journal=Astronomical and Astrophysical Transactions |volume=1 |issue=4 |pages=313–320 |year=1992 |doi=10.1080/10556799208230532 |bibcode = 1991A&AT....1..313B }}
| |
| * {{cite journal |author=Dewar, R.C. |title=Information theory explanation of the fluctuation theorem, maximum entropy production and self-organized criticality in non-equilibrium stationary states |journal=J. Phys. A: Math. Gen. |volume=36 |pages=631–41 |year=2003 |arxiv=cond-mat/0005382 |doi=10.1088/0305-4470/36/3/303 |issue=3 |url=http://iopscience.iop.org/0305-4470/36/3/303|bibcode = 2003JPhA...36..631D }}
| |
| * {{cite journal |author=Dewar, R.C. |authormask=1 |title=Maximum entropy production and the fluctuation theorem |journal=J. Phys. A: Math. Gen. |volume=38 |pages=L371–81 |year=2005 |url=http://www.iop.org/EJ/abstract/0305-4470/38/21/L01/ |doi=10.1088/0305-4470/38/21/L01 |issue=21|bibcode = 2005JPhA...38L.371D }}
| |
| * {{cite journal |author=Grinstein, G., Linsker, R. |title=Comments on a derivation and application of the 'maximum entropy production' principle |journal=J. Phys. A: Math. Theor. |volume=40 |pages=9717–20 |year=2007 |url=http://www.iop.org/EJ/abstract/1751-8121/40/31/N01/ |doi=10.1088/1751-8113/40/31/N01 |issue=31|bibcode = 2007JPhA...40.9717G }} Shows invalidity of Dewar's derivations (a) of maximum entropy production (MaxEP) from fluctuation theorem for far-from-equilibrium systems, and (b) of a claimed link between MaxEP and self-organized criticality.
| |
| * Grandy, W. T., 1987. ''Foundations of Statistical Mechanics. Vol 1: Equilibrium Theory; Vol. 2: Nonequilibrium Phenomena''. Dordrecht: D. Reidel. ''Vol. 1'': ISBN 90-277-2489-X. ''Vol. 2'': ISBN 90-277-2649-3.
| |
| * {{cite journal |author=Grandy, W.T. |authormask=1 |title=Three papers in nonequilibrium statistical mechanics |journal=Found. Phys. |volume=34 |issue=1 |pages=21, 771 |year=2004 |url=http://physics.uwyo.edu/~tgrandy/Statistical_Mechanics.html|arxiv = cond-mat/0303291 |bibcode = 2004FoPh...34...21G |doi = 10.1023/B:FOOP.0000012008.36856.c1 }}
| |
| * {{cite book |author=Gull, S.F. |chapter=Some misconceptions about entropy |chapterurl=http://www.ucl.ac.uk/~ucesjph/reality/entropy/text.html |editor=Buck, B., Macaulay, V.A. |title=Maximum Entropy in Action |publisher=Oxford University Press |year=1991 |isbn=0-19-853963-0 }}
| |
| *{{harvnb|Jaynes|1979}}
| |
| * [http://bayes.wustl.edu/etj/node1.html Extensive archive of further papers] by E.T. Jaynes on probability and physics. Many are collected in {{cite book |editor=Rosenkrantz, R.D. |title=E.T. Jaynes — Papers on probability, statistics and statistical physics |publisher=D. Reidel. |location=Dordrecht, Netherlands |year=1983 |isbn=90-277-1448-7 }}
| |
| * {{cite journal |author=Lorenz, R. |title=Full steam ahead — probably |journal=Science |volume=299 |pages=837–8 |year=2003 |url=http://www.lpl.arizona.edu/~rlorenz/fullsteamahead.pdf |format=PDF |doi=10.1126/science.1081280 |issue=5608}}
| |
| * {{cite arXiv |author=Rau, Jochen |title=Statistical Mechanics in a Nutshell |year=1998 |eprint=physics/9805024 |class=physics.ed-ph}}
| |
| {{refend}}
| |
| | |
| [[Category:Statistical mechanics]]
| |
| [[Category:Philosophy of thermal and statistical physics]]
| |
| [[Category:Non-equilibrium thermodynamics]]
| |
| [[Category:Information theory]]
| |
| [[Category:Thermodynamics]]
| |
| [[Category:Thermodynamic entropy]]
| |