en>Mogism: Cleanup/Typo fixing, typos fixed: surpressed → suppressed, and and → and using AWB

2013-08-21T21:51:42Z

Cleanup/Typo fixing, typos fixed: surpressed → suppressed, and and → and using AWB

New page

The '''free energy principle''' tries to explain how (biological) systems maintain their order by restricting themselves to a limited number of states.<ref>Ashby, W. R. (1962). [http://csis.pace.edu/~marchese/CS396x/Computing/Ashby.pdf Principles of the self-organizing system].in Principles of Self-Organization: Transactions of the University of Illinois Symposium, H. Von Foerster and G. W. Zopf, Jr. (eds.), Pergamon Press: London, UK, pp. 255–278.</ref> It states that biological systems minimise a free energy function of their internal states, which entail beliefs about hidden states in their environment. The implicit minimisation of variational free energy is formally related to [[variational Bayesian methods]] and was originally introduced by [[Karl Friston]] as an explanation for embodied perception in neuroscience,<ref>Friston, K., Kilner, J., & Harrison, L. (2006). [http://www.fil.ion.ucl.ac.uk/~karl/A%20free%20energy%20principle%20for%20the%20brain.pdf A free energy principle for the brain]. J Physiol Paris. , 100 (1–3), 70–87.</ref> where it is also known as '''active inference'''.

== Background ==

The notion that [[self-organisation|self-organising]] biological systems – like a cell or brain – can be understood as minimising variational free energy is based upon Helmholtz’s observations on [[unconscious inference]]<ref name="Helmholtz">Helmholtz, H. (1866/1962). Concerning the perceptions in general. In Treatise on physiological optics (J. Southall, Trans., 3rd ed., Vol. III). New York: Dover.</ref> and subsequent treatments in psychology <ref>Gregory, R. L. (1980). [http://www.jstor.org/stable/2395424 Perceptions as hypotheses]. Phil Trans R Soc Lond B. , 290, 181–197.</ref> and machine learning.<ref name="Dayan">Dayan, P., Hinton, G. E., & Neal, R. (1995). [http://www.gatsby.ucl.ac.uk/~dayan/papers/hm95.pdf The Helmholtz machine]. Neural Computation , 7, 889–904.</ref> Variational free energy is a functional of some outcomes and a probability density over their (hidden) causes. This [[Calculus of variations|variational]] density is defined in relation to a probabilistic model that generates outcomes from causes. In this setting, free energy provides an (upper bound) approximation to [[Marginal likelihood|Bayesian model evidence]].<ref>Beal, M. J. (2003). [http://www.cse.buffalo.edu/faculty/mbeal/papers/beal03.pdf Variational Algorithms for Approximate Bayesian Inference]. PhD. Thesis, University College London.</ref> Its minimisation can therefore be used to explain Bayesian inference and learning. When a system actively samples outcomes to minimise free energy, it implicitly performs active inference and maximises the evidence for its (generative) model.

However, free energy is also an upper bound on the [[self-information]] (or surprise) of outcomes, where the long-term average of [[Self-information|surprise]] is entropy. This means that if a system acts to minimise free energy, it will implicitly place an upper bound on the entropy of the outcomes – or sensory states – it samples.<ref name="Friston">Friston, K. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/A%20Free%20Energy%20Principle%20for%20Biological%20Systems.pdf A Free Energy Principle for Biological Systems]. Entropy , 14, 2100–2121.</ref>

=== Relationship to other theories ===

Active inference is closely related to the [[Good Regulator|good regulator theorem]] <ref>Conant, R. C., & Ashby, R. W. (1970). Every Good Regulator of a system must be a model of that system. Int. J. Systems Sci. , 1 (2), 89–97.</ref> and related accounts of [[self-organisation]],<ref>Kauffman, S. (1993). The Origins of Order: Self-Organization and Selection in Evolution. Oxford: Oxford University Press.</ref><ref>Nicolis, G., & Prigogine, I. (1977). Self-organization in non-equilibrium systems. New York: John Wiley.</ref> such as [[self-assembly]], [[pattern formation]] and [[autopoiesis]].<ref>Maturana, H. R., & Varela, F. (1980). [http://topologicalmedialab.net/xinwei/classes/readings/Maturana/autopoesis_and_cognition.pdf Autopoiesis: the organization of the living]. In V. F. Maturana HR (Ed.), Autopoiesis and Cognition. Dordrecht, Netherlands: Reidel.</ref> It addresses the themes considered in [[cybernetics]], [[Synergetics (Haken)|synergetics]] <ref>Haken, H. (1983). Synergetics: An introduction. Non-equilibrium phase transition and self-organisation in physics, chemistry and biology (3rd ed.). Berlin: Springer Verlag.</ref> and [[embodied cognition]]. Because free energy can be expressed as the expected energy (of outcomes) under the variational density minus its entropy, it is also related to the [[maximum entropy principle]].<ref>Jaynes, E. T. (1957). [http://bayes.wustl.edu/etj/articles/theory.1.pdf Information Theory and Statistical Mechanics]. Physical Review Series II , 106 (4), 620–30.</ref> Finally, because the time average of energy is action, the principle of minimum variational free energy is a [[principle of least action]].

== Definition ==

[[Image:MarokovBlanketFreeEnergyFigure.jpg|500px|right|These schematics illustrate the partition of states into internal and hidden or external states that are separated by a Markov blanket – comprising sensory and active states. The upper panel shows this partition as it would be applied to action and perception in the brain; where active and internal states minimise a free energy functional of sensory states. The ensuing self-organisation of internal states then correspond perception, while action couples brain states back to external states. The lower panel shows exactly the same dependencies but rearranged so that the internal states are associated with the intracellular states of a cell, while the sensory states become the surface states of the cell membrane overlying active states (e.g., the actin filaments of the cytoskeleton).]]

'''Definition''' (continuous formulation): Active inference rests on the tuple <math>(\Omega,\Psi,S,A,R,q,p)</math>,
* ''A sample space'' <math>\Omega</math> – from which random fluctuations <math>\omega \in \Omega</math> are drawn
* ''Hidden or external states'' <math>\Psi:\Psi\times A \times \Omega \to \mathbb{R}</math> – that cause sensory states and depend on action
* ''Sensory states'' <math>S:\Psi \times A \times \Omega \to \mathbb{R}</math> – a probabilistic mapping from action and hidden states
* ''Action'' <math>A:S\times R \to \mathbb{R}</math> – that depends on sensory and internal states
* ''Internal states'' <math>R:R\times S \to \mathbb{R}</math> – that cause action and depend on sensory states
* ''Generative density'' <math>p(s, \psi | m)</math> – over sensory and hidden states under a generative model <math>m</math>
* ''Variational density'' <math>q(\psi | \mu)</math> – over hidden states <math>\psi \in \Psi</math> that is parameterised by internal states <math>\mu \in R</math>

=== Action and perception ===

The objective is to maximise model evidence <math>p(s|m)</math> or minimise surprise <math>-\log p(s|m)</math> . This generally involves an intractable marginalisation over hidden states, so surprise is replaced with an upper variational free energy bound.<ref>Feynman, R. P. (1972). Statistical mechanics. Reading MA: Benjamin.</ref> However, this means that internal states must also minimise free energy, because free energy is a functional of sensory and internal states:

: <math> a(t) = \underset{a}{\operatorname{arg\,min}} \{ F(s(t),\mu(t)) \}</math>
: <math>\mu(t) = \underset{\mu}{\operatorname{arg\,min}} \{ F(s(t),\mu)) \} </math>

: <math>\underset{\mathrm{free-energy}} {\underbrace{F(s,\mu)}} = \underset{energy} {\underbrace{ E_q[-\log p(s,\psi \mid m)]}} - \underset{\mathrm{entropy}} {\underbrace{ H[q(\psi | \mu)]}}
= \underset{surprise} {\underbrace{ -\log p(s | m)}} + \underset{divergence} {\underbrace{ D_{\mathrm{KL}}[q(\psi | \mu) \| p(\psi \mid s,m)]}}
\geq \underset{surprise} {\underbrace{ -\log p(s | m)}} </math>

This induces a dual minimisation with respect to action and internal states that correspond to action and perception respectively.

== Free energy minimisation ==

=== Free energy minimisation and self-organisation ===

Free energy minimisation has been proposed as a hallmark of self-organising systems, when cast as [[random dynamical system]]s.<ref>Crauel, H., & Flandoli, F. (1994). Attractors for random dynamical systems. Probab Theory Relat Fields , 100, 365–393.</ref> This formulation rests on a [[Markov blanket]] (comprising action and sensory states) that separates internal and external states. If internal states and action minimise free energy, then they place an upper bound on the entropy of sensory states

<math> \lim_{T\to\infty} \frac{1}{T} \underset{free-action} {\underbrace{\int_0^T F(s(t),\mu (t))dt}} \ge
\lim_{T\to\infty} \frac{1}{T} \int_0^T \underset{surprise}{\underbrace{-\log p(s(t)|m)}} dt = H[p(s|m)] </math>

This is because – under [[Ergodic theory|ergodic]] assumptions – the long-term average of surprise is entropy. This bound resists a natural tendency to disorder – of the sort associated with the [[second law of thermodynamics]] and the [[fluctuation theorem]].

=== Free energy minimisation and Bayesian inference ===

All Bayesian inference can be cast in terms of free energy minimisation; e.g.,.<ref>Roweis, S., & Ghahramani, Z. (1999). [http://authors.library.caltech.edu/13697/1/ROWnc99.pdf A unifying review of linear Gaussian models]. Neural Computat. , 11 (2), 305–45.</ref> When free energy is minimised with respect to internal states, the [[Kullback–Leibler divergence]] between the variational and posterior density over hidden states is minimised. This corresponds to approximate [[Bayesian inference]] – when the form of the variational density is fixed – and exact [[Bayesian inference]] otherwise. Free energy minimisation therefore provides a generic description of Bayesian inference and filtering (e.g., [[Kalman filter]]ing). It is also used in Bayesian [[model selection]], where free energy can be usefully decomposed into complexity and accuracy:

: <math> \underset{free-energy} {\underbrace{ F(s,\mu)}} = \underset{complexity} {\underbrace{ D_\mathrm{KL}[q(\psi|\mu)\| p(\psi|m)]}} - \underset{accuracy} {\underbrace{E_q[\log p(s|\psi,m)]}}</math>

Models with minimum free energy provide an accurate explanation of data, under complexity costs (c.f., Occam's razor and more formal treatments of computational costs <ref>Ortega, P. A., & Braun, D. A. (2012). [http://rspa.royalsocietypublishing.org/content/469/2153/20120683 Thermodynamics as a theory of decision-making with information processing costs]. Proceedings of the Royal Society A, vol. 469, no. 2153 (20120683) .</ref>). Here, complexity is the divergence between the variational density and prior beliefs about hidden states (i.e., the effective degrees of freedom used to explain the data).

=== Free energy minimisation and thermodynamics ===

Variational free energy is an information theoretic functional and is distinct from thermodynamic (Helmholtz) [[free energy]].<ref>Evans, D. J. (2003). [http://rscweb.anu.edu.au/~evans/papers/NEFET.pdf A non-equilibrium free energy theorem for deterministic systems]. Molecular Physics , 101, 15551–4.</ref> However, the complexity term of variational free energy shares the same fixed point as Helmholtz free energy (under the assumption the system is thermodynamically closed but not isolated). This is because if sensory perturbations are suspended (for a suitably long period of time), complexity is minimised (because accuracy can be neglected). At this point, the system is at equilibrium and internal states minimise Helmholtz free energy, by the [[principle of minimum energy]].<ref>Jarzynski, C. (1997). [http://arxiv.org/pdf/cond-mat/9610209.pdf Nonequilibrium equality for free energy differences]. Phys. Rev. Lett., 78, 2690.</ref>

=== Free energy minimisation and information theory ===

Free energy minimisation is equivalent to maximising the mutual information between sensory states and internal states that parameterise the variational density (for a fixed entropy variational density).<ref name="Friston" /> This relates free energy minimization to the principle of minimum redundancy <ref>Barlow, H. (1961). [http://www.trin.cam.ac.uk/horacebarlow/21.pdf Possible principles underlying the transformations of sensory messages]. In W. Rosenblith (Ed.), Sensory Communication (pp. 217-34). Cambridge, MA: MIT Press.</ref> and related treatments using information theory to describe optimal behaviour.<ref>Linsker, R. (1990). [http://psych.stanford.edu/~jlm/pdfs/LinskerAnnRev90.pdf Perceptual neural organization: some approaches based on network models and information theory]. Annu Rev Neurosci. , 13, 257–81.</ref><ref>Bialek, W., Nemenman, I., & Tishby, N. (2001). [http://www.princeton.edu/~wbialek/our_papers/bnt_01a.pdf Predictability, complexity, and learning]. Neural Computat., 13 (11), 2409–63.</ref>

== Free energy minimisation in neuroscience ==

Free energy minimisation provides a useful way to formulate normative (Bayes optimal) models of neuronal inference and learning under uncertainty <ref>Friston, K. (2010). [http://www.fil.ion.ucl.ac.uk/~karl/The%20free-energy%20principle%20A%20unified%20brain%20theory.pdf The free-energy principle: a unified brain theory?] Nat Rev Neurosci. , 11 (2), 127–38.</ref> and therefore subscribes to the [[Bayesian brain]] hypothesis.<ref>Knill, D. C., & Pouget, A. (2004). [http://mrl.isr.uc.pt/pub/bscw.cgi/d27540/ReviewKnillPouget2.pdf The Bayesian brain: the role of uncertainty in neural coding and computation]. Trends Neurosci. , 27 (12), 712–9.</ref> The neuronal processes described by free energy minimisation depend on the nature of hidden states: <math> \Psi = X \times \Theta \times \Pi </math> that can comprise time dependent variables, time invariant parameters and the precision (inverse variance or temperature) of random fluctuations. Minimising variables, parameters and precision corresponds to inference, learning and the encoding of uncertainty, respectively:

=== Perceptual inference and categorisation ===

Free energy minimisation formalises the notion of unconscious inference in perception <ref name="Helmholtz" /><ref name="Dayan" /> and provides a normative (Bayesian) theory of neuronal processing. The associated process theory of neuronal dynamics is based on minimising free energy through gradient descent. This corresponds to [[Generalized filtering|generalised Bayesian filtering]] (where ~ denotes a variable in generalised coordinates of motion and <math>D</math> is a derivative matrix operator):<ref>Friston, K., Stephan, K., Li, B., & Daunizeau, J. (2010). [http://www.fil.ion.ucl.ac.uk/~karl/Generalised%20Filtering.pdf Generalised Filtering]. Mathematical Problems in Engineering , vol., 2010, 621670</ref>

: <math>\dot{\tilde{\mu}} = D \tilde{\mu} - \partial_{\tilde{\mu}} F(s,\tilde{\mu})</math>

Usually, the generative models that define free energy are non-linear and hierarchical (like cortical hierarchies in the brain). Special cases of generalised filtering include [[Kalman filter]]ing, which is formally equivalent to predictive coding <ref>Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci. , 2 (1), 79–87.</ref> – a popular metaphor for message passing in the brain. Under hierarchical models, predictive coding involves the recurrent exchange of ascending (bottom-up) prediction errors and descending (top-down) predictions <ref name="Mumford">Mumford, D. (1992). [http://cs.brown.edu/people/tld/projects/cortex/course/suggested_reading_list/supplements/documents/MumfordBC-92.pdf On the computational architecture of the neocortex]. II. Biol. Cybern. , 66, 241–51.</ref> that is consistent with the anatomy and physiology of sensory <ref>Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Canonical%20Microcircuits%20for%20Predictive%20Coding.pdf Canonical microcircuits for predictive coding]. Neuron , 76 (4), 695–711.</ref> and motor systems.<ref>Adams, R. A., Shipp, S., & Friston, K. J. (2013). [http://www.fil.ion.ucl.ac.uk/~karl/Predictions%20not%20commands%20-%20active%20inference%20in%20the%20motor%20system.pdf Predictions not commands: active inference in the motor system]. Brain Struct Funct. , 218 (3), 611–43</ref>

=== Perceptual learning and memory ===

In predictive coding, optimising model parameters through a gradient ascent on the time integral of free energy (free action) reduces to associative or [[Hebbian theory|Hebbian plasticity]] and is associated with [[synaptic plasticity]] in the brain.

=== Perceptual precision, attention and salience ===

Optimising the precision parameters corresponds to optimising the gain of prediction errors (c.f., Kalman gain). In neuronally plausible implementations of predictive coding,<ref name="Mumford" /> this corresponds to optimising the excitability superficial pyramidal cells and has been interpreted in terms of attentional gain.<ref name="Feldman">Feldman, H., & Friston, K. J. (2010). [http://www.fil.ion.ucl.ac.uk/~karl/Attention%20uncertainty%20and%20free-energy.pdf Attention, uncertainty, and free-energy]. Frontiers in Human Neuroscience , 4, 215.</ref>

== Active inference ==

When gradient descent is applied to action <math> \dot{a} = -\partial_aF(s,\tilde{\mu}) </math> , motor control can be understood in terms of classical reflex arcs that are engaged by descending (corticospinal) predictions. This provides a formalism that generalizes the equilibrium point solution – to the [[degrees of freedom problem]] <ref>Feldman, A. G., & Levin, M. F. (1995). [http://e.guigon.free.fr/rsc/article/FeldmanLevin95.pdf The origin and use of positional frames of reference in motor control]. Behav Brain Sci. , 18, 723–806.</ref> – to movement trajectories.

=== Active inference and optimal control ===

Active inference is related to [[optimal control]] by replacing value or cost-to-go functions with prior beliefs about state transitions or flow.<ref>Friston, K., (2011). [http://www.fil.ion.ucl.ac.uk/~karl/What%20Is%20Optimal%20about%20Motor%20Control.pdf What is optimal about motor control?]. Neuron, 72(3), 488–98.</ref> This exploits the close connection between Bayesian filtering and the solution to the [[Bellman equation]]. However, active inference starts with (priors over) flow <math> f = \Gamma \cdot \nabla V + \nabla \times W </math> that are specified with scalar <math> V(x) </math> and vector <math> W(x) </math> value functions of state space (c.f., the [[Helmholtz decomposition]]). Here, <math> \Gamma </math> is the amplitude of random fluctuations and cost is <math> c(x) = f \cdot \nabla V + \nabla \cdot \Gamma \cdot V</math>. The priors over flow <math> p(\tilde{x}|m) </math> induce a prior over states <math> p(x|m) = \exp (V(x)) </math> that is the solution to the appropriate forward [[Kolmogorov equations]].<ref>Friston, K., & Ao, P. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Free%20Energy%20Value%20and%20Attractors.pdf Free-energy, value and attractors]. Computational and mathematical methods in medicine, 2012, 937860.</ref> In contrast, optimal control optimises the flow, given a cost function, under the assumption that <math> W = 0 </math> (i.e., the flow is curl free or has detailed balance). Usually, this entails solving backward [[Kolmogorov equations]].<ref>Kappen, H., (2005). [http://arxiv.org/pdf/physics/0505066v4.pdf Path integrals and symmetry breaking for optimal control theory]. Journal of Statistical Mechanics: Theory and Experiment, 11, p. P11011.</ref>

=== Active inference and optimal decision (game) theory ===

[[Optimal decision]] problems (usually formulated as [[partially observable Markov decision process]]es) are treated within active inference by absorbing [[Utility| utility functions]] into prior beliefs. In this setting, states that have a high utility (low cost) are states an agent expects to occupy. By equipping the generative model with hidden states that model control, policies (control sequences) that minimise variational free energy lead to high utility states.<ref>Friston, K., Samothrakis, S. & Montague, R., (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Active%20inference%20and%20agency%20optimal%20control%20without%20cost%20functions.pdf Active inference and agency: optimal control without cost functions]. Biol. Cybernetics, 106(8–9), 523–41.</ref>

Neurobiologically, neuromodulators like [[dopamine]] are considered to report the precision of prediction errors by modulating the gain of principal cells encoding prediction error.<ref name="Friston_a">Friston, K. J. Shiner T, FitzGerald T, Galea JM, Adams R, Brown H, Dolan RJ, Moran R, Stephan KE, Bestmann S. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Dopamine%20Affordance%20and%20Active%20Inference.pdf Dopamine, affordance and active inference]. PLoS Comput Biol., 8(1), p. e1002327.</ref> This is closely related to – but formally distinct from – the role of dopamine in reporting prediction errors ''per se'' <ref>Fiorillo, C. D., Tobler, P. N. & Schultz, W., (2003). [http://e.guigon.free.fr/rsc/article/FiorilloEtAl03.pdf Discrete coding of reward probability and uncertainty by dopamine neurons]. Science, 299(5614), 1898–902.</ref> and related computational accounts.<ref>Frank, M. J., (2005). [http://ski.cog.brown.edu/papers/Frank_JOCN.pdf Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism]. J Cogn Neurosci., Jan, 1, 51–72.</ref>

=== Active inference and cognitive neuroscience ===

Active inference has been used to address a range of issues in [[cognitive neuroscience]], brain function and neuropsychiatry, including: action observation,<ref>Friston, K., Mattout, J. & Kilner, J., (2011). [http://www.fil.ion.ucl.ac.uk/~karl/Action%20understanding%20and%20active%20inference.pdf Action understanding and active inference]. Biol Cybern., 104, 137–160.</ref> mirror neurons,<ref>Kilner, J. M., Friston, K. J. & Frith, C. D., (2007). [http://www.fil.ion.ucl.ac.uk/~karl/Predictive%20coding%20an%20account%20of%20the%20mirror%20neuron%20system.pdf Predictive coding: an account of the mirror neuron system]. Cogn Process., 8(3), pp. 159–66.</ref> saccades and visual search,<ref>Friston, K., Adams, R. A., Perrinet, L. & Breakspear, M., (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Perceptions%20as%20hypotheses%20saccades%20as%20experiments.pdf Perceptions as hypotheses: saccades as experiments]. Front Psychol., 3, 151.</ref> sleep,<ref>Hobson, J. A. & Friston, K. J., (2012). Waking and dreaming consciousness: Neurobiological and functional considerations. Prog Neurobiol, 98(1), pp. 82–98.</ref> illusions,<ref>Brown, H., & Friston, K. J. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/Free-energy%20and%20illusions%20the%20Cornsweet%20effect.pdf Free-energy and illusions: the cornsweet effect]. Front Psychol , 3, 43.</ref> attention,<ref name="Feldman" /> action section,<ref name="Friston_a" /> hysteria <ref>Edwards, M. J., Adams, R. A., Brown, H., Pareés, I., & Friston, K. J. (2012). [http://www.fil.ion.ucl.ac.uk/~karl/A%20Bayesian%20account%20of%20hysteria.pdf A Bayesian account of 'hysteria']. Brain , 135(Pt 11):3495–512.</ref> and psychosis.<ref>Adams RA, Perrinet LU, Friston K. (2012). Smooth pursuit and visual occlusion: active inference and oculomotor control in schizophrenia. PLoS One. , 12;7(10):e47502</ref>

== See also ==

* [[Action-specific perception]]
* [[Affordance]]
* [[Autopoiesis]]
* [[Bayesian approaches to brain function|Bayesian brain]]
* [[Decision theory]]
* [[Embodied cognition]]
* [[Free energy]]
* [[Optimal control]]
* [[Self-organization]]
* [[Synergetics (Haken)]]
* [[Variational Bayesian methods]]

== References ==
{{Reflist|3}}

== External links ==
* [http://www.fil.ion.ucl.ac.uk/Free_Energy_Principle_Workshop/videos.html Lectures ]
* [http://www.fil.ion.ucl.ac.uk/~karl/welcome.html#_Free-energy_principle Publications ]
* [http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=8918803 Behavioral and Brain Sciences (by Andy Clark)]

[[Category:Systems]]
[[Category:Systems theory]]

Elliott formula - Revision history

en>Mogism: Cleanup/Typo fixing, typos fixed: surpressed → suppressed, and and → and using AWB