|
|
| Line 1: |
Line 1: |
| {{Technical|date=September 2010}}
| | Hello and welcome. My title is Irwin and I totally dig that name. What I love doing is taking part in baseball but I haven't made a dime with it. South Dakota is where me and my husband reside. My working day occupation is a meter reader.<br><br>Also visit my page :: [http://www.bonuslife.org/node/145034 at home std testing] |
| '''Q-learning''' is a model-free [[reinforcement learning]] technique. Specifically, Q-learning can be used to find an optimal action-selection policy for any given (finite) [[Markov decision process]] (MDP). It works by learning an [[action-value function]] that ultimately gives the expected utility of taking a given action in a given state and following the optimal policy thereafter. When such an action-value function is learned, the optimal policy can be constructed by simply selecting the action with the highest value in each state. One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment. Additionally, Q-learning can handle problems with stochastic transitions and rewards, without requiring any adaptations. It has been proven that for any finite MDP, Q-learning eventually finds an optimal policy.
| |
| | |
| == Algorithm ==
| |
| | |
| The problem model, the MDP, consists of an agent, states ''S'' and a set of actions per state ''A''. By performing an action <math>a \in A</math>, the agent can move from state to state. Each state provides the agent a reward (a real or natural number). The goal of the agent is to maximize its total reward. It does this by learning which action is optimal for each state.
| |
| | |
| The algorithm therefore has a function which calculates the Quality of a state-action combination:
| |
| | |
| :<math>Q: S \times A \to \mathbb{R}</math>
| |
| | |
| Before learning has started, ''Q'' returns an (arbitrary) fixed value, chosen by the designer. Then, each time the agent selects an action, and observes a reward and a new state that both may depend on both the previous state and the selected action. The core of the algorithm is a simple [[Markov_decision_process#Value_iteration|value iteration update]]. It assumes the old value and makes a correction based on the new information.
| |
| | |
| :<math>Q_{t+1}(s_{t},a_{t}) = \underbrace{Q_t(s_t,a_t)}_{\rm old~value} + \underbrace{\alpha_t(s_t,a_t)}_{\rm learning~rate} \times \left[ \overbrace{\underbrace{R_{t+1}}_{\rm reward} + \underbrace{\gamma}_{\rm discount~factor} \underbrace{\max_{a}Q_t(s_{t+1}, a)}_{\rm estimate~of~optimal~future~value}}^{\rm learned~value} - \underbrace{Q_t(s_t,a_t)}_{\rm old~value} \right]</math>
| |
| | |
| where ''<math>R_{t+1}</math>'' is the reward observed after performing <math>a_{t}</math> in <math>s_{t}</math>, and where <math>\alpha_t(s, a)</math> (<math>0 < \alpha \le 1</math>) is the learning rate (may be the same for all pairs). The discount factor <math>\gamma</math> (<math>0 \le \gamma \le 1</math>) trades off the importance of sooner versus later rewards.
| |
| | |
| An episode of the algorithm ends when state <math>s_{t+1}</math> is a final state (or, "absorbing state"). However, Q-learning can also learn in non-episodic tasks. If the discount factor is lower than 1, the action values are finite even if the problem can contain infinite loops.
| |
| | |
| Note that for all final states <math>s_f</math>, <math>Q(s_f, a)</math> is never updated and thus retains its initial value. In most cases, <math>Q(s_f,a)</math> can be taken to be equal to zero.
| |
| | |
| == Influence of variables on the algorithm ==
| |
| | |
| === Learning rate ===
| |
| | |
| The learning rate determines to what extent the newly acquired information will override the old information. A factor of 0 will make the agent not learn anything, while a factor of 1 would make the agent consider only the most recent information. In fully deterministic environments, a learning rate of <math>\alpha_t(s,a) = 1</math> is optimal. When the problem is stochastic, the algorithms still converges under some technical conditions on the learning rate, that require it to decrease to zero. In practice, often a constant learning rate is used, such as <math>\alpha_t(s,a) = 0.1</math> for all <math>t</math>.<ref>[http://www.cs.ualberta.ca/~sutton/book/ebook/the-book.html Reinforcement Learning: An Introduction]. Richard Sutton and Andrew Barto. MIT Press, 1998.</ref>
| |
| | |
| === Discount factor ===
| |
| | |
| The discount factor determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term high reward. If the discount factor meets or exceeds 1, the action values may diverge.
| |
| | |
| === Initial conditions (<math>Q(s_0,a_0)</math>) ===
| |
| | |
| Since Q-learning is an iterative algorithm, it implicitly assumes an initial condition before the first update occur. A high (infinite) initial value, also known as "optimistic initial conditions",<ref>http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node21.html</ref> can encourage exploration: no matter what action will take place, the update rule will cause it to have lower values than the other alternative, thus increasing their choice probability. Recently, it was suggested that the first reward <math>r</math> could be used to reset the initial conditions. According to this idea, the first time an action is taken the reward is used to set the value of <math>Q</math>. This will allow immediate learning in case of fix deterministic rewards. Surprisingly, this resetting-of-initial-conditions (RIC) approach seems to be consistent with human behaviour in repeated binary choice experiments.<ref>[http://www.ncbi.nlm.nih.gov/pubmed/22924882 The Role of First Impression in Operant Learning. Shteingart H, Neiman T, Loewenstein Y. J Exp Psychol Gen. 2013 May; 142(2):476-88. doi: 10.1037/a0029550. Epub 2012 Aug 27.]</ref>
| |
| | |
| == Implementation ==
| |
| Q-learning at its simplest uses tables to store data. This very quickly loses viability with increasing levels of complexity of the system it is monitoring/controlling. One answer to this problem is to use an (adapted) [[artificial neural network]] as a function approximator, as demonstrated by Tesauro in his [[Backgammon]] playing [[temporal difference learning]] research.<ref name='CACM'>{{cite journal|title=Temporal Difference Learning and TD-Gammon|journal=Communications of the ACM|date=March 1995|first=Gerald|last=Tesauro|coauthors=|volume=38|issue=3|pages=|id= |url=http://www.research.ibm.com/massive/tdl.html|accessdate=2010-02-08 }}</ref>
| |
| | |
| More generally, Q-learning can be combined with [[function approximation]].<ref>Hado van Hasselt. Reinforcement Learning in Continuous State and Action Spaces. In: Reinforcement Learning: State of the Art, Springer, pages 207-251, 2012</ref> This makes it possible to apply the algorithm to larger problems, even when the state space is continuous, and therefore infinitely large. Additionally, it may speed up learning in finite problems, due to the fact that the algorithm can generalize earlier experiences to previously unseen states.
| |
| | |
| == Early study ==
| |
| | |
| Q-learning was first introduced by Watkins<ref>Watkins, C.J.C.H., (1989), Learning from Delayed Rewards. Ph.D. thesis, Cambridge University.</ref> in 1989.
| |
| The convergence proof was presented later by Watkins and Dayan<ref>Watkins and Dayan, C.J.C.H., (1992), 'Q-learning.Machine Learning', ISBN : 8:279-292</ref> in 1992.
| |
| | |
| == Variants ==
| |
| Delayed Q-learning is an alternative implementation of the online Q-learning algorithm, with [[Probably approximately correct learning|Probably approximately correct learning (PAC)]].<ref>Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. Pac model-free
| |
| reinforcement learning. In Proc. 23nd{{Clarify|date=September 2013|reason=Should this be '23rd', '22nd' or something else?}} ICML 2006, pages 881–888, 2006.</ref>
| |
| | |
| Due to the fact that the maximum approximated action value is used in the Q-learning update, in noisy environments Q-learning can sometimes overestimate the actions values, slowing the learning. A recent variant called Double Q-learning was proposed to correct this.
| |
| <ref>Hado van Hasselt. [http://books.nips.cc/papers/files/nips23/NIPS2010_0208.pdf Double Q-learning]. In Advances in Neural Information Processing Systems 23, pages 2613-2622, 2011.</ref>
| |
| | |
| Greedy GQ is a variant of Q-learning to use in combination with (linear) function approximation.<ref>Hamid Maei, and Csaba Szepesv{\'a}ri, Shalabh Bhatnagar and Richard Sutton. Toward off-policy learning control with function approximation. In proceedings of the 27th International Conference on Machine Learning, pages 719-726, 2010.</ref> The advantage of Greedy GQ is that convergence guarantees can be given even when function approximation is used to estimate the action values.
| |
| | |
| == See also ==
| |
| * [[Reinforcement learning]]
| |
| * [[Temporal difference learning]]
| |
| * [[SARSA]]
| |
| * [[Prisoner's dilemma#The iterated prisoner.27s dilemma|Iterated prisoner's dilemma]]
| |
| * [[Game theory]]
| |
| * [[Fitted Q iteration algorithm]]
| |
| | |
| == External links ==
| |
| * [http://www.cs.rhul.ac.uk/~chrisw/thesis.html Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. PhD thesis, Cambridge University, Cambridge, England.]
| |
| * [http://portal.acm.org/citation.cfm?id=1143955 Strehl, Li, Wiewiora, Langford, Littman (2006). PAC model-free reinforcement learning]
| |
| * [http://people.revoledu.com/kardi/tutorial/ReinforcementLearning/index.html Q-Learning by Examples]
| |
| * [http://www.cs.ualberta.ca/%7Esutton/book/the-book.html ''Reinforcement Learning: An Introduction''] by Richard Sutton and Andrew S. Barto, an online textbook. See [http://www.cs.ualberta.ca/~sutton/book/ebook/node65.html "6.5 Q-Learning: Off-Policy TD Control"].
| |
| * [http://elsy.gdan.pl/index.php Connectionist Q-learning Java Framework]
| |
| * [http://sourceforge.net/projects/piqle/ Piqle: a Generic Java Platform for Reinforcement Learning]
| |
| * [http://ccl.northwestern.edu/netlogo/models/community/Reinforcement%20Learning%20Maze Reinforcement Learning Maze], a demonstration of guiding an ant through a maze using Q-learning.
| |
| * [http://www.research.ibm.com/infoecon/paps/html/ijcai99_qnn/node4.html Q-learning work by Gerald Tesauro]
| |
| * [http://citeseer.comp.nus.edu.sg/352693.html Q-learning work by Tesauro Citeseer Link]
| |
| * [http://github.com/sandropaganotti/processing.org-q-learning-td-lambda-/tree/master Q-learning algorithm implemented in processing.org language]
| |
| | |
| == References ==
| |
| | |
| <references/>
| |
| | |
| [[Category:Machine learning algorithms]]
| |
Hello and welcome. My title is Irwin and I totally dig that name. What I love doing is taking part in baseball but I haven't made a dime with it. South Dakota is where me and my husband reside. My working day occupation is a meter reader.
Also visit my page :: at home std testing