Ornstein isomorphism theorem

In probability and statistics a Markov renewal process is a random process that generalizes the notion of Markov jump processes. Other random processes like Markov chain, Poisson process, and renewal process can be derived as a special case of an MRP (Markov renewal process).

Definition

Consider a state space $\mathrm {S} .$ Consider a set of random variables $(X_{n},T_{n})$, where $T_{n}$ are the jump times and $X_{n}$ are the associated states in the Markov chain (see Figure). Let the inter-arrival time, $\tau _{n}=T_{n}-T_{n-1}$. Then the sequence (X_n, T_n) is called a Markov renewal process if

$\Pr(\tau _{n+1}\leq t,X_{n+1}=j|(X_{0},T_{0}),(X_{1},T_{1}),\ldots ,(X_{n}=i,T_{n}))$

$=\Pr(\tau _{n+1}\leq t,X_{n+1}=j|X_{n}=i)\,\forall n\geq 1,t\geq 0,i,j\in \mathrm {S}$

An illustration of a Markov renewal process

Relation to other stochastic processes

1. If we define a new stochastic process $Y_{t}:=X_{n}$ for $t\in [T_{n},T_{n+1})$, then the process $Y_{t}$ is called a semi-Markov process. Note the main difference between an MRP and a semi-Markov process is that the former is defined as a two-tuple of states and times, whereas the latter is the actual random process that evolves over time. The entire process is not Markovian, i.e., memoryless, as happens in a CTMC. Instead the process is Markovian only at the specified jump instants. This is the rationale behind the name, Semi-Markov.
2. A semi-Markov process where all the holding times are exponentially distributed is called a continuous time Markov chain/process (CTMC). In other words, if the inter-arrival times are exponentially distributed and if the waiting time in a state and the next state reached are independent, we have a CTMC.

   $\Pr(\tau _{n+1}\leq t,X_{n+1}=j|(X_{0},T_{0}),(X_{1},T_{1}),\ldots ,(X_{n}=i,T_{n}))=\Pr(\tau _{n+1}\leq t,X_{n+1}=j|X_{n}=i)$

   $=\Pr(X_{n+1}=j|X_{n}=i)(1-e^{-\lambda _{i}t}),{\text{ for all }}n\geq 1,t\geq 0,i,j\in \mathrm {S}$

3. The sequence $X_{n}$ in the MRP is a discrete-time Markov chain. In other words, if the time variables are ignored in the MRP equation, we end up with a DTMC.

   $\Pr(X_{n+1}=j|X_{0},X_{1},\ldots ,X_{n}=i)=\Pr(X_{n+1}=j|X_{n}=i)\,\forall n\geq 1,i,j\in \mathrm {S}$

4. If the sequence of $\tau$s are independent and identically distributed, and if their distribution does not depend on the state $X_{n}$, then the process is a renewal process. So, if the exact states are ignored and we have a chain of iid times, then we have a renewal process.

   $\Pr(\tau _{n+1}\leq t|T_{0},T_{1},\ldots ,T_{n})=\Pr(\tau _{n+1}\leq t)\,\forall n\geq 1,\forall t\geq 0$

See also

• Markov process
• Renewal theory
• Variable-order Markov model

References and Further Reading

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.