Nested word

This article is supplemental for “Convergence of random variables” and provides proofs for selected results.

Several results will be established using the portmanteau lemma: A sequence {X_n} converges in distribution to X if and only if any of the following conditions are met:

1. E[f(X_n)] → E[f(X)] for all bounded, continuous functions f;
2. E[f(X_n)] → E[f(X)] for all bounded, Lipschitz functions f;
3. limsup{Pr(X_n ∈ C)} ≤ Pr(X ∈ C) for all closed sets C;

<propA1>...</propA1> Convergence almost surely implies convergence in probability

$X_n\stackrel{as}{\to }X\Rightarrow X_n\stackrel{p}{\to }X$

Proof: If {X_n} converges to X almost surely, it means that the set of points {ω: lim X_n(ω) ≠ X(ω)} has measure zero; denote this set O. Now fix ε > 0 and consider a sequence of sets

$A_n=\bigcup _{m\ge n}\{|X_m-X|>\epsilon \}$

This sequence of sets is decreasing: A_n ⊇ A_n+1 ⊇ ..., and it decreases towards the set

$A_{\mathrm{\infty }}=\bigcap _{n\ge 1}{A}_n.$

For this decreasing sequence of events, their probabilities are also a decreasing sequence, and it decreases towards the Pr(A_∞); we shall show now that this number is equal to zero. Now any point ω in the complement of O is such that lim X_n(ω) = X(ω), which implies that |X_n(ω) − X(ω)| < ε for all n greater than a certain number N. Therefore, for all n ≥ N the point ω will not belong to the set A_n, and consequently it will not belong to A_∞. This means that A_∞ is disjoint with O, or equivalently, A_∞ is a subset of O and therefore Pr(A_∞) = 0.

Finally, consider

$\mathrm{Pr}\left(\right|X_n-X|>\epsilon )\le \mathrm{Pr}(A_n)\underset{n\to \mathrm{\infty }}{\to }0,$

which by definition means that X_n converges in probability to X.

<propA1i>...</propA1i> Convergence in probability does not imply almost sure convergence in the discrete case

If X_n are independent random variables assuming value one with probability 1/n and zero otherwise, then X_n converges to zero in probability but not almost surely. This can be verified using the Borel–Cantelli lemmas.

<propA2>...</propA2> Convergence in probability implies convergence in distribution

$X_n\stackrel{p}{\to }X\Rightarrow X_n\stackrel{d}{\to }X,$

Proof for the case of scalar random variables

Lemma. Let X, Y be random variables, a a real number and ε > 0. Then

$\mathrm{Pr}(Y\le a)\le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(|Y-X|>\epsilon ).$

Proof of lemma:

$\begin{array}{rl}\mathrm{Pr}(Y\le a)& =\mathrm{Pr}(Y\le a,X\le a+\epsilon )+\mathrm{Pr}(Y\le a,X>a+\epsilon )\\ & \le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(Y-X\le a-X,a-X<-\epsilon )\\ & \le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(Y-X<-\epsilon )\\ & \le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(Y-X<-\epsilon )+\mathrm{Pr}(Y-X>\epsilon )\\ & =\mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(|Y-X|>\epsilon )\end{array}$

Proof of the theorem: Recall that in order to prove convergence in distribution, one must show that the sequence of cumulative distribution functions converges to the F_X at every point where F_X is continuous. Let a be such a point. For every ε > 0, due to the preceding lemma, we have:

$\begin{array}{rl}\mathrm{Pr}(X_n\le a)& \le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(|X_n-X|>\epsilon )\\ \mathrm{Pr}(X\le a-\epsilon )& \le \mathrm{Pr}(X_n\le a)+\mathrm{Pr}(|X_n-X|>\epsilon )\end{array}$

So, we have

$\mathrm{Pr}(X\le a-\epsilon )-\mathrm{Pr}(|X_n-X|>\epsilon )\le \mathrm{Pr}(X_n\le a)\le \mathrm{Pr}(X\le a+\epsilon )+\mathrm{Pr}(|X_n-X|>\epsilon ).$

Taking the limit as n → ∞, we obtain:

$F_X(a-\epsilon )\le \underset{n\to \mathrm{\infty }}{\lim }\mathrm{Pr}(X_n\le a)\le F_X(a+\epsilon ),$

where F_X(a) = Pr(X ≤ a) is the cumulative distribution function of X. This function is continuous at a by assumption, and therefore both F_X(a−ε) and F_X(a+ε) converge to F_X(a) as ε → 0⁺. Taking this limit, we obtain

$\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Pr}(X_n\le a)=\mathrm{Pr}(X\le a),$

which means that {X_n} converges to X in distribution.

Proof for the generic case

We see that |X_n − X| converges in probability to zero, and also X converges to X in distribution trivially. Applying the property proved later on this page we conclude that X_n converges to X in distribution.

<propB1>...</propB1> Convergence in distribution to a constant implies convergence in probability

$X_n\stackrel{d}{\to }c\Rightarrow X_n\stackrel{p}{\to }c,$ provided c is a constant.

Proof: Fix ε > 0. Let B_ε(c) be the open ball of radius ε around point c, and B_ε^c(c)its complement. Then

$\mathrm{Pr}\left(\right|X_n-c|\ge \epsilon )=\mathrm{Pr}(X_n\in B_{\epsilon }^c(c)).$

By the portmanteau lemma (part C), if X_n converges in distribution to c, then the limsup of the latter probability must be less than or equal to Pr(c ∈ B_ε^c(c)),which is obviously equal to zero. Therefore

$\begin{array}{rl}\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Pr}(|X_n-c|\ge \epsilon )& \le {\mathrm{lim\; sup}}_{n\to \mathrm{\infty }}\mathrm{Pr}(|X_n-c|\ge \epsilon )\\ & ={\mathrm{lim\; sup}}_{n\to \mathrm{\infty }}\mathrm{Pr}(X_n\in B_{\epsilon }^c(c))\\ & \le \mathrm{Pr}(c\in B_{\epsilon }^c(c))=0\end{array}$

which by definition means that X_n converges to c in probability.

<propB2>...</propB2> Convergence in probability to a sequence converging in distribution implies convergence to the same distribution

$|Y_n-X_n|\stackrel{p}{\to }0,X_n\stackrel{d}{\to }X\Rightarrow Y_n\stackrel{d}{\to }X$

Proof: We will prove this theorem using the portmanteau lemma, part B. As required in that lemma, consider any bounded function f (i.e. |f(x)| ≤ M) which is also Lipschitz:

$\mathrm{\exists }K>0,\mathrm{\forall }x,y:|f(x)-f(y)|\le K|x-y|.$

Take some ε > 0 and majorize the expression |E[f(Y_n)] − E[f(X_n)]| as

$\begin{array}{rl}|\mathrm{E}[f(Y_n)]-\mathrm{E}[f(X_n)]|& \le \mathrm{E}\left[|f(Y_n)-f(X_n)|\right]\\ & =\mathrm{E}\left[|f(Y_n)-f(X_n)|{\mathbf{1}}_{\left\{\right|Y_n-X_n|<\epsilon \}}\right]+\mathrm{E}\left[|f(Y_n)-f(X_n)|{\mathbf{1}}_{\left\{\right|Y_n-X_n|\ge \epsilon \}}\right]\\ & \le \mathrm{E}\left[K|Y_n-X_n|{\mathbf{1}}_{\left\{\right|Y_n-X_n|<\epsilon \}}\right]+\mathrm{E}\left[2M{\mathbf{1}}_{\left\{\right|Y_n-X_n|\ge \epsilon \}}\right]\\ & \le K\epsilon \mathrm{Pr}(|Y_n-X_n|<\epsilon )+2M\mathrm{Pr}(|Y_n-X_n|\ge \epsilon )\\ & \le K\epsilon +2M\mathrm{Pr}(|Y_n-X_n|\ge \epsilon )\end{array}$

(here 1_{...} denotes the indicator function; the expectation of the indicator function is equal to the probability of corresponding event). Therefore

$\begin{array}{rl}|\mathrm{E}[f(Y_n)]-\mathrm{E}[f(X)]|& \le |\mathrm{E}[f(Y_n)]-\mathrm{E}[f(X_n)]|+|\mathrm{E}[f(X_n)]-\mathrm{E}[f(X)]|\\ & \le K\epsilon +2M\mathrm{Pr}\left(\right|Y_n-X_n|\ge \epsilon )+|\mathrm{E}[f(X_n)]-\mathrm{E}[f(X)]|.\end{array}$

If we take the limit in this expression as n → ∞, the second term will go to zero since {Y_n−X_n} converges to zero in probability; and the third term will also converge to zero, by the portmanteau lemma and the fact that X_n converges to X in distribution. Thus

$\underset{n\to \mathrm{\infty }}{\lim }|\mathrm{E}[f(Y_n)]-\mathrm{E}[f(X)]|\le K\epsilon .$

Since ε was arbitrary, we conclude that the limit must in fact be equal to zero, and therefore E[f(Y_n)] → E[f(X)], which again by the portmanteau lemma implies that {Y_n} converges to X in distribution. QED.

<propB3>...</propB3> Convergence of one sequence in distribution and another to a constant implies joint convergence in distribution

$X_n\stackrel{d}{\to }X,Y_n\stackrel{d}{\to }c\Rightarrow (X_n,Y_n)\stackrel{d}{\to }(X,c)$ provided c is a constant.

Proof: We will prove this statement using the portmanteau lemma, part A.

First we want to show that (X_n, c) converges in distribution to (X, c). By the portmanteau lemma this will be true if we can show that E[f(X_n, c)] → E[f(X, c)] for any bounded continuous function f(x, y). So let f be such arbitrary bounded continuous function. Now consider the function of a single variable g(x) := f(x, c). This will obviously be also bounded and continuous, and therefore by the portmanteau lemma for sequence {X_n} converging in distribution to X, we will have that E[g(X_n)] → E[g(X)]. However the latter expression is equivalent to “E[f(X_n, c)] → E[f(X, c)]”, and therefore we now know that (X_n, c) converges in distribution to (X, c).

Secondly, consider |(X_n, Y_n) − (X_n, c)| = |Y_n − c|. This expression converges in probability to zero because Y_n converges in probability to c. Thus we have demonstrated two facts:

$\{\begin{array}{l}|(X_n,Y_n)-(X_n,c)|\stackrel{p}{\to }0,\\ (X_n,c)\stackrel{d}{\to }(X,c).\end{array}$

By the property proved earlier, these two facts imply that (X_n, Y_n) converge in distribution to (X, c).

<propB4>...</propB4> Convergence of two sequences in probability implies joint convergence in probability

$X_n\stackrel{p}{\to }X,Y_n\stackrel{p}{\to }Y\Rightarrow (X_n,Y_n)\stackrel{p}{\to }(X,Y)$

Proof:

$\begin{array}{rl}\mathrm{Pr}(|(X_n,Y_n)-(X,Y)|\ge \epsilon )& \le \mathrm{Pr}\left(\right|X_n-X|+|Y_n-Y|\ge \epsilon )\\ & \le \mathrm{Pr}\left(\right|X_n-X|\ge \frac{\epsilon }{2})+\mathrm{Pr}\left(\right|Y_n-Y|\ge \frac{\epsilon }{2})\end{array}$

Each of the probabilities on the right-hand side converge to zero as n → ∞ by definition of the convergence of {X_n} and {Y_n} in probability to X and Y respectively. Taking the limit we conclude that the left-hand side also converges to zero, and therefore the sequence {(X_n, Y_n)} converges in probability to {(X, Y)}.

See also

• Convergence of random variables

References

• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534