🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:
Response:
Calculated Shard: 115 (from laksa168)

2. Crawled Status Check

Query:
Response:

3. Robots.txt Check

Query:
Response:

4. Spam/Ban Check

Query:
Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄
INDEXABLE
✅
CRAWLED
2 days ago
🤖
ROBOTS ALLOWED

Page Info Filters

FilterStatusConditionDetails
HTTP statusPASSdownload_http_code = 200HTTP 200
Age cutoffPASSdownload_stamp > now() - 6 MONTH0.1 months ago
History dropPASSisNull(history_drop_reason)No drop reason
Spam/banPASSfh_dont_index != 1 AND ml_spam_score = 0ml_spam_score=0
CanonicalPASSmeta_canonical IS NULL OR = '' OR = src_unparsedNot set

Page Details

PropertyValue
URLhttps://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php
Last Crawled2026-04-14 01:15:58 (2 days ago)
First Indexed2018-10-09 15:45:12 (7 years ago)
HTTP Status Code200
Meta TitleStationary and Limiting Distributions
Meta Descriptionnull
Meta Canonicalnull
Boilerpipe Text
Here, we would like to discuss long-term behavior of Markov chains. In particular, we would like to know the fraction of times that the Markov chain spends in each state as n becomes large. More specifically, we would like to study the distributions π ( n ) = [ P ( X n = 0 ) P ( X n = 1 ) ⋯ ] as n → ∞ . To better understand the subject, we will first look at an example and then provide a general analysis. Example Consider a Markov chain with two possible states, S = { 0 , 1 } . In particular, suppose that the transition matrix is given by P = [ 1 − a a b 1 − b ] , where a and b are two real numbers in the interval [ 0 , 1 ] such that 0 < a + b < 2 . Suppose that the system is in state 0 at time n = 0 with probability α , i.e., π ( 0 ) = [ P ( X 0 = 0 ) P ( X 0 = 1 ) ] = [ α 1 − α ] , where α ∈ [ 0 , 1 ] . Using induction (or any other method), show that P n = 1 a + b [ b a b a ] + ( 1 − a − b ) n a + b [ a − a − b b ] . Show that lim n → ∞ P n = 1 a + b [ b a b a ] . Show that lim n → ∞ π ( n ) = [ b a + b a a + b ] . Solution For n = 1 , we have P 1 = [ 1 − a a b 1 − b ] = 1 a + b [ b a b a ] + 1 − a − b a + b [ a − a − b b ] . Assuming that the statement of the problem is true for n , we can write P n + 1 as P n + 1 = P n P = 1 a + b ( [ b a b a ] + ( 1 − a − b ) n [ a − a − b b ] ) ⋅ [ 1 − a a b 1 − b ] = 1 a + b [ b a b a ] + ( 1 − a − b ) n + 1 a + b [ a − a − b b ] , which completes the proof. By assumption 0 < a + b < 2 , which implies − 1 < 1 − a − b < 1 . Thus, lim n → ∞ ( 1 − a − b ) n = 0. Therefore, lim n → ∞ P n = 1 a + b [ b a b a ] . We have lim n → ∞ π ( n ) = lim n → ∞ [ π ( 0 ) P n ] = π ( 0 ) lim n → ∞ P n = [ α 1 − α ] ⋅ 1 a + b [ b a b a ] = [ b a + b a a + b ] . In the above example, the vector lim n → ∞ π ( n ) = [ b a + b a a + b ] is called the limiting distribution of the Markov chain. Note that the limiting distribution does not depend on the initial probabilities α and 1 − α . In other words, the initial state ( X 0 ) does not matter as n becomes large. Thus, for i = 1 , 2 , we can write lim n → ∞ P ( X n = 0 | X 0 = i ) = b a + b , lim n → ∞ P ( X n = 1 | X 0 = i ) = a a + b . Remember that we show P ( X n = j | X 0 = i ) by P i j ( n ) , which is the entry in the i th row and j th column of P n . Limiting Distributions The probability distribution π = [ π 0 , π 1 , π 2 , ⋯ ] is called the limiting distribution of the Markov chain X n if π j = lim n → ∞ P ( X n = j | X 0 = i ) for all i , j ∈ S , and we have ∑ j ∈ S π j = 1. By the above definition, when a limiting distribution exists, it does not depend on the initial state ( X 0 = i ), so we can write π j = lim n → ∞ P ( X n = j ) , for all  j ∈ S . So far we have shown that the Markov chain in Example 11.12 has the following limiting distribution: π = [ π 0 π 1 ] = [ b a + b a a + b ] . Let's now look at mean return times for this Markov chain. Example Consider a Markov chain in Example 11.12 : a Markov chain with two possible states, S = { 0 , 1 } , and the transition matrix P = [ 1 − a a b 1 − b ] , where a and b are two real numbers in the interval [ 0 , 1 ] such that 0 < a + b < 2 . Find the mean return times, r 0 and r 1 , for this Markov chain. Solution We can use the method of the law of total probability that we explained before to find the mean return times ( Example 11.11 ). We can also find r 0 and r 1 directly as follows: Let R be the first return time to state 0 , i.e., r 0 = E [ R | X 0 = 0 ] . If X 0 = 0 , then X 1 = 0 with probability 1 − a , and X 1 = 1 with probability a . Thus, using the law of total probability, and assuming X 0 = 0 , we can write r 0 = E [ R | X 1 = 0 , X 0 = 0 ] P ( X 1 = 0 | X 0 = 0 ) + E [ R | X 1 = 1 , X 0 = 0 ] P ( X 1 = 1 | X 0 = 0 ) = E [ R | X 1 = 0 ] ⋅ ( 1 − a ) + E [ R | X 1 = 1 ] ⋅ a . If X 1 = 0 , then R = 1 , so E [ R | X 1 = 0 ] = 1. If X 1 = 1 , then R ∼ 1 + G e o m e t r i c ( b ) , so E [ R | X 1 = 1 ] = 1 + E [ G e o m e t r i c ( b ) ] = 1 + 1 b . We conclude r 0 = E [ R | X 1 = 0 ] P ( X 1 = 0 | X 0 = 0 ) + E [ R | X 1 = 1 ] P ( X 1 = 1 | X 0 = 0 ) = 1 ⋅ ( 1 − a ) + ( 1 + 1 b ) ⋅ a = a + b b . Similarly, we can obtain the mean return time to state 1 : r 1 = a + b a . We notice that for this example, the mean return times are given by the inverse of the limiting probabilities. In particular, we have r 0 = 1 π 0 , r 1 = 1 π 1 . As we will see shortly, this is not a coincidence. In fact, we can explain this intuitively. The larger the π i is, the smaller the r i will be. For example, if π i = 1 4 , we conclude that the chain is in state i one-fourth of the time. In this case, r i = 4 , which means that on average it takes the chain four time units to go back to state i . The two-state Markov chain discussed above is a "nice" one in the sense that it has a well-defined limiting behavior that does not depend on the initial probability distribution (PMF of X 0 ). However, not all Markov chains are like that. For example, consider the same Markov chain; however, choose a = b = 1 . In this case, the chain has a periodic behavior, i.e., X n + 2 = X n ,  for all  n . In particular, X n = { X 0 if  n  is even X 1 if  n  is odd In this case, the distribution of X n does not converge to a single PMF. Also, the distribution of X n depends on the initial distribution. As another example, if we choose a = b = 0 , the chain will consist of two disconnected nodes. In this case, X n = X 0 ,  for all  n . Here again, the PMF of X n depends on the initial distribution. Now, the question that arises here is: when does a Markov chain have a limiting distribution (that does not depend on the initial PMF)? We will next discuss this question. We will first consider finite Markov chains and then discuss infinite Markov chains. Finite Markov Chains: Here, we consider Markov chains with a finite number of states. In general, a finite Markov chain can consist of several transient as well as recurrent states. As n becomes large the chain will enter a recurrent class and it will stay there forever. Therefore, when studying long-run behaviors we focus only on the recurrent classes. If a finite Markov chain has more than one recurrent class, then the chain will get absorbed in one of the recurrent classes. Thus, the first question is: in which recurrent class does the chain get absorbed? We have already seen how to address this when we discussed absorption probabilities (see Section 11.2.5 , and Problem 2 of in Section 11.2.7 ). Thus, we can limit our attention to the case where our Markov chain consists of one recurrent class. In other words, we have an irreducible Markov chain. Note that as we showed in Example 11.7 , in any finite Markov chain, there is at least one recurrent class. Therefore, in finite irreducible chains, all states are recurrent. It turns out that in this case the Markov chain has a well-defined limiting behavior if it is aperiodic (states have period 1 ). How do we find the limiting distribution? The trick is to find a stationary distribution . Here is the idea: If π = [ π 1 , π 2 , ⋯ ] is a limiting distribution for a Markov chain, then we have π = lim n → ∞ π ( n ) = lim n → ∞ [ π ( 0 ) P n ] . Similarly, we can write π = lim n → ∞ π ( n + 1 ) = lim n → ∞ [ π ( 0 ) P n + 1 ] = lim n → ∞ [ π ( 0 ) P n P ] = [ lim n → ∞ π ( 0 ) P n ] P = π P . We can explain the equation π = π P intuitively: Suppose that X n has distribution π . As we saw before, π P gives the probability distribution of X n + 1 . If we have π = π P , we conclude that X n and X n + 1 have the same distribution. In other words, the chain has reached its steady-state (limiting) distribution. We can equivalently write π = π P as π j = ∑ k ∈ S π k P k j ,  for all  j ∈ S . The righthand side gives the probability of going to state j in the next step. When we equate both sides, we are implying that the probability of being in state j in the next step is the same as the probability of being in state j now. Example Consider a Markov chain in Example 11.12 : a Markov chain with two possible states, S = { 0 , 1 } , and the transition matrix P = [ 1 − a a b 1 − b ] , where a and b are two real numbers in the interval [ 0 , 1 ] such that 0 < a + b < 2 . Using π = π P , find the limiting distribution of this Markov chain. Solution Let π = [ π 0 , π 1 ] . Then, we can write [ π 0 , π 1 ] = π P = [ π 0 , π 1 ] [ 1 − a a b 1 − b ] = [ π 0 ( 1 − a ) + π 1 b π 0 a + π 1 ( 1 − b ) ] . We obtain two equations; however, they both simplify to π 0 a = π 1 b . We remember that π must be a valid probability distribution, i.e., π 0 + π 1 = 1 . Thus, we can obtain a unique solution, i.e., π = [ π 0 π 1 ] = [ b a + b a a + b ] which is the same answer that we obtained previously. We now summarize the above discussion in the following theorem. Consider a finite Markov chain { X n , n = 0 , 1 , 2 , . . . } where X n ∈ S = { 0 , 1 , 2 , ⋯ , r } . Assume that the chain is irreducible and aperiodic . Then, The set of equations π = π P , ∑ j ∈ S π j = 1 has a unique solution. The unique solution to the above equations is the limiting distribution of the Markov chain, i.e., π j = lim n → ∞ P ( X n = j | X 0 = i ) , for all i , j ∈ S . We have r j = 1 π j , for all  j ∈ S , where r j is the mean return time to state j . In practice, if we are given a finite irreducible Markov chain with states 0 , 1 , 2 , ⋯ , r , we first find a stationary distribution. That is, we find a probability distribution π that satisfies π j = ∑ k ∈ S π k P k j ,  for all  j ∈ S , ∑ j ∈ S π j = 1. In this case, if the chain is also aperiodic, we conclude that the stationary distribution is a limiting distribution. Example Consider the Markov chain shown in Figure 11.14. Figure 11.14- A state transition diagram. Is this chain irreducible? Is this chain aperiodic? Find the stationary distribution for this chain. Is the stationary distribution a limiting distribution for the chain? Solution The chain is irreducible since we can go from any state to any other states in a finite number of steps. Since there is a self-transition, i.e., p 11 > 0 , we conclude that the chain is aperiodic. To find the stationary distribution, we need to solve π 1 = 1 4 π 1 + 1 3 π 2 + 1 2 π 3 , π 2 = 1 2 π 1 , π 3 = 1 4 π 1 + 2 3 π 2 + 1 2 π 3 , π 1 + π 2 + π 3 = 1. We find π 1 = 3 8 , π 2 = 3 16 , π 3 = 7 16 . Since the chain is irreducible and aperiodic, we conclude that the above stationary distribution is a limiting distribution. Countably Infinite Markov Chains: When a Markov chain has an infinite (but countable) number of states, we need to distinguish between two types of recurrent states: positive recurrent and null recurrent states. Remember that if state i is recurrent, then that state will be visited an infinite number of times (any time that we visit that state, we will return to it with probability one in the future). We previously defined r i as the expected number of transitions between visits to state i . Consider a recurrent state i . If r i < ∞ , then state i is a positive recurrent state. Otherwise, it is called null recurrent. Let i be a recurrent state. Assuming X 0 = i , let R i be the number of transitions needed to return to state i , i.e., R i = min { n ≥ 1 : X n = i } . If r i = E [ R i | X 0 = i ] < ∞ , then i is said to be positive recurrent . If E [ R i | X 0 = i ] = ∞ , then i is said to be null recurrent . Theorem Consider an infinite Markov chain { X n , n = 0 , 1 , 2 , . . . } where X n ∈ S = { 0 , 1 , 2 , ⋯ } . Assume that the chain is irreducible and aperiodic . Then, one of the following cases can occur: All states are transient , and lim n → ∞ P ( X n = j | X 0 = i ) = 0 ,  for all  i , j . All states are null recurrent , and lim n → ∞ P ( X n = j | X 0 = i ) = 0 ,  for all  i , j . All states are positive recurrent . In this case, there exists a limiting distribution, π = [ π 0 , π 1 , ⋯ ] , where π j = lim n → ∞ P ( X n = j | X 0 = i ) > 0 , for all i , j ∈ S . The limiting distribution is the unique solution to the equations π j = ∑ k = 0 ∞ π k P k j ,  for  j = 0 , 1 , 2 , ⋯ , ∑ j = 0 ∞ π j = 1. We also have r j = 1 π j , for all  j = 0 , 1 , 2 , ⋯ , where r j is the mean return time to state j . How do we use the above theorem? Consider an infinite Markov chain { X n , n = 0 , 1 , 2 , . . . } , where X n ∈ S = { 0 , 1 , 2 , ⋯ } . Assume that the chain is irreducible and aperiodic. We first try to find a stationary distribution π by solving the equations π j = ∑ k = 0 ∞ π k P k j ,  for  j = 0 , 1 , 2 , ⋯ , ∑ j = 0 ∞ π j = 1. If the above equations have a unique solution, we conclude that the chain is positive recurrent and the stationary distribution is the limiting distribution of this chain. On the other hand, if no stationary solution exists, we conclude that the chain is either transient or null recurrent, so lim n → ∞ P ( X n = j | X 0 = i ) = 0 ,  for all  i , j . Example Consider the Markov chain shown in Figure 11.15. Assume that 0 < p < 1 2 . Does this chain have a limiting distribution? Figure 11.15 - A state transition diagram. Solution This chain is irreducible since all states communicate with each other. It is also aperiodic since it includes a self-transition, P 00 > 0 . Let's write the equations for a stationary distribution. For state 0 , we can write π 0 = ( 1 − p ) π 0 + ( 1 − p ) π 1 , which results in π 1 = p 1 − p π 0 . For state 1 , we can write π 1 = p π 0 + ( 1 − p ) π 2 = ( 1 − p ) π 1 + ( 1 − p ) π 2 , which results in π 2 = p 1 − p π 1 . Similarly, for any j ∈ { 1 , 2 , ⋯ } , we obtain π j = α π j − 1 , where α = p 1 − p . Note that since 0 < p < 1 2 , we conclude that 0 < α < 1 . We obtain π j = α j π 0 ,  for  j = 1 , 2 , ⋯   . Finally, we must have 1 = ∑ j = 0 ∞ π j = ∑ j = 0 ∞ α j π 0 , ( where  0 < α < 1 ) = 1 1 − α π 0 ( geometric series ) . Thus, π 0 = 1 − α . Therefore, the stationary distribution is given by π j = ( 1 − α ) α j ,  for  j = 0 , 1 , 2 , ⋯   . Since this chain is irreducible and aperiodic and we have found a stationary distribution, we conclude that all states are positive recurrent and π = [ π 0 , π 1 , ⋯ ] is the limiting distribution. The print version of the book is available on Amazon . Practical uncertainty: Useful Ideas in Decision-Making, Risk, Randomness, & AI
Markdown
- [HOME](https://www.probabilitycourse.com/) - [VIDEOS](https://www.probabilitycourse.com/videos/videos.php) - [CALCULATOR](https://www.probabilitycourse.com/calculator/calculator.php) - [COMMENTS](https://www.probabilitycourse.com/comments.php) - [COURSES](https://www.probabilitycourse.com/courses.php) - [FOR INSTRUCTOR](https://www.probabilitycourse.com/for_instructors.php) - [LOG IN](https://www.probabilitycourse.com/Login/mobile_login.php) [![](https://www.probabilitycourse.com/images/Template/logo_probability_website.png)](https://www.probabilitycourse.com/) Chapters Menu - [HOME](https://www.probabilitycourse.com/) - [VIDEOS](https://www.probabilitycourse.com/videos/videos.php) - [CALCULATOR](https://www.probabilitycourse.com/calculator/calculator.php) - [COMMENTS](https://www.probabilitycourse.com/comments.php) - [COURSES](https://www.probabilitycourse.com/courses.php) - [FOR INSTRUCTORS](https://www.probabilitycourse.com/for_instructors.php) - [Sign In]() [Forgot password?](https://www.probabilitycourse.com/Login/forgot_password.php) [←]() [previous](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) [next](https://www.probabilitycourse.com/chapter11/chapter11/11_2_7_solved_probs.php) [→]() *** ## 11\.2.6 Stationary and Limiting Distributions Here, we would like to discuss long-term behavior of Markov chains. In particular, we would like to know the fraction of times that the Markov chain spends in each state as n n becomes large. More specifically, we would like to study the distributions π(n)\=\[P(Xn\=0)P(Xn\=1)⋯\] π ( n ) \= \[ P ( X n \= 0 ) P ( X n \= 1 ) ⋯ \] as n→∞ n → ∞ . To better understand the subject, we will first look at an example and then provide a general analysis. *** Example Consider a Markov chain with two possible states, S\={0,1} S \= { 0 , 1 } . In particular, suppose that the transition matrix is given by P\=\[1−aba1−b\], P \= \[ 1 − a a b 1 − b \] , where a a and b b are two real numbers in the interval \[0,1\] \[ 0 , 1 \] such that 0\<a\+b\<2 0 \< a \+ b \< 2 . Suppose that the system is in state 0 0 at time n\=0 n \= 0 with probability α α , i.e., π(0)\=\[P(X0\=0)P(X0\=1)\]\=\[α1−α\], π ( 0 ) \= \[ P ( X 0 \= 0 ) P ( X 0 \= 1 ) \] \= \[ α 1 − α \] , where α∈\[0,1\] α ∈ \[ 0 , 1 \] . 1. Using induction (or any other method), show that Pn\=1a\+b\[bbaa\]\+(1−a−b)na\+b\[a−b−ab\]. P n \= 1 a \+ b \[ b a b a \] \+ ( 1 − a − b ) n a \+ b \[ a − a − b b \] . 2. Show that limn→∞Pn\=1a\+b\[bbaa\]. lim n → ∞ P n \= 1 a \+ b \[ b a b a \] . 3. Show that limn→∞π(n)\=\[ba\+baa\+b\]. lim n → ∞ π ( n ) \= \[ b a \+ b a a \+ b \] . - [**Solution**]() - 1. For n\=1 n \= 1 , we have P1\=\[1−aba1−b\]\=1a\+b\[bbaa\]\+1−a−ba\+b\[a−b−ab\]. P 1 \= \[ 1 − a a b 1 − b \] \= 1 a \+ b \[ b a b a \] \+ 1 − a − b a \+ b \[ a − a − b b \] . Assuming that the statement of the problem is true for n n , we can write Pn\+1 P n \+ 1 as Pn\+1\=PnP\=1a\+b(\[bbaa\]\+(1−a−b)n\[a−b−ab\])⋅\[1−aba1−b\]\=1a\+b\[bbaa\]\+(1−a−b)n\+1a\+b\[a−b−ab\], P n \+ 1 \= P n P \= 1 a \+ b ( \[ b a b a \] \+ ( 1 − a − b ) n \[ a − a − b b \] ) ⋅ \[ 1 − a a b 1 − b \] \= 1 a \+ b \[ b a b a \] \+ ( 1 − a − b ) n \+ 1 a \+ b \[ a − a − b b \] , which completes the proof. 2. By assumption 0\<a\+b\<2 0 \< a \+ b \< 2 , which implies −1\<1−a−b\<1 − 1 \< 1 − a − b \< 1 . Thus, limn→∞(1−a−b)n\=0\. lim n → ∞ ( 1 − a − b ) n \= 0\. Therefore, limn→∞Pn\=1a\+b\[bbaa\]. lim n → ∞ P n \= 1 a \+ b \[ b a b a \] . 3. We have limn→∞π(n)\=limn→∞\[π(0)Pn\]\=π(0)limn→∞Pn\=\[α1−α\]⋅1a\+b\[bbaa\]\=\[ba\+baa\+b\]. lim n → ∞ π ( n ) \= lim n → ∞ \[ π ( 0 ) P n \] \= π ( 0 ) lim n → ∞ P n \= \[ α 1 − α \] ⋅ 1 a \+ b \[ b a b a \] \= \[ b a \+ b a a \+ b \] . *** In the above example, the vector limn→∞π(n)\=\[ba\+baa\+b\] lim n → ∞ π ( n ) \= \[ b a \+ b a a \+ b \] is called the *limiting distribution* of the Markov chain. Note that the limiting distribution does not depend on the initial probabilities α α and 1−α 1 − α . In other words, the initial state ( X0 X 0 ) does not matter as n n becomes large. Thus, for i\=1,2 i \= 1 , 2 , we can write limn→∞P(Xn\=0\|X0\=i)\=ba\+b,limn→∞P(Xn\=1\|X0\=i)\=aa\+b. lim n → ∞ P ( X n \= 0 \| X 0 \= i ) \= b a \+ b , lim n → ∞ P ( X n \= 1 \| X 0 \= i ) \= a a \+ b . Remember that we show P(Xn\=j\|X0\=i) P ( X n \= j \| X 0 \= i ) by P(n)ij P i j ( n ) , which is the entry in the i i th row and j j th column of Pn P n . Limiting Distributions The probability distribution π\=\[π0,π1,π2,⋯\] π \= \[ π 0 , π 1 , π 2 , ⋯ \] is called the **limiting distribution** of the Markov chain Xn X n if πj\=limn→∞P(Xn\=j\|X0\=i) π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) for all i,j∈S i , j ∈ S , and we have ∑j∈Sπj\=1\. ∑ j ∈ S π j \= 1\. By the above definition, when a limiting distribution exists, it does not depend on the initial state ( X0\=i X 0 \= i ), so we can write πj\=limn→∞P(Xn\=j),for all j∈S. π j \= lim n → ∞ P ( X n \= j ) , for all j ∈ S . So far we have shown that the Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12) has the following limiting distribution: π\=\[π0π1\]\=\[ba\+baa\+b\]. π \= \[ π 0 π 1 \] \= \[ b a \+ b a a \+ b \] . Let's now look at mean return times for this Markov chain. *** Example Consider a Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12): a Markov chain with two possible states, S\={0,1} S \= { 0 , 1 } , and the transition matrix P\=\[1−aba1−b\], P \= \[ 1 − a a b 1 − b \] , where a a and b b are two real numbers in the interval \[0,1\] \[ 0 , 1 \] such that 0\<a\+b\<2 0 \< a \+ b \< 2 . Find the mean return times, r0 r 0 and r1 r 1 , for this Markov chain. - [**Solution**]() - We can use the method of the law of total probability that we explained before to find the mean return times ([Example 11.11](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php#example11_11)). We can also find r0 r 0 and r1 r 1 directly as follows: Let R R be the first return time to state 0 0, i.e., r0\=E\[R\|X0\=0\] r 0 \= E \[ R \| X 0 \= 0 \]. If X0\=0 X 0 \= 0, then X1\=0 X 1 \= 0 with probability 1−a 1 − a, and X1\=1 X 1 \= 1 with probability a a. Thus, using the law of total probability, and assuming X0\=0 X 0 \= 0, we can write r0\=E\[R\|X1\=0,X0\=0\]P(X1\=0\|X0\=0)\+E\[R\|X1\=1,X0\=0\]P(X1\=1\|X0\=0)\=E\[R\|X1\=0\]⋅(1−a)\+E\[R\|X1\=1\]⋅a. r 0 \= E \[ R \| X 1 \= 0 , X 0 \= 0 \] P ( X 1 \= 0 \| X 0 \= 0 ) \+ E \[ R \| X 1 \= 1 , X 0 \= 0 \] P ( X 1 \= 1 \| X 0 \= 0 ) \= E \[ R \| X 1 \= 0 \] ⋅ ( 1 − a ) \+ E \[ R \| X 1 \= 1 \] ⋅ a . If X1\=0 X 1 \= 0 , then R\=1 R \= 1 , so E\[R\|X1\=0\]\=1\. E \[ R \| X 1 \= 0 \] \= 1\. If X1\=1 X 1 \= 1 , then R∼1\+Geometric(b) R ∼ 1 \+ G e o m e t r i c ( b ) , so E\[R\|X1\=1\]\=1\+E\[Geometric(b)\]\=1\+1b. E \[ R \| X 1 \= 1 \] \= 1 \+ E \[ G e o m e t r i c ( b ) \] \= 1 \+ 1 b . We conclude r0\=E\[R\|X1\=0\]P(X1\=0\|X0\=0)\+E\[R\|X1\=1\]P(X1\=1\|X0\=0)\=1⋅(1−a)\+(1\+1b)⋅a\=a\+bb. r 0 \= E \[ R \| X 1 \= 0 \] P ( X 1 \= 0 \| X 0 \= 0 ) \+ E \[ R \| X 1 \= 1 \] P ( X 1 \= 1 \| X 0 \= 0 ) \= 1 ⋅ ( 1 − a ) \+ ( 1 \+ 1 b ) ⋅ a \= a \+ b b . Similarly, we can obtain the mean return time to state 1 1 : r1\=a\+ba. r 1 \= a \+ b a . We notice that for this example, the mean return times are given by the inverse of the limiting probabilities. In particular, we have r0\=1π0,r1\=1π1. r 0 \= 1 π 0 , r 1 \= 1 π 1 . As we will see shortly, this is not a coincidence. In fact, we can explain this intuitively. The larger the πi π i is, the smaller the ri r i will be. For example, if πi\=14 π i \= 1 4 , we conclude that the chain is in state i i one-fourth of the time. In this case, ri\=4 r i \= 4 , which means that on average it takes the chain four time units to go back to state i i . *** The two-state Markov chain discussed above is a "nice" one in the sense that it has a well-defined limiting behavior that does not depend on the initial probability distribution (PMF of X0 X 0 ). However, not all Markov chains are like that. For example, consider the same Markov chain; however, choose a\=b\=1 a \= b \= 1 . In this case, the chain has a periodic behavior, i.e., Xn\+2\=Xn, for all n. X n \+ 2 \= X n , for all n . In particular, Xn\=⎧⎩⎨⎪⎪X0X1if n is evenif n is odd X n \= { X 0 if n is even X 1 if n is odd In this case, the distribution of Xn X n does not converge to a single PMF. Also, the distribution of Xn X n depends on the initial distribution. As another example, if we choose a\=b\=0 a \= b \= 0 , the chain will consist of two disconnected nodes. In this case, Xn\=X0, for all n. X n \= X 0 , for all n . Here again, the PMF of Xn X n depends on the initial distribution. Now, the question that arises here is: when does a Markov chain have a limiting distribution (that does not depend on the initial PMF)? We will next discuss this question. We will first consider finite Markov chains and then discuss infinite Markov chains. ### Finite Markov Chains: Here, we consider Markov chains with a finite number of states. In general, a finite Markov chain can consist of several transient as well as recurrent states. As n n becomes large the chain will enter a recurrent class and it will stay there forever. Therefore, when studying long-run behaviors we focus only on the recurrent classes. If a finite Markov chain has more than one recurrent class, then the chain will get absorbed in one of the recurrent classes. Thus, the first question is: in which recurrent class does the chain get absorbed? We have already seen how to address this when we discussed absorption probabilities (see [Section 11.2.5](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php), and [Problem 2](https://www.probabilitycourse.com/chapter11/chapter11/11_2_7_solved_probs.php#problem11_2) of in [Section 11.2.7](https://www.probabilitycourse.com/chapter11/chapter11/11_2_7_solved_probs.php)). Thus, we can limit our attention to the case where our Markov chain consists of one recurrent class. In other words, we have an irreducible Markov chain. Note that as we showed in [Example 11.7](https://www.probabilitycourse.com/chapter11/chapter11/11_2_4_classification_of_states.php#example11_7), in any finite Markov chain, there is at least one recurrent class. Therefore, in finite irreducible chains, all states are recurrent. It turns out that in this case the Markov chain has a well-defined limiting behavior if it is aperiodic (states have period 1 1 ). How do we find the limiting distribution? The trick is to find a *stationary distribution*. Here is the idea: If π\=\[π1,π2,⋯\] π \= \[ π 1 , π 2 , ⋯ \] is a limiting distribution for a Markov chain, then we have π\=limn→∞π(n)\=limn→∞\[π(0)Pn\]. π \= lim n → ∞ π ( n ) \= lim n → ∞ \[ π ( 0 ) P n \] . Similarly, we can write π\=limn→∞π(n\+1)\=limn→∞\[π(0)Pn\+1\]\=limn→∞\[π(0)PnP\]\=\[limn→∞π(0)Pn\]P\=πP. π \= lim n → ∞ π ( n \+ 1 ) \= lim n → ∞ \[ π ( 0 ) P n \+ 1 \] \= lim n → ∞ \[ π ( 0 ) P n P \] \= \[ lim n → ∞ π ( 0 ) P n \] P \= π P . We can explain the equation π\=πP π \= π P intuitively: Suppose that Xn X n has distribution π π . As we saw before, πP π P gives the probability distribution of Xn\+1 X n \+ 1 . If we have π\=πP π \= π P , we conclude that Xn X n and Xn\+1 X n \+ 1 have the same distribution. In other words, the chain has reached its *steady-state* (limiting) distribution. We can equivalently write π\=πP π \= π P as πj\=∑k∈SπkPkj, for all j∈S. π j \= ∑ k ∈ S π k P k j , for all j ∈ S . The righthand side gives the probability of going to state j j in the next step. When we equate both sides, we are implying that the probability of being in state j j in the next step is the same as the probability of being in state j j now. *** Example Consider a Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12): a Markov chain with two possible states, S\={0,1} S \= { 0 , 1 } , and the transition matrix P\=\[1−aba1−b\], P \= \[ 1 − a a b 1 − b \] , where a a and b b are two real numbers in the interval \[0,1\] \[ 0 , 1 \] such that 0\<a\+b\<2 0 \< a \+ b \< 2 . Using π\=πP, π \= π P , find the limiting distribution of this Markov chain. - [**Solution**]() - Let π\=\[π0,π1\] π \= \[ π 0 , π 1 \] . Then, we can write \[π0,π1\]\=πP\=\[π0,π1\]\[1−aba1−b\]\=\[π0(1−a)\+π1bπ0a\+π1(1−b)\]. \[ π 0 , π 1 \] \= π P \= \[ π 0 , π 1 \] \[ 1 − a a b 1 − b \] \= \[ π 0 ( 1 − a ) \+ π 1 b π 0 a \+ π 1 ( 1 − b ) \] . We obtain two equations; however, they both simplify to π0a\=π1b. π 0 a \= π 1 b . We remember that π π must be a valid probability distribution, i.e., π0\+π1\=1 π 0 \+ π 1 \= 1 . Thus, we can obtain a unique solution, i.e., π\=\[π0π1\]\=\[ba\+baa\+b\] π \= \[ π 0 π 1 \] \= \[ b a \+ b a a \+ b \] which is the same answer that we obtained previously. *** We now summarize the above discussion in the following theorem. Consider a finite Markov chain {Xn,n\=0,1,2,...} { X n , n \= 0 , 1 , 2 , . . . } where Xn∈S\={0,1,2,⋯,r} X n ∈ S \= { 0 , 1 , 2 , ⋯ , r } . Assume that the chain is irreducible and aperiodic . Then, 1. The set of equations π\=πP,∑j∈Sπj\=1 π \= π P , ∑ j ∈ S π j \= 1 has a unique solution. 2. The unique solution to the above equations is the limiting distribution of the Markov chain, i.e., πj\=limn→∞P(Xn\=j\|X0\=i), π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) , for all i,j∈S i , j ∈ S . 3. We have rj\=1πj,for all j∈S, r j \= 1 π j , for all j ∈ S , where rj r j is the mean return time to state j j . In practice, if we are given a finite irreducible Markov chain with states 0,1,2,⋯,r 0 , 1 , 2 , ⋯ , r , we first find a stationary distribution. That is, we find a probability distribution π π that satisfies πj\=∑k∈SπkPkj, for all j∈S,∑j∈Sπj\=1\. π j \= ∑ k ∈ S π k P k j , for all j ∈ S , ∑ j ∈ S π j \= 1\. In this case, if the chain is also aperiodic, we conclude that the stationary distribution is a limiting distribution. *** Example Consider the Markov chain shown in Figure 11.14. ![MC-diagram-tek](https://www.probabilitycourse.com/chapter11/images/chapter11/MC-diagram-tek.png) Figure 11.14- A state transition diagram. 1. Is this chain irreducible? 2. Is this chain aperiodic? 3. Find the stationary distribution for this chain. 4. Is the stationary distribution a limiting distribution for the chain? - [**Solution**]() - 1. The chain is irreducible since we can go from any state to any other states in a finite number of steps. 2. Since there is a self-transition, i.e., p11\>0 p 11 \> 0 , we conclude that the chain is aperiodic. 3. To find the stationary distribution, we need to solve π1\=14π1\+13π2\+12π3,π2\=12π1,π3\=14π1\+23π2\+12π3,π1\+π2\+π3\=1\. π 1 \= 1 4 π 1 \+ 1 3 π 2 \+ 1 2 π 3 , π 2 \= 1 2 π 1 , π 3 \= 1 4 π 1 \+ 2 3 π 2 \+ 1 2 π 3 , π 1 \+ π 2 \+ π 3 \= 1\. We find π1\=38,π2\=316,π3\=716. π 1 \= 3 8 , π 2 \= 3 16 , π 3 \= 7 16 . 4. Since the chain is irreducible and aperiodic, we conclude that the above stationary distribution is a limiting distribution. *** ### Countably Infinite Markov Chains: When a Markov chain has an infinite (but countable) number of states, we need to distinguish between two types of recurrent states: *positive* recurrent and *null* recurrent states. Remember that if state i i is recurrent, then that state will be visited an infinite number of times (any time that we visit that state, we will return to it with probability one in the future). We previously defined ri r i as the expected number of transitions between visits to state i i. Consider a recurrent state i i. If ri\<∞ r i \< ∞, then state i i is a *positive* recurrent state. Otherwise, it is called *null* recurrent. Let i i be a recurrent state. Assuming X0\=i X 0 \= i , let Ri R i be the number of transitions needed to return to state i i , i.e., Ri\=min{n≥1:Xn\=i}. R i \= min { n ≥ 1 : X n \= i } . If ri\=E\[Ri\|X0\=i\]\<∞ r i \= E \[ R i \| X 0 \= i \] \< ∞ , then i i is said to be **positive recurrent**. If E\[Ri\|X0\=i\]\=∞ E \[ R i \| X 0 \= i \] \= ∞ , then i i is said to be **null recurrent**. Theorem Consider an infinite Markov chain {Xn,n\=0,1,2,...} { X n , n \= 0 , 1 , 2 , . . . } where Xn∈S\={0,1,2,⋯} X n ∈ S \= { 0 , 1 , 2 , ⋯ } . Assume that the chain is irreducible and aperiodic . Then, one of the following cases can occur: 1. All states are transient , and limn→∞P(Xn\=j\|X0\=i)\=0, for all i,j. lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . 2. All states are null recurrent , and limn→∞P(Xn\=j\|X0\=i)\=0, for all i,j. lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . 3. All states are positive recurrent . In this case, there exists a limiting distribution, π\=\[π0,π1,⋯\] π \= \[ π 0 , π 1 , ⋯ \] , where πj\=limn→∞P(Xn\=j\|X0\=i)\>0, π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) \> 0 , for all i,j∈S i , j ∈ S . The limiting distribution is the unique solution to the equations πj\=∑k\=0∞πkPkj, for j\=0,1,2,⋯,∑j\=0∞πj\=1\. π j \= ∑ k \= 0 ∞ π k P k j , for j \= 0 , 1 , 2 , ⋯ , ∑ j \= 0 ∞ π j \= 1\. We also have rj\=1πj,for all j\=0,1,2,⋯, r j \= 1 π j , for all j \= 0 , 1 , 2 , ⋯ , where rj r j is the mean return time to state j j . How do we use the above theorem? Consider an infinite Markov chain {Xn,n\=0,1,2,...} { X n , n \= 0 , 1 , 2 , . . . } , where Xn∈S\={0,1,2,⋯} X n ∈ S \= { 0 , 1 , 2 , ⋯ } . Assume that the chain is irreducible and aperiodic. We first try to find a stationary distribution π π by solving the equations πj\=∑k\=0∞πkPkj, for j\=0,1,2,⋯,∑j\=0∞πj\=1\. π j \= ∑ k \= 0 ∞ π k P k j , for j \= 0 , 1 , 2 , ⋯ , ∑ j \= 0 ∞ π j \= 1\. If the above equations have a unique solution, we conclude that the chain is positive recurrent and the stationary distribution is the limiting distribution of this chain. On the other hand, if no stationary solution exists, we conclude that the chain is either transient or null recurrent, so limn→∞P(Xn\=j\|X0\=i)\=0, for all i,j. lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . *** Example Consider the Markov chain shown in Figure 11.15. Assume that 0\<p\<12 0 \< p \< 1 2 . Does this chain have a limiting distribution? ![MC-diagram-inf-1](https://www.probabilitycourse.com/chapter11/images/chapter11/MC-diagram-inf-1.png) Figure 11.15 - A state transition diagram. - [**Solution**]() - This chain is irreducible since all states communicate with each other. It is also aperiodic since it includes a self-transition, P00\>0 P 00 \> 0 . Let's write the equations for a stationary distribution. For state 0 0 , we can write π0\=(1−p)π0\+(1−p)π1, π 0 \= ( 1 − p ) π 0 \+ ( 1 − p ) π 1 , which results in π1\=p1−pπ0. π 1 \= p 1 − p π 0 . For state 1 1 , we can write π1\=pπ0\+(1−p)π2\=(1−p)π1\+(1−p)π2, π 1 \= p π 0 \+ ( 1 − p ) π 2 \= ( 1 − p ) π 1 \+ ( 1 − p ) π 2 , which results in π2\=p1−pπ1. π 2 \= p 1 − p π 1 . Similarly, for any j∈{1,2,⋯} j ∈ { 1 , 2 , ⋯ } , we obtain πj\=απj−1, π j \= α π j − 1 , where α\=p1−p α \= p 1 − p . Note that since 0\<p\<12 0 \< p \< 1 2 , we conclude that 0\<α\<1 0 \< α \< 1 . We obtain πj\=αjπ0, for j\=1,2,⋯ . π j \= α j π 0 , for j \= 1 , 2 , ⋯ . Finally, we must have 1\=∑j\=0∞πj\=∑j\=0∞αjπ0,\=11−απ0(where 0\<α\<1)(geometric series). 1 \= ∑ j \= 0 ∞ π j \= ∑ j \= 0 ∞ α j π 0 , ( where 0 \< α \< 1 ) \= 1 1 − α π 0 ( geometric series ) . Thus, π0\=1−α π 0 \= 1 − α . Therefore, the stationary distribution is given by πj\=(1−α)αj, for j\=0,1,2,⋯ . π j \= ( 1 − α ) α j , for j \= 0 , 1 , 2 , ⋯ . Since this chain is irreducible and aperiodic and we have found a stationary distribution, we conclude that all states are positive recurrent and π\=\[π0,π1,⋯\] π \= \[ π 0 , π 1 , ⋯ \] is the limiting distribution. *** [←]() [previous](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) [next](https://www.probabilitycourse.com/chapter11/chapter11/11_2_7_solved_probs.php) [→]() *** | | |---| | The print version of the book is available on [Amazon](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https://www.probabilitycourse.com/images/home/book_cover.jpg)](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) | | **Practical uncertainty:** *Useful Ideas in Decision-Making, Risk, Randomness, & AI* [![ractical Uncertaintly Cover](https://www.probabilitycourse.com/images/home/practical_uncertaintly.PNG)](https://www.amazon.com/dp/B0CH2BHRVH/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) | Open Menu - [0 Preface](https://www.probabilitycourse.com/preface.php) - [1 Basic Concepts]() - [1\.0 Introduction](https://www.probabilitycourse.com/chapter1/1_0_0_introduction.php) - [1\.1 Introduction]() - [1\.1.0 What Is Probability?](https://www.probabilitycourse.com/chapter1/1_1_0_what_is_probability.php) - [1\.1.1 Example](https://www.probabilitycourse.com/chapter1/1_1_1_example.php) - [1\.2 Review of Set Theory]() - [1\.2.0 Review](https://www.probabilitycourse.com/chapter1/1_2_0_review_set_theory.php) - [1\.2.1 Venn Diagrams](https://www.probabilitycourse.com/chapter1/1_2_1_venn.php) - [1\.2.2 Set Operations](https://www.probabilitycourse.com/chapter1/1_2_2_set_operations.php) - [1\.2.3 Cardinality](https://www.probabilitycourse.com/chapter1/1_2_3_cardinality.php) - [1\.2.4 Functions](https://www.probabilitycourse.com/chapter1/1_2_4_functions.php) - [1\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter1/1_2_5_solved1.php) - [1\.3 Random Experiments and Probabilities]() - [1\.3.1 Random Experiments](https://www.probabilitycourse.com/chapter1/1_3_1_random_experiments.php) - [1\.3.2 Probability](https://www.probabilitycourse.com/chapter1/1_3_2_probability.php) - [1\.3.3 Finding Probabilities](https://www.probabilitycourse.com/chapter1/1_3_3_finding_probabilities.php) - [1\.3.4 Discrete Models](https://www.probabilitycourse.com/chapter1/1_3_4_discrete_models.php) - [1\.3.5 Continuous Models](https://www.probabilitycourse.com/chapter1/1_3_5_continuous_models.php) - [1\.3.6 Solved Problems](https://www.probabilitycourse.com/chapter1/1_3_6_solved2.php) - [1\.4 Conditional Probability]() - [1\.4.0 Conditional Probability](https://www.probabilitycourse.com/chapter1/1_4_0_conditional_probability.php) - [1\.4.1 Independence](https://www.probabilitycourse.com/chapter1/1_4_1_independence.php) - [1\.4.2 Law of Total Probability](https://www.probabilitycourse.com/chapter1/1_4_2_total_probability.php) - [1\.4.3 Bayes' Rule](https://www.probabilitycourse.com/chapter1/1_4_3_bayes_rule.php) - [1\.4.4 Conditional Independence](https://www.probabilitycourse.com/chapter1/1_4_4_conditional_independence.php) - [1\.4.5 Solved Problems](https://www.probabilitycourse.com/chapter1/1_4_5_solved3.php) - [1\.5 Problems]() - [1\.5.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter1/1_5_0_chapter1_problems.php) - [2 Combinatorics: Counting Methods]() - [2\.1 Combinatorics]() - [2\.1.0 Finding Probabilities with Counting Methods](https://www.probabilitycourse.com/chapter2/2_1_0_counting.php) - [2\.1.1 Ordered with Replacement](https://www.probabilitycourse.com/chapter2/2_1_1_ordered_with_replacement.php) - [2\.1.2 Ordered without Replacement](https://www.probabilitycourse.com/chapter2/2_1_2_ordered_without_replacement.php) - [2\.1.3 Unordered without Replacement](https://www.probabilitycourse.com/chapter2/2_1_3_unordered_without_replacement.php) - [2\.1.4 Unordered with Replacement](https://www.probabilitycourse.com/chapter2/2_1_4_unordered_with_replacement.php) - [2\.1.5 Solved Problems](https://www.probabilitycourse.com/chapter2/2_1_5_solved2_1.php) - [2\.2 Problems]() - [2\.2.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter2/2_3_0_chapter2_problems.php) - [3 Discrete Random Variables]() - [3\.1 Basic Concepts]() - [3\.1.1 Random Variables](https://www.probabilitycourse.com/chapter3/3_1_1_random_variables.php) - [3\.1.2 Discrete Random Variables](https://www.probabilitycourse.com/chapter3/3_1_2_discrete_random_var.php) - [3\.1.3 Probability Mass Function](https://www.probabilitycourse.com/chapter3/3_1_3_pmf.php) - [3\.1.4 Independent Random Variables](https://www.probabilitycourse.com/chapter3/3_1_4_independent_random_var.php) - [3\.1.5 Special Distributions](https://www.probabilitycourse.com/chapter3/3_1_5_special_discrete_distr.php) - [3\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter3/3_1_6_solved3_1.php) - [3\.2 More about Discrete Random Variables]() - [3\.2.1 Cumulative Distribution Function](https://www.probabilitycourse.com/chapter3/3_2_1_cdf.php) - [3\.2.2 Expectation](https://www.probabilitycourse.com/chapter3/3_2_2_expectation.php) - [3\.2.3 Functions of Random Variables](https://www.probabilitycourse.com/chapter3/3_2_3_functions_random_var.php) - [3\.2.4 Variance](https://www.probabilitycourse.com/chapter3/3_2_4_variance.php) - [3\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter3/3_2_5_solved3_2.php) - [3\.3 Problems]() - [3\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter3/3_3_0_chapter3_problems.php) - [4 Continuous and Mixed Random Variables]() - [4\.0 Introduction](https://www.probabilitycourse.com/chapter4/4_0_0_intro.php) - [4\.1 Continuous Random Variables]() - [4\.1.0 Continuous Random Variables and their Distributions](https://www.probabilitycourse.com/chapter4/4_1_0_continuous_random_vars_distributions.php) - [4\.1.1 Probability Density Function](https://www.probabilitycourse.com/chapter4/4_1_1_pdf.php) - [4\.1.2 Expected Value and Variance](https://www.probabilitycourse.com/chapter4/4_1_2_expected_val_variance.php) - [4\.1.3 Functions of Continuous Random Variables](https://www.probabilitycourse.com/chapter4/4_1_3_functions_continuous_var.php) - [4\.1.4 Solved Problems](https://www.probabilitycourse.com/chapter4/4_1_4_solved4_1.php) - [4\.2 Special Distributions]() - [4\.2.1 Uniform Distribution](https://www.probabilitycourse.com/chapter4/4_2_1_uniform.php) - [4\.2.2 Exponential Distribution](https://www.probabilitycourse.com/chapter4/4_2_2_exponential.php) - [4\.2.3 Normal (Gaussian) Distribution](https://www.probabilitycourse.com/chapter4/4_2_3_normal.php) - [4\.2.4 Gamma Distribution](https://www.probabilitycourse.com/chapter4/4_2_4_Gamma_distribution.php) - [4\.2.5 Other Distributions](https://www.probabilitycourse.com/chapter4/4_2_5_other_distr.php) - [4\.2.6 Solved Problems](https://www.probabilitycourse.com/chapter4/4_2_6_solved4_2.php) - [4\.3 Mixed Random Variables]() - [4\.3.1 Mixed Random Variables](https://www.probabilitycourse.com/chapter4/4_3_1_mixed.php) - [4\.3.2 Using the Delta Function](https://www.probabilitycourse.com/chapter4/4_3_2_delta_function.php) - [4\.3.3 Solved Problems](https://www.probabilitycourse.com/chapter4/4_3_3_solved4_3.php) - [4\.4 Problems]() - [4\.4.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter4/4_4_0_chapter4_problems.php) - [5 Joint Distributions]() - [5\.1 Two Discrete Random Variables]() - [5\.1.0 Two Random Variables](https://www.probabilitycourse.com/chapter5/5_1_0_joint_distributions.php) - [5\.1.1 Joint Probability Mass Function (PMF)](https://www.probabilitycourse.com/chapter5/5_1_1_joint_pmf.php) - [5\.1.2 Joint Cumulative Distribution Function (CDF)](https://www.probabilitycourse.com/chapter5/5_1_2_joint_cdf.php) - [5\.1.3 Conditioning and Independence](https://www.probabilitycourse.com/chapter5/5_1_3_conditioning_independence.php) - [5\.1.4 Functions of Two Random Variables](https://www.probabilitycourse.com/chapter5/5_1_4_functions_two_variables.php) - [5\.1.5 Conditional Expectation](https://www.probabilitycourse.com/chapter5/5_1_5_conditional_expectation.php) - [5\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter5/5_1_6_solved_prob.php) - [5\.2 Two Continuous Random Variables]() - [5\.2.0 Two Continuous Random Variables](https://www.probabilitycourse.com/chapter5/5_2_0_continuous_vars.php) - [5\.2.1 Joint Probability Density Function](https://www.probabilitycourse.com/chapter5/5_2_1_joint_pdf.php) - [5\.2.2 Joint Cumulative Distribution Function](https://www.probabilitycourse.com/chapter5/5_2_2_joint_cdf.php) - [5\.2.3 Conditioning and Independence](https://www.probabilitycourse.com/chapter5/5_2_3_conditioning_independence.php) - [5\.2.4 Functions of Two Continuous Random Variables](https://www.probabilitycourse.com/chapter5/5_2_4_functions.php) - [5\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter5/5_2_5_solved_prob.php) - [5\.3 More Topics]() - [5\.3.1 Covariance and Correlation](https://www.probabilitycourse.com/chapter5/5_3_1_covariance_correlation.php) - [5\.3.2 Bivariate Normal Distribution](https://www.probabilitycourse.com/chapter5/5_3_2_bivariate_normal_dist.php) - [5\.3.3 Solved Problems](https://www.probabilitycourse.com/chapter5/5_3_3_solved_probs.php) - [5\.4 Problems]() - [5\.4.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter5/5_4_0_chapter_problems.php) - [6 Multiple Random Variables]() - [6\.0 Introduction](https://www.probabilitycourse.com/chapter6/6_0_0_intro.php) - [6\.1 Methods for More Than Two Random Variables]() - [6\.1.1 Joint Distributions and Independence](https://www.probabilitycourse.com/chapter6/6_1_1_joint_distributions_independence.php) - [6\.1.2 Sums of Random Variables](https://www.probabilitycourse.com/chapter6/6_1_2_sums_random_variables.php) - [6\.1.3 Moment Generating Functions](https://www.probabilitycourse.com/chapter6/6_1_3_moment_functions.php) - [6\.1.4 Characteristic Functions](https://www.probabilitycourse.com/chapter6/6_1_4_characteristic_functions.php) - [6\.1.5 Random Vectors](https://www.probabilitycourse.com/chapter6/6_1_5_random_vectors.php) - [6\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter6/6_1_6_solved_probs.php) - [6\.2 Probability Bounds]() - [6\.2.0 Probability Bounds](https://www.probabilitycourse.com/chapter6/6_2_0_probability_bounds.php) - [6\.2.1 Union Bound and Extension](https://www.probabilitycourse.com/chapter6/6_2_1_union_bound_and_exten.php) - [6\.2.2 Markov Chebyshev Inequalities](https://www.probabilitycourse.com/chapter6/6_2_2_markov_chebyshev_inequalities.php) - [6\.2.3 Chernoff Bounds](https://www.probabilitycourse.com/chapter6/6_2_3_chernoff_bounds.php) - [6\.2.4 Cauchy Schwarz Inequality](https://www.probabilitycourse.com/chapter6/6_2_4_cauchy_schwarz.php) - [6\.2.5 Jensen's Inequality](https://www.probabilitycourse.com/chapter6/6_2_5_jensen's_inequality.php) - [6\.2.6 Solved Problems](https://www.probabilitycourse.com/chapter6/6_2_6_solved6_2.php) - [6\.3 Problems]() - [6\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter6/6_3_0_chapter_problems.php) - [7 Limit Theorems and Convergence of Random Variables]() - [7\.0 Introduction](https://www.probabilitycourse.com/chapter7/7_0_0_intro.php) - [7\.1 Limit Theorems]() - [7\.1.0 Limit Theorems](https://www.probabilitycourse.com/chapter7/7_1_0_limit_theorems.php) - [7\.1.1 Law of Large Numbers](https://www.probabilitycourse.com/chapter7/7_1_1_law_of_large_numbers.php) - [7\.1.2 Central Limit Theorem (CLT)](https://www.probabilitycourse.com/chapter7/7_1_2_central_limit_theorem.php) - [7\.1.3 Solved Problems](https://www.probabilitycourse.com/chapter7/7_1_3_solved_probs.php) - [7\.2 Convergence of Random Variables]() - [7\.2.0 Convergence of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_0_convergence_of_random_variables.php) - [7\.2.1 Convergence of Sequence of Numbers](https://www.probabilitycourse.com/chapter7/7_2_1_convergence_of_a_seq_of_nums.php) - [7\.2.2 Sequence of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_2_sequence_of_random_variables.php) - [7\.2.3 Different Types of Convergence for Sequences of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_3_different_types_of_convergence_for_sequences_of_random_variables.php) - [7\.2.4 Convergence in Distribution](https://www.probabilitycourse.com/chapter7/7_2_4_convergence_in_distribution.php) - [7\.2.5 Convergence in Probability](https://www.probabilitycourse.com/chapter7/7_2_5_convergence_in_probability.php) - [7\.2.6 Convergence in Mean](https://www.probabilitycourse.com/chapter7/7_2_6_convergence_in_mean.php) - [7\.2.7 Almost Sure Convergence](https://www.probabilitycourse.com/chapter7/7_2_7_almost_sure_convergence.php) - [7\.2.8 Solved Problems](https://www.probabilitycourse.com/chapter7/7_2_8_solved_probs.php) - [7\.3 Problems]() - [7\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter7/7_3_0_chapter_problem.php) - [8 Statistical Inference I: Classical Methods]() - [8\.1 Introduction]() - [8\.1.0 Introduction](https://www.probabilitycourse.com/chapter8/8_1_0_intro.php) - [8\.1.1 Random Sampling](https://www.probabilitycourse.com/chapter8/8_1_1_random_sampling.php) - [8\.2 Point Estimation]() - [8\.2.0 Point Estimation](https://www.probabilitycourse.com/chapter8/8_2_0_point_estimation.php) - [8\.2.1 Evaluating Estimators](https://www.probabilitycourse.com/chapter8/8_2_1_evaluating_estimators.php) - [8\.2.2 Point Estimators for Mean and Variance](https://www.probabilitycourse.com/chapter8/8_2_2_point_estimators_for_mean_and_var.php) - [8\.2.3 Maximum Likelihood Estimation (MLE)](https://www.probabilitycourse.com/chapter8/8_2_3_max_likelihood_estimation.php) - [8\.2.4 Asymptotic Properties of MLEs](https://www.probabilitycourse.com/chapter8/8_2_4_asymptotic_probs_of_MLE.php) - [8\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter8/8_2_5_solved_probs.php) - [8\.3 Interval Estimation (Confidence Intervals)]() - [8\.3.0 Interval Estimation (Confidence Intervals)](https://www.probabilitycourse.com/chapter8/8_3_0_interval_estimation.php) - [8\.3.1 The general framework of Interval Estimation](https://www.probabilitycourse.com/chapter8/8_3_1_gen_framework_of_int_estimation.php) - [8\.3.2 Finding Interval Estimators](https://www.probabilitycourse.com/chapter8/8_3_2_finding_interval_estimators.php) - [8\.3.3 Confidence Intervals for Normal Samples](https://www.probabilitycourse.com/chapter8/8_3_3_confidence_intervals_for_norm_samples.php) - [8\.3.4 Solved Problems](https://www.probabilitycourse.com/chapter8/8_3_4_solved_probs.php) - [8\.4 Hypothesis Testing]() - [8\.4.1 Introduction](https://www.probabilitycourse.com/chapter8/8_4_1_intro.php) - [8\.4.2 General Setting and Definitions](https://www.probabilitycourse.com/chapter8/8_4_2_general_setting_definitions.php) - [8\.4.3 Hypothesis Testing for the Mean](https://www.probabilitycourse.com/chapter8/8_4_3_hypothesis_testing_for_mean.php) - [8\.4.4 P-Values](https://www.probabilitycourse.com/chapter8/8_4_4_p_vals.php) - [8\.4.5 Likelihood Ratio Tests](https://www.probabilitycourse.com/chapter8/8_4_5_likelihood_ratio_tests.php) - [8\.4.6 Solved Problems](https://www.probabilitycourse.com/chapter8/8_4_6_solved_probs.php) - [8\.5 Linear Regression]() - [8\.5.0 Linear Regression](https://www.probabilitycourse.com/chapter8/8_5_0_linear_regression.php) - [8\.5.1 Simple Linear Regression Model](https://www.probabilitycourse.com/chapter8/8_5_1_simple_linear_regression_model.php) - [8\.5.2 The First Method for Finding beta](https://www.probabilitycourse.com/chapter8/8_5_2_first_method_for_finding_beta.php) - [8\.5.3 The Method of Least Squares](https://www.probabilitycourse.com/chapter8/8_5_3_the_method_of_least_squares.php) - [8\.5.4 Extensions and Issues](https://www.probabilitycourse.com/chapter8/8_5_4_extensions_and_issues.php) - [8\.5.5 Solved Problems](https://www.probabilitycourse.com/chapter8/8_5_5_solved_probs.php) - [8\.6 Problems]() - [8\.6.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter8/8_6_0_ch_probs.php) - [9 Statistical Inference II: Bayesian Inference]() - [9\.1 Bayesian Inference]() - [9\.1.0 Bayesian Inference](https://www.probabilitycourse.com/chapter9/9_1_0_bayesian_inference.php) - [9\.1.1 Prior and Posterior](https://www.probabilitycourse.com/chapter9/9_1_1_prior_and_posterior.php) - [9\.1.2 Maximum A Posteriori (MAP) Estimation](https://www.probabilitycourse.com/chapter9/9_1_2_MAP_estimation.php) - [9\.1.3 Comparison to ML Estimation](https://www.probabilitycourse.com/chapter9/9_1_3_comparison_to_ML_estimation.php) - [9\.1.4 Conditional Expectation (MMSE)](https://www.probabilitycourse.com/chapter9/9_1_4_conditional_expectation_MMSE.php) - [9\.1.5 Mean Squared Error (MSE)](https://www.probabilitycourse.com/chapter9/9_1_5_mean_squared_error_MSE.php) - [9\.1.6 Linear MMSE Estimation of Random Variables](https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_estimat_of_random_vars.php) - [9\.1.7 Estimation for Random Vectors](https://www.probabilitycourse.com/chapter9/9_1_7_estimation_for_random_vectors.php) - [9\.1.8 Bayesian Hypothesis Testing](https://www.probabilitycourse.com/chapter9/9_1_8_bayesian_hypothesis_testing.php) - [9\.1.9 Bayesian Interval Estimation](https://www.probabilitycourse.com/chapter9/9_1_9_bayesian_interval_estimation.php) - [9\.1.10 Solved Problems](https://www.probabilitycourse.com/chapter9/9_1_10_solved_probs.php) - [9\.2 Problems]() - [9\.2.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter9/9_2_0_ch_probs.php) - [10 Introduction to Random Processes]() - [10\.1 Basic Concepts]() - [10\.1.0 Basic Concepts](https://www.probabilitycourse.com/chapter10/10_1_0_basic_concepts.php) - [10\.1.1 PDFs and CDFs](https://www.probabilitycourse.com/chapter10/10_1_1_PDFs_and_CDFs.php) - [10\.1.2 Mean and Correlation Functions](https://www.probabilitycourse.com/chapter10/10_1_2_mean_and_correlation_functions.php) - [10\.1.3 Multiple Random Processes](https://www.probabilitycourse.com/chapter10/10_1_3_multiple_random_processes.php) - [10\.1.4 Stationary Processes](https://www.probabilitycourse.com/chapter10/10_1_4_stationary_processes.php) - [10\.1.5 Gaussian Random Processes](https://www.probabilitycourse.com/chapter10/10_1_5_gaussian_random_processes.php) - [10\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter10/10_1_6_solved_probs.php) - [10\.2 Processing of Random Signals]() - [10\.2.0 Processing of Random Signals](https://www.probabilitycourse.com/chapter10/10_2_0_processing_of_random_signals.php) - [10\.2.1 Power Spectral Density](https://www.probabilitycourse.com/chapter10/10_2_1_power_spectral_density.php) - [10\.2.2 Linear Time-Invariant (LTI) Systems with Random Inputs](https://www.probabilitycourse.com/chapter10/10_2_2_LTI_systems_with_random_inputs.php) - [10\.2.3 Power in a Frequency Band](https://www.probabilitycourse.com/chapter10/10_2_3_power_in_a_frequency_band.php) - [10\.2.4 White Noise](https://www.probabilitycourse.com/chapter10/10_2_4_white_noise.php) - [10\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter10/10_2_5_solved_probs.php) - [10\.3 Problems]() - [10\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter10/10_3_0_ch_probs.php) - [11 Some Important Random Processes]() - [11\.1 Poisson Processes]() - [11\.1.0 Introduction](https://www.probabilitycourse.com/chapter11/11_0_0_intro.php) - [11\.1.1 Counting Processes](https://www.probabilitycourse.com/chapter11/11_1_1_counting_processes.php) - [11\.1.2 Basic Concepts of the Poisson Process](https://www.probabilitycourse.com/chapter11/11_1_2_basic_concepts_of_the_poisson_process.php) - [11\.1.3 Merging and Splitting Poisson Processes](https://www.probabilitycourse.com/chapter11/11_1_3_merging_and_splitting_poisson_processes.php) - [11\.1.4 Nonhomogeneous Poisson Processes](https://www.probabilitycourse.com/chapter11/11_1_4_nonhomogeneous_poisson_processes.php) - [11\.1.5 Solved Problems](https://www.probabilitycourse.com/chapter11/11_1_5_solved_probs.php) - [11\.2 Discrete-Time Markov Chains]() - [11\.2.1 Introduction](https://www.probabilitycourse.com/chapter11/11_2_1_introduction.php) - [11\.2.2 State Transition Matrix and Diagram](https://www.probabilitycourse.com/chapter11/11_2_2_state_transition_matrix_and_diagram.php) - [11\.2.3 Probability Distributions](https://www.probabilitycourse.com/chapter11/11_2_3_probability_distributions.php) - [11\.2.4 Classification of States](https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php) - [11\.2.5 Using the Law of Total Probability with Recursion](https://www.probabilitycourse.com/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) - [11\.2.6 Stationary and Limiting Distributions](https://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php) - [11\.2.7 Solved Problems](https://www.probabilitycourse.com/chapter11/11_2_7_solved_probs.php) - [11\.3 Continuous-Time Markov Chains]() - [11\.3.1 Introduction](https://www.probabilitycourse.com/chapter11/11_3_1_introduction.php) - [11\.3.2 Stationary and Limiting Distributions](https://www.probabilitycourse.com/chapter11/11_3_2_stationary_and_limiting_distributions.php) - [11\.3.3 The Generator Matrix](https://www.probabilitycourse.com/chapter11/11_3_3_the_generator_matrix.php) - [11\.3.4 Solved Problems](https://www.probabilitycourse.com/chapter11/11_3_4_solved_probs.php) - [11\.4 Brownian Motion (Wiener Process)]() - [11\.4.0 Brownian Motion (Wiener Process)](https://www.probabilitycourse.com/chapter11/11_4_0_brownian_motion_wiener_process.php) - [11\.4.1 Brownian Motion as the Limit of a Symmetric Random Walk](https://www.probabilitycourse.com/chapter11/11_4_1_brownian_motion_as_the_limit_of_a_symmetric_random_walk.php) - [1\.4.2 Definition and Some Properties](https://www.probabilitycourse.com/chapter11/11_4_2_definition_and_some_properties.php) - [11\.4.3 Solved Problems](https://www.probabilitycourse.com/chapter11/11_4_3_solved_probs.php) - [11\.5 Problems]() - [11\.5.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter11/11_5_0_end_of_chapter_problems.php) - [12 Introduction to Simulation Using MATLAB](https://www.probabilitycourse.com/chapter12/chapter12.php) - [13 Introduction to Simulation Using R](https://www.probabilitycourse.com/chapter13/chapter13.php) - [14 Introduction to Simulation Using Python](https://www.probabilitycourse.com/chapter14/chapter14.php) - [15 Recursive Methods](https://www.probabilitycourse.com/chapter15/chapter15.php) - [Appendix]() - [Some Important Distributions](https://www.probabilitycourse.com/appendix/some_important_distributions.php) - [Review of the Fourier Transform](https://www.probabilitycourse.com/appendix/review_fourier_transform.php) - [Bibliography](https://www.probabilitycourse.com/bibliography.php) [![Creative Commons License](https://i.creativecommons.org/l/by-nc-nd/3.0/88x31.png)](https://creativecommons.org/licenses/by-nc-nd/3.0/deed.en_US) Introduction to Probability by [Hossein Pishro-Nik](https://websites.umass.edu/pishro/) is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License](https://creativecommons.org/licenses/by-nc-nd/3.0/deed.en_US)
Readable Markdown
*** Here, we would like to discuss long-term behavior of Markov chains. In particular, we would like to know the fraction of times that the Markov chain spends in each state as n becomes large. More specifically, we would like to study the distributions π ( n ) \= \[ P ( X n \= 0 ) P ( X n \= 1 ) ⋯ \] as n → ∞. To better understand the subject, we will first look at an example and then provide a general analysis. *** Example Consider a Markov chain with two possible states, S \= { 0 , 1 }. In particular, suppose that the transition matrix is given by P \= \[ 1 − a a b 1 − b \] , where a and b are two real numbers in the interval \[ 0 , 1 \] such that 0 \< a \+ b \< 2. Suppose that the system is in state 0 at time n \= 0 with probability α, i.e., π ( 0 ) \= \[ P ( X 0 \= 0 ) P ( X 0 \= 1 ) \] \= \[ α 1 − α \] , where α ∈ \[ 0 , 1 \]. 1. Using induction (or any other method), show that P n \= 1 a \+ b \[ b a b a \] \+ ( 1 − a − b ) n a \+ b \[ a − a − b b \] . 2. Show that lim n → ∞ P n \= 1 a \+ b \[ b a b a \] . 3. Show that lim n → ∞ π ( n ) \= \[ b a \+ b a a \+ b \] . - [**Solution**]() - 1. For n \= 1 , we have P 1 \= \[ 1 − a a b 1 − b \] \= 1 a \+ b \[ b a b a \] \+ 1 − a − b a \+ b \[ a − a − b b \] . Assuming that the statement of the problem is true for n , we can write P n \+ 1 as P n \+ 1 \= P n P \= 1 a \+ b ( \[ b a b a \] \+ ( 1 − a − b ) n \[ a − a − b b \] ) ⋅ \[ 1 − a a b 1 − b \] \= 1 a \+ b \[ b a b a \] \+ ( 1 − a − b ) n \+ 1 a \+ b \[ a − a − b b \] , which completes the proof. 2. By assumption 0 \< a \+ b \< 2 , which implies − 1 \< 1 − a − b \< 1 . Thus, lim n → ∞ ( 1 − a − b ) n \= 0\. Therefore, lim n → ∞ P n \= 1 a \+ b \[ b a b a \] . 3. We have lim n → ∞ π ( n ) \= lim n → ∞ \[ π ( 0 ) P n \] \= π ( 0 ) lim n → ∞ P n \= \[ α 1 − α \] ⋅ 1 a \+ b \[ b a b a \] \= \[ b a \+ b a a \+ b \] . *** In the above example, the vector lim n → ∞ π ( n ) \= \[ b a \+ b a a \+ b \] is called the *limiting distribution* of the Markov chain. Note that the limiting distribution does not depend on the initial probabilities α and 1 − α. In other words, the initial state (X 0) does not matter as n becomes large. Thus, for i \= 1 , 2, we can write lim n → ∞ P ( X n \= 0 \| X 0 \= i ) \= b a \+ b , lim n → ∞ P ( X n \= 1 \| X 0 \= i ) \= a a \+ b . Remember that we show P ( X n \= j \| X 0 \= i ) by P i j ( n ), which is the entry in the ith row and jth column of P n. Limiting Distributions The probability distribution π \= \[ π 0 , π 1 , π 2 , ⋯ \] is called the **limiting distribution** of the Markov chain X n if π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) for all i , j ∈ S, and we have ∑ j ∈ S π j \= 1\. By the above definition, when a limiting distribution exists, it does not depend on the initial state (X 0 \= i), so we can write π j \= lim n → ∞ P ( X n \= j ) , for all j ∈ S . So far we have shown that the Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12) has the following limiting distribution: π \= \[ π 0 π 1 \] \= \[ b a \+ b a a \+ b \] . Let's now look at mean return times for this Markov chain. *** Example Consider a Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12): a Markov chain with two possible states, S \= { 0 , 1 }, and the transition matrix P \= \[ 1 − a a b 1 − b \] , where a and b are two real numbers in the interval \[ 0 , 1 \] such that 0 \< a \+ b \< 2. Find the mean return times, r 0 and r 1, for this Markov chain. - [**Solution**]() - We can use the method of the law of total probability that we explained before to find the mean return times ([Example 11.11](https://www.probabilitycourse.com/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php#example11_11)). We can also find r 0 and r 1 directly as follows: Let R be the first return time to state 0, i.e., r 0 \= E \[ R \| X 0 \= 0 \]. If X 0 \= 0, then X 1 \= 0 with probability 1 − a, and X 1 \= 1 with probability a. Thus, using the law of total probability, and assuming X 0 \= 0, we can write r 0 \= E \[ R \| X 1 \= 0 , X 0 \= 0 \] P ( X 1 \= 0 \| X 0 \= 0 ) \+ E \[ R \| X 1 \= 1 , X 0 \= 0 \] P ( X 1 \= 1 \| X 0 \= 0 ) \= E \[ R \| X 1 \= 0 \] ⋅ ( 1 − a ) \+ E \[ R \| X 1 \= 1 \] ⋅ a . If X 1 \= 0 , then R \= 1 , so E \[ R \| X 1 \= 0 \] \= 1\. If X 1 \= 1 , then R ∼ 1 \+ G e o m e t r i c ( b ) , so E \[ R \| X 1 \= 1 \] \= 1 \+ E \[ G e o m e t r i c ( b ) \] \= 1 \+ 1 b . We conclude r 0 \= E \[ R \| X 1 \= 0 \] P ( X 1 \= 0 \| X 0 \= 0 ) \+ E \[ R \| X 1 \= 1 \] P ( X 1 \= 1 \| X 0 \= 0 ) \= 1 ⋅ ( 1 − a ) \+ ( 1 \+ 1 b ) ⋅ a \= a \+ b b . Similarly, we can obtain the mean return time to state 1 : r 1 \= a \+ b a . We notice that for this example, the mean return times are given by the inverse of the limiting probabilities. In particular, we have r 0 \= 1 π 0 , r 1 \= 1 π 1 . As we will see shortly, this is not a coincidence. In fact, we can explain this intuitively. The larger the π i is, the smaller the r i will be. For example, if π i \= 1 4 , we conclude that the chain is in state i one-fourth of the time. In this case, r i \= 4 , which means that on average it takes the chain four time units to go back to state i . *** The two-state Markov chain discussed above is a "nice" one in the sense that it has a well-defined limiting behavior that does not depend on the initial probability distribution (PMF of X 0). However, not all Markov chains are like that. For example, consider the same Markov chain; however, choose a \= b \= 1. In this case, the chain has a periodic behavior, i.e., X n \+ 2 \= X n , for all n . In particular, X n \= { X 0 if n is even X 1 if n is odd In this case, the distribution of X n does not converge to a single PMF. Also, the distribution of X n depends on the initial distribution. As another example, if we choose a \= b \= 0, the chain will consist of two disconnected nodes. In this case, X n \= X 0 , for all n . Here again, the PMF of X n depends on the initial distribution. Now, the question that arises here is: when does a Markov chain have a limiting distribution (that does not depend on the initial PMF)? We will next discuss this question. We will first consider finite Markov chains and then discuss infinite Markov chains. ### Finite Markov Chains: Here, we consider Markov chains with a finite number of states. In general, a finite Markov chain can consist of several transient as well as recurrent states. As n becomes large the chain will enter a recurrent class and it will stay there forever. Therefore, when studying long-run behaviors we focus only on the recurrent classes. If a finite Markov chain has more than one recurrent class, then the chain will get absorbed in one of the recurrent classes. Thus, the first question is: in which recurrent class does the chain get absorbed? We have already seen how to address this when we discussed absorption probabilities (see [Section 11.2.5](https://www.probabilitycourse.com/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php), and [Problem 2](https://www.probabilitycourse.com/chapter11/11_2_7_solved_probs.php#problem11_2) of in [Section 11.2.7](https://www.probabilitycourse.com/chapter11/11_2_7_solved_probs.php)). Thus, we can limit our attention to the case where our Markov chain consists of one recurrent class. In other words, we have an irreducible Markov chain. Note that as we showed in [Example 11.7](https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php#example11_7), in any finite Markov chain, there is at least one recurrent class. Therefore, in finite irreducible chains, all states are recurrent. It turns out that in this case the Markov chain has a well-defined limiting behavior if it is aperiodic (states have period 1). How do we find the limiting distribution? The trick is to find a *stationary distribution*. Here is the idea: If π \= \[ π 1 , π 2 , ⋯ \] is a limiting distribution for a Markov chain, then we have π \= lim n → ∞ π ( n ) \= lim n → ∞ \[ π ( 0 ) P n \] . Similarly, we can write π \= lim n → ∞ π ( n \+ 1 ) \= lim n → ∞ \[ π ( 0 ) P n \+ 1 \] \= lim n → ∞ \[ π ( 0 ) P n P \] \= \[ lim n → ∞ π ( 0 ) P n \] P \= π P . We can explain the equation π \= π P intuitively: Suppose that X n has distribution π. As we saw before, π P gives the probability distribution of X n \+ 1. If we have π \= π P, we conclude that X n and X n \+ 1 have the same distribution. In other words, the chain has reached its *steady-state* (limiting) distribution. We can equivalently write π \= π P as π j \= ∑ k ∈ S π k P k j , for all j ∈ S . The righthand side gives the probability of going to state j in the next step. When we equate both sides, we are implying that the probability of being in state j in the next step is the same as the probability of being in state j now. *** Example Consider a Markov chain in [Example 11.12](https://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php#example11_12): a Markov chain with two possible states, S \= { 0 , 1 }, and the transition matrix P \= \[ 1 − a a b 1 − b \] , where a and b are two real numbers in the interval \[ 0 , 1 \] such that 0 \< a \+ b \< 2. Using π \= π P , find the limiting distribution of this Markov chain. - [**Solution**]() - Let π \= \[ π 0 , π 1 \] . Then, we can write \[ π 0 , π 1 \] \= π P \= \[ π 0 , π 1 \] \[ 1 − a a b 1 − b \] \= \[ π 0 ( 1 − a ) \+ π 1 b π 0 a \+ π 1 ( 1 − b ) \] . We obtain two equations; however, they both simplify to π 0 a \= π 1 b . We remember that π must be a valid probability distribution, i.e., π 0 \+ π 1 \= 1 . Thus, we can obtain a unique solution, i.e., π \= \[ π 0 π 1 \] \= \[ b a \+ b a a \+ b \] which is the same answer that we obtained previously. *** We now summarize the above discussion in the following theorem. Consider a finite Markov chain { X n , n \= 0 , 1 , 2 , . . . } where X n ∈ S \= { 0 , 1 , 2 , ⋯ , r }. Assume that the chain is irreducible and aperiodic . Then, 1. The set of equations π \= π P , ∑ j ∈ S π j \= 1 has a unique solution. 2. The unique solution to the above equations is the limiting distribution of the Markov chain, i.e., π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) , for all i , j ∈ S . 3. We have r j \= 1 π j , for all j ∈ S , where r j is the mean return time to state j . In practice, if we are given a finite irreducible Markov chain with states 0 , 1 , 2 , ⋯ , r, we first find a stationary distribution. That is, we find a probability distribution π that satisfies π j \= ∑ k ∈ S π k P k j , for all j ∈ S , ∑ j ∈ S π j \= 1\. In this case, if the chain is also aperiodic, we conclude that the stationary distribution is a limiting distribution. *** Example Consider the Markov chain shown in Figure 11.14. ![MC-diagram-tek](https://www.probabilitycourse.com/images/chapter11/MC-diagram-tek.png) Figure 11.14- A state transition diagram. 1. Is this chain irreducible? 2. Is this chain aperiodic? 3. Find the stationary distribution for this chain. 4. Is the stationary distribution a limiting distribution for the chain? - [**Solution**]() - 1. The chain is irreducible since we can go from any state to any other states in a finite number of steps. 2. Since there is a self-transition, i.e., p 11 \> 0 , we conclude that the chain is aperiodic. 3. To find the stationary distribution, we need to solve π 1 \= 1 4 π 1 \+ 1 3 π 2 \+ 1 2 π 3 , π 2 \= 1 2 π 1 , π 3 \= 1 4 π 1 \+ 2 3 π 2 \+ 1 2 π 3 , π 1 \+ π 2 \+ π 3 \= 1\. We find π 1 \= 3 8 , π 2 \= 3 16 , π 3 \= 7 16 . 4. Since the chain is irreducible and aperiodic, we conclude that the above stationary distribution is a limiting distribution. *** ### Countably Infinite Markov Chains: When a Markov chain has an infinite (but countable) number of states, we need to distinguish between two types of recurrent states: *positive* recurrent and *null* recurrent states. Remember that if state i is recurrent, then that state will be visited an infinite number of times (any time that we visit that state, we will return to it with probability one in the future). We previously defined r i as the expected number of transitions between visits to state i. Consider a recurrent state i. If r i \< ∞, then state i is a *positive* recurrent state. Otherwise, it is called *null* recurrent. Let i be a recurrent state. Assuming X 0 \= i, let R i be the number of transitions needed to return to state i, i.e., R i \= min { n ≥ 1 : X n \= i } . If r i \= E \[ R i \| X 0 \= i \] \< ∞, then i is said to be **positive recurrent**. If E \[ R i \| X 0 \= i \] \= ∞, then i is said to be **null recurrent**. Theorem Consider an infinite Markov chain { X n , n \= 0 , 1 , 2 , . . . } where X n ∈ S \= { 0 , 1 , 2 , ⋯ }. Assume that the chain is irreducible and aperiodic . Then, one of the following cases can occur: 1. All states are transient , and lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . 2. All states are null recurrent , and lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . 3. All states are positive recurrent . In this case, there exists a limiting distribution, π \= \[ π 0 , π 1 , ⋯ \] , where π j \= lim n → ∞ P ( X n \= j \| X 0 \= i ) \> 0 , for all i , j ∈ S . The limiting distribution is the unique solution to the equations π j \= ∑ k \= 0 ∞ π k P k j , for j \= 0 , 1 , 2 , ⋯ , ∑ j \= 0 ∞ π j \= 1\. We also have r j \= 1 π j , for all j \= 0 , 1 , 2 , ⋯ , where r j is the mean return time to state j . How do we use the above theorem? Consider an infinite Markov chain { X n , n \= 0 , 1 , 2 , . . . }, where X n ∈ S \= { 0 , 1 , 2 , ⋯ }. Assume that the chain is irreducible and aperiodic. We first try to find a stationary distribution π by solving the equations π j \= ∑ k \= 0 ∞ π k P k j , for j \= 0 , 1 , 2 , ⋯ , ∑ j \= 0 ∞ π j \= 1\. If the above equations have a unique solution, we conclude that the chain is positive recurrent and the stationary distribution is the limiting distribution of this chain. On the other hand, if no stationary solution exists, we conclude that the chain is either transient or null recurrent, so lim n → ∞ P ( X n \= j \| X 0 \= i ) \= 0 , for all i , j . *** Example Consider the Markov chain shown in Figure 11.15. Assume that 0 \< p \< 1 2. Does this chain have a limiting distribution? ![MC-diagram-inf-1](https://www.probabilitycourse.com/images/chapter11/MC-diagram-inf-1.png) Figure 11.15 - A state transition diagram. - [**Solution**]() - This chain is irreducible since all states communicate with each other. It is also aperiodic since it includes a self-transition, P 00 \> 0 . Let's write the equations for a stationary distribution. For state 0 , we can write π 0 \= ( 1 − p ) π 0 \+ ( 1 − p ) π 1 , which results in π 1 \= p 1 − p π 0 . For state 1 , we can write π 1 \= p π 0 \+ ( 1 − p ) π 2 \= ( 1 − p ) π 1 \+ ( 1 − p ) π 2 , which results in π 2 \= p 1 − p π 1 . Similarly, for any j ∈ { 1 , 2 , ⋯ } , we obtain π j \= α π j − 1 , where α \= p 1 − p . Note that since 0 \< p \< 1 2 , we conclude that 0 \< α \< 1 . We obtain π j \= α j π 0 , for j \= 1 , 2 , ⋯ . Finally, we must have 1 \= ∑ j \= 0 ∞ π j \= ∑ j \= 0 ∞ α j π 0 , ( where 0 \< α \< 1 ) \= 1 1 − α π 0 ( geometric series ) . Thus, π 0 \= 1 − α . Therefore, the stationary distribution is given by π j \= ( 1 − α ) α j , for j \= 0 , 1 , 2 , ⋯ . Since this chain is irreducible and aperiodic and we have found a stationary distribution, we conclude that all states are positive recurrent and π \= \[ π 0 , π 1 , ⋯ \] is the limiting distribution. *** *** | | |---| | The print version of the book is available on [Amazon](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https://www.probabilitycourse.com/images/home/book_cover.jpg)](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) | | **Practical uncertainty:** *Useful Ideas in Decision-Making, Risk, Randomness, & AI* [![ractical Uncertaintly Cover](https://www.probabilitycourse.com/images/home/practical_uncertaintly.PNG)](https://www.amazon.com/dp/B0CH2BHRVH/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) |
Shard115 (laksa)
Root Hash8531609719846996715
Unparsed URLcom,probabilitycourse!www,/chapter11/11_2_6_stationary_and_limiting_distributions.php s443