🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 115 (from laksa017)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa115.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.probabilitycourse.com\/chapter11\/11_2_4_classification_of_states.php","crawl_time":1776098541,"first_indexed_time":1544880125,"http_code":200,"src_unparsed":"com,probabilitycourse!www,\/chapter11\/11_2_4_classification_of_states.php s443","src_root_hash":"8531609719846996715","history_drop_reason":null,"meta_title":"Classification of States","meta_descriptions":[""],"attrs_boilerpipe_text":"To better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state\ni\nto state\nj\n, we say that state\nj\nis\naccessible\nfrom state\ni\n. In particular, we can provide the following definitions.\nWe say that state\nj\nis\naccessible\nfrom state\ni\n, written as\ni\n→\nj\n, if\np\ni\nj\n(\nn\n)\n>\n0\nfor some\nn\n. We assume every state is accessible from itself since\np\ni\ni\n(\n0\n)\n=\n1\n.\nTwo states\ni\nand\nj\nare said to\ncommunicate\n, written as\ni\n↔\nj\n, if they are\naccessible\nfrom each other. In other words,\ni\n↔\nj\n means \ni\n→\nj\n and \nj\n→\ni\n.\nCommunication is an\nequivalence\nrelation. That means that\n−\nevery state communicates with itself,\ni\n↔\ni\n;\n−\nif\ni\n↔\nj\n, then\nj\n↔\ni\n;\n−\nif\ni\n↔\nj\nand\nj\n↔\nk\n, then\ni\n↔\nk\n.\nTherefore, the states of a Markov chain can be partitioned into communicating\nclasses\nsuch that only members of the same class communicate with each other. That is, two states\ni\nand\nj\nbelong to the same class if and only if\ni\n↔\nj\n.\nExample\nConsider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state\ni\nto state\nj\n, then\np\ni\nj\n>\n0\n. Find the equivalence classes for this Markov chain.\nFigure 11.9 - A state transition diagram.\nSolution\nThere are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states\n1\nand\n2\ncommunicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes\n3\nand\n4\ncommunicate with each other, but they do not communicate with any other nodes in the graph. State\n5\ndoes not communicate with any other states, so it by itself is a class. Finally, states\n6\n,\n7\n, and\n8\nconstruct another class. Thus, here are the classes:\nClass 1\n=\n{\nstate \n1\n,\nstate \n2\n}\n,\nClass 2\n=\n{\nstate \n3\n,\nstate \n4\n}\n,\nClass 3\n=\n{\nstate \n5\n}\n,\nClass 4\n=\n{\nstate \n6\n,\nstate \n7\n,\nstate \n8\n}\n.\nFigure 11.10 - Equivalence classes.\nA Markov chain is said to be\nirreducible\nif it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior.\nA Markov chain is said to be\nirreducible\nif all states communicate with each other.\nLooking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class\n4\n, it will always stay in that class. On the other hand, for other classes this is not true. For example, if\nX\n0\n=\n1\n, then the Markov chain might stay in Class\n1\nfor a while, but at some point, it  will leave that class and it will never return to that class again. The states in Class\n4\nare called\nrecurrent\nstates, while the other states in this chain are called\ntransient\n.\nIn general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition:\nFor any state\ni\n, we define\nf\ni\ni\n=\nP\n(\nX\nn\n=\ni\n,\n for some \nn\n≥\n1\n|\nX\n0\n=\ni\n)\n.\nState\ni\nis\nrecurrent\nif\nf\ni\ni\n=\n1\n, and it is\ntransient\nif\nf\ni\ni\n<\n1\n.\nIt is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes.\nConsider a Markov chain and assume\nX\n0\n=\ni\n. If\ni\nis a recurrent state, then the chain will return to state\ni\nany time it leaves that state. Therefore, the chain will visit state\ni\nan infinite number of times. On the other hand, if\ni\nis a transient state, the chain will return to state\ni\nwith probability\nf\ni\ni\n<\n1\n. Thus, in that case, the total number of visits to state\ni\nwill be a Geometric random variable with parameter\n1\n−\nf\ni\ni\n.\nConsider a discrete-time Markov chain. Let\nV\nbe the total number of visits to state\ni\n.\nIf\ni\nis a recurrent state, then\nP\n(\nV\n=\n∞\n|\nX\n0\n=\ni\n)\n=\n1.\nIf\ni\nis a transient state, then\nV\n|\nX\n0\n=\ni\n∼\nG\ne\no\nm\ne\nt\nr\ni\nc\n(\n1\n−\nf\ni\ni\n)\n.\nExample\nShow that in a finite Markov chain, there is at least one recurrent class.\nSolution\nConsider a finite Markov chain with\nr\nstates,\nS\n=\n{\n1\n,\n2\n,\n⋯\n,\nr\n}\n. Suppose that all states are transient. Then, starting from time\n0\n, the chain might visit state\n1\nseveral times, but at some point the chain will leave state\n1\nand will never return to it. That is, there exists an integer\nM\n1\n>\n0\nsuch that\nX\nn\n≠\n1\n, for all\nn\n≥\nM\n1\n. Similarly, there exists an integer\nM\n2\n>\n0\nsuch that\nX\nn\n≠\n2\n, for all\nn\n≥\nM\n2\n, and so on. Now, if you choose\nn\n≥\nmax\n{\nM\n1\n,\nM\n2\n,\n⋯\n,\nM\nr\n}\n,\nthen\nX\nn\ncannot be equal to any of the states\n1\n,\n2\n,\n⋯\n,\nr\n. This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class.\nPeriodicity:\nConsider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state\n0\n, we only return to\n0\nat times\nn\n=\n3\n,\n6\n,\n⋯\n. In other words,\np\n00\n(\nn\n)\n=\n0\n, if\nn\nis not divisible by\n3\n. Such a state is called a\nperiodic\nstate with period\nd\n(\n0\n)\n=\n3\n.\nFigure 11.11 - A state transition diagram.\nThe\nperiod\nof a state\ni\nis the largest integer\nd\nsatisfying the following property:\np\ni\ni\n(\nn\n)\n=\n0\n, whenever\nn\nis not divisible by\nd\n. The period of\ni\nis shown by\nd\n(\ni\n)\n. If\np\ni\ni\n(\nn\n)\n=\n0\n, for all\nn\n>\n0\n, then we let\nd\n(\ni\n)\n=\n∞\n.\n−\nIf\nd\n(\ni\n)\n>\n1\n, we say that state\ni\nis\nperiodic\n.\n−\nIf\nd\n(\ni\n)\n=\n1\n, we say that state\ni\nis\naperiodic\n.\nYou can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic.\nIf\ni\n↔\nj\n, then\nd\n(\ni\n)\n=\nd\n(\nj\n)\n.\nWhy is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic.\nHow do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers\nm\nand\nl\nare said to be\nco-prime\nif their greatest common divisor (gcd) is\n1\n, i.e.,\ngcd\n(\nl\n,\nm\n)\n=\n1\n. Now, suppose that we can find two co-prime numbers\nl\nand\nm\nsuch that\np\ni\ni\n(\nl\n)\n>\n0\nand\np\ni\ni\n(\nm\n)\n>\n0\n. That is, we can go from state\ni\nto itself in\nl\nsteps, and also in\nm\nsteps. Then, we can conclude state\ni\nis aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number\n1\nis co-prime to every integer, any state with a self-transition is aperiodic.\nConsider a finite\nirreducible\nMarkov chain\nX\nn\n:\nIf there is a self-transition in the chain (\np\ni\ni\n>\n0\nfor some\ni\n), then the chain is aperiodic.\nSuppose that you can go from state\ni\nto state\ni\nin\nl\nsteps, i.e.,\np\ni\ni\n(\nl\n)\n>\n0\n. Also suppose that\np\ni\ni\n(\nm\n)\n>\n0\n. If\ngcd\n(\nl\n,\nm\n)\n=\n1\n, then state\ni\nis aperiodic.\nThe chain is aperiodic if and only if there exists a positive integer\nn\nsuch that all elements of the matrix\nP\nn\nare strictly positive, i.e.,\np\ni\nj\n(\nn\n)\n>\n0\n,\n for all \ni\n,\nj\n∈\nS\n.\nExample\nConsider the Markov chain in\nExample 11.6\n.\nIs\nClass 1\n=\n{\nstate \n1\n,\nstate \n2\n}\naperiodic?\nIs\nClass 2\n=\n{\nstate \n3\n,\nstate \n4\n}\naperiodic?\nIs\nClass 4\n=\n{\nstate \n6\n,\nstate \n7\n,\nstate \n8\n}\naperiodic?\nSolution\nClass 1\n=\n{\nstate \n1\n,\nstate \n2\n}\nis\naperiodic\nsince it has a self-transition,\np\n22\n>\n0\n.\nClass 2\n=\n{\nstate \n3\n,\nstate \n4\n}\nis\nperiodic\nwith period\n2\n.\nClass 4\n=\n{\nstate \n6\n,\nstate \n7\n,\nstate \n8\n}\nis\naperiodic\n. For example, note that we can go from state\n6\nto state\n6\nin two steps (\n6\n−\n7\n−\n6\n) and in three steps (\n6\n−\n7\n−\n8\n−\n6\n). Since\ngcd\n(\n2\n,\n3\n)\n=\n1\n, we conclude state\n6\nand its class are aperiodic.\nThe print version of the book is available on\nAmazon\n.\nPractical uncertainty:\nUseful Ideas in Decision-Making, Risk, Randomness, & AI","attrs_markdown":"- [HOME](https:\/\/www.probabilitycourse.com\/)\n- [VIDEOS](https:\/\/www.probabilitycourse.com\/videos\/videos.php)\n- [CALCULATOR](https:\/\/www.probabilitycourse.com\/calculator\/calculator.php)\n- [COMMENTS](https:\/\/www.probabilitycourse.com\/comments.php)\n- [COURSES](https:\/\/www.probabilitycourse.com\/courses.php)\n- [FOR INSTRUCTOR](https:\/\/www.probabilitycourse.com\/for_instructors.php)\n- [LOG IN](https:\/\/www.probabilitycourse.com\/Login\/mobile_login.php)\n\n[![](https:\/\/www.probabilitycourse.com\/images\/Template\/logo_probability_website.png)](https:\/\/www.probabilitycourse.com\/)\n\nChapters\n\nMenu\n\n- [HOME](https:\/\/www.probabilitycourse.com\/)\n- [VIDEOS](https:\/\/www.probabilitycourse.com\/videos\/videos.php)\n- [CALCULATOR](https:\/\/www.probabilitycourse.com\/calculator\/calculator.php)\n- [COMMENTS](https:\/\/www.probabilitycourse.com\/comments.php)\n- [COURSES](https:\/\/www.probabilitycourse.com\/courses.php)\n- [FOR INSTRUCTORS](https:\/\/www.probabilitycourse.com\/for_instructors.php)\n\n- [Sign In]()\n\n[Forgot password?](https:\/\/www.probabilitycourse.com\/Login\/forgot_password.php)\n\n[←]() [previous](https:\/\/www.probabilitycourse.com\/chapter11\/chapter11\/11_2_3_probability_distributions.php)\n\n[next](https:\/\/www.probabilitycourse.com\/chapter11\/chapter11\/11_2_5_using_the_law_of_total_probability_with_recursion.php) [→]()\n***\n## 11\\.2.4 Classification of States\nTo better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state\n\ni\n\ni\n\nto state\n\nj\n\nj\n\n, we say that state\n\nj\n\nj\n\nis *accessible* from state\n\ni\n\ni\n\n. In particular, we can provide the following definitions.\n\nWe say that state\n\nj\n\nj\n\nis **accessible** from state\n\ni\n\ni\n\n, written as\n\ni→j\n\ni\n\n→\n\nj\n\n, if\n\np(n)ij\\>0\n\np\n\ni\n\nj\n\n(\n\nn\n\n)\n\n\\>\n\n0\n\nfor some\n\nn\n\nn\n\n. We assume every state is accessible from itself since\n\np(0)ii\\=1\n\np\n\ni\n\ni\n\n(\n\n0\n\n)\n\n\\=\n\n1\n\n.\n\nTwo states\n\ni\n\ni\n\nand\n\nj\n\nj\n\nare said to **communicate**, written as\n\ni↔j\n\ni\n\n↔\n\nj\n\n, if they are **accessible** from each other. In other words,\n\ni↔j means i→j and j→i.\n\ni\n\n↔\n\nj\n\nmeans\n\ni\n\n→\n\nj\n\nand\n\nj\n\n→\n\ni\n\n.\n\nCommunication is an *equivalence* relation. That means that\n\n−\n\n−\n\nevery state communicates with itself,\n\ni↔i\n\ni\n\n↔\n\ni\n\n;\n\n−\n\n−\n\nif\n\ni↔j\n\ni\n\n↔\n\nj\n\n, then\n\nj↔i\n\nj\n\n↔\n\ni\n\n;\n\n−\n\n−\n\nif\n\ni↔j\n\ni\n\n↔\n\nj\n\nand\n\nj↔k\n\nj\n\n↔\n\nk\n\n, then\n\ni↔k\n\ni\n\n↔\n\nk\n\n.\n\nTherefore, the states of a Markov chain can be partitioned into communicating *classes* such that only members of the same class communicate with each other. That is, two states\n\ni\n\ni\n\nand\n\nj\n\nj\n\nbelong to the same class if and only if\n\ni↔j\n\ni\n\n↔\n\nj\n\n.\n***\nExample  \n Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state\n\ni\n\ni\n\nto state\n\nj\n\nj\n\n, then\n\npij\\>0\n\np\n\ni\n\nj\n\n\\>\n\n0\n\n. Find the equivalence classes for this Markov chain.\n\n![MC-diagram-3](https:\/\/www.probabilitycourse.com\/chapter11\/images\/chapter11\/MC-diagram-3.png)\n\nFigure 11.9 - A state transition diagram.\n\n- [**Solution**]()\n  - There are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states\n    1\n    \n    1\n    and\n    2\n    \n    2\n    communicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes\n    3\n    \n    3\n    and\n    4\n    \n    4\n    communicate with each other, but they do not communicate with any other nodes in the graph. State\n    5\n    \n    5\n    does not communicate with any other states, so it by itself is a class. Finally, states\n    6\n    \n    6\n    ,\n    7\n    \n    7\n    , and\n    8\n    \n    8\n    construct another class. Thus, here are the classes:\n    Class 1\\={state 1,state 2},Class 2\\={state 3,state 4},Class 3\\={state 5},Class 4\\={state 6,state 7,state 8}.\n    \n    Class 1\n    \n    \\=\n    \n    {\n    \n    state\n    \n    1\n    \n    ,\n    \n    state\n    \n    2\n    \n    }\n    \n    ,\n    \n    Class 2\n    \n    \\=\n    \n    {\n    \n    state\n    \n    3\n    \n    ,\n    \n    state\n    \n    4\n    \n    }\n    \n    ,\n    \n    Class 3\n    \n    \\=\n    \n    {\n    \n    state\n    \n    5\n    \n    }\n    \n    ,\n    \n    Class 4\n    \n    \\=\n    \n    {\n    \n    state\n    \n    6\n    \n    ,\n    \n    state\n    \n    7\n    \n    ,\n    \n    state\n    \n    8\n    \n    }\n    \n    .\n    ![MC-diagram-4](https:\/\/www.probabilitycourse.com\/chapter11\/images\/chapter11\/MC-diagram-4.png)\n    \n    Figure 11.10 - Equivalence classes.\n***\nA Markov chain is said to be *irreducible* if it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior.\n\nA Markov chain is said to be **irreducible** if all states communicate with each other.\n\nLooking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class 4 4, it will always stay in that class. On the other hand, for other classes this is not true. For example, if X0\\=1 X 0 \\= 1, then the Markov chain might stay in Class 1 1 for a while, but at some point, it will leave that class and it will never return to that class again. The states in Class 4 4 are called *recurrent*  states, while the other states in this chain are called *transient*.\n\nIn general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition:\n\nFor any state\n\ni\n\ni\n\n, we define\n\nfii\\=P(Xn\\=i, for some n≥1\\|X0\\=i).\n\nf\n\ni\n\ni\n\n\\=\n\nP\n\n(\n\nX\n\nn\n\n\\=\n\ni\n\n,\n\nfor some\n\nn\n\n≥\n\n1\n\n\\|\n\nX\n\n0\n\n\\=\n\ni\n\n)\n\n.\n\nState\n\ni\n\ni\n\nis **recurrent** if\n\nfii\\=1\n\nf\n\ni\n\ni\n\n\\=\n\n1\n\n, and it is **transient** if\n\nfii\\<1\n\nf\n\ni\n\ni\n\n\\<\n\n1\n\n.\n\nIt is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes.\n\nConsider a Markov chain and assume X0\\=i X 0 \\= i. If i i is a recurrent state, then the chain will return to state i i any time it leaves that state. Therefore, the chain will visit state i i an infinite number of times. On the other hand, if i i is a transient state, the chain will return to state i i with probability fii\\<1 f i i \\< 1. Thus, in that case, the total number of visits to state i i will be a Geometric random variable with parameter 1−fii 1 − f i i.\n\nConsider a discrete-time Markov chain. Let\n\nV\n\nV\n\nbe the total number of visits to state\n\ni\n\ni\n\n.\n\n1. If\n   i\n   \n   i\n   is a recurrent state, then\n   P(V\\=∞\\|X0\\=i)\\=1\\.\n   \n   P\n   \n   (\n   \n   V\n   \n   \\=\n   \n   ∞\n   \n   \\|\n   \n   X\n   \n   0\n   \n   \\=\n   \n   i\n   \n   )\n   \n   \\=\n   \n   1\\.\n2. If\n   i\n   \n   i\n   is a transient state, then\n   V\\|X0\\=i∼Geometric(1−fii).\n   \n   V\n   \n   \\|\n   \n   X\n   \n   0\n   \n   \\=\n   \n   i\n   \n   ∼\n   \n   G\n   \n   e\n   \n   o\n   \n   m\n   \n   e\n   \n   t\n   \n   r\n   \n   i\n   \n   c\n   \n   (\n   \n   1\n   \n   −\n   \n   f\n   \n   i\n   \n   i\n   \n   )\n   \n   .\n***\nExample   \n Show that in a finite Markov chain, there is at least one recurrent class.\n\n- [**Solution**]()\n  - Consider a finite Markov chain with\n    r\n    \n    r\n    states,\n    S\\={1,2,⋯,r}\n    \n    S\n    \n    \\=\n    \n    {\n    \n    1\n    \n    ,\n    \n    2\n    \n    ,\n    \n    ⋯\n    \n    ,\n    \n    r\n    \n    }\n    . Suppose that all states are transient. Then, starting from time\n    0\n    \n    0\n    , the chain might visit state\n    1\n    \n    1\n    several times, but at some point the chain will leave state\n    1\n    \n    1\n    and will never return to it. That is, there exists an integer\n    M1\\>0\n    \n    M\n    \n    1\n    \n    \\>\n    \n    0\n    such that\n    Xn≠1\n    \n    X\n    \n    n\n    \n    ≠\n    \n    1\n    , for all\n    n≥M1\n    \n    n\n    \n    ≥\n    \n    M\n    \n    1\n    . Similarly, there exists an integer\n    M2\\>0\n    \n    M\n    \n    2\n    \n    \\>\n    \n    0\n    such that\n    Xn≠2\n    \n    X\n    \n    n\n    \n    ≠\n    \n    2\n    , for all\n    n≥M2\n    \n    n\n    \n    ≥\n    \n    M\n    \n    2\n    , and so on. Now, if you choose\n    n≥max{M1,M2,⋯,Mr},\n    \n    n\n    \n    ≥\n    \n    max\n    \n    {\n    \n    M\n    \n    1\n    \n    ,\n    \n    M\n    \n    2\n    \n    ,\n    \n    ⋯\n    \n    ,\n    \n    M\n    \n    r\n    \n    }\n    \n    ,\n    then\n    Xn\n    \n    X\n    \n    n\n    cannot be equal to any of the states\n    1,2,⋯,r\n    \n    1\n    \n    ,\n    \n    2\n    \n    ,\n    \n    ⋯\n    \n    ,\n    \n    r\n    . This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class.\n***\n### Periodicity:\nConsider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state\n\n0\n\n0\n\n, we only return to\n\n0\n\n0\n\nat times\n\nn\\=3,6,⋯\n\nn\n\n\\=\n\n3\n\n,\n\n6\n\n,\n\n⋯\n\n. In other words,\n\np(n)00\\=0\n\np\n\n00\n\n(\n\nn\n\n)\n\n\\=\n\n0\n\n, if\n\nn\n\nn\n\nis not divisible by\n\n3\n\n3\n\n. Such a state is called a *periodic* state with period\n\nd(0)\\=3\n\nd\n\n(\n\n0\n\n)\n\n\\=\n\n3\n\n.\n\n![MC-diagram-periodic](https:\/\/www.probabilitycourse.com\/chapter11\/images\/chapter11\/MC-diagram-periodic.png)\n\nFigure 11.11 - A state transition diagram.\n\nThe **period** of a state\n\ni\n\ni\n\nis the largest integer\n\nd\n\nd\n\nsatisfying the following property:\n\np(n)ii\\=0\n\np\n\ni\n\ni\n\n(\n\nn\n\n)\n\n\\=\n\n0\n\n, whenever\n\nn\n\nn\n\nis not divisible by\n\nd\n\nd\n\n. The period of\n\ni\n\ni\n\nis shown by\n\nd(i)\n\nd\n\n(\n\ni\n\n)\n\n. If\n\np(n)ii\\=0\n\np\n\ni\n\ni\n\n(\n\nn\n\n)\n\n\\=\n\n0\n\n, for all\n\nn\\>0\n\nn\n\n\\>\n\n0\n\n, then we let\n\nd(i)\\=∞\n\nd\n\n(\n\ni\n\n)\n\n\\=\n\n∞\n\n.\n\n−\n\n−\n\nIf\n\nd(i)\\>1\n\nd\n\n(\n\ni\n\n)\n\n\\>\n\n1\n\n, we say that state\n\ni\n\ni\n\nis **periodic**.\n\n−\n\n−\n\nIf\n\nd(i)\\=1\n\nd\n\n(\n\ni\n\n)\n\n\\=\n\n1\n\n, we say that state\n\ni\n\ni\n\nis **aperiodic**.\n\nYou can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic.\n\nIf\n\ni↔j\n\ni\n\n↔\n\nj\n\n, then\n\nd(i)\\=d(j)\n\nd\n\n(\n\ni\n\n)\n\n\\=\n\nd\n\n(\n\nj\n\n)\n\n.\n\nWhy is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic.\n\nHow do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers m m and l l are said to be *co-prime* if their greatest common divisor (gcd) is 1 1, i.e., gcd(l,m)\\=1 gcd ( l , m ) \\= 1. Now, suppose that we can find two co-prime numbers l l and m m such that p(l)ii\\>0 p i i ( l ) \\> 0 and p(m)ii\\>0 p i i ( m ) \\> 0. That is, we can go from state i i to itself in l l steps, and also in m m steps. Then, we can conclude state i i is aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number 1 1 is co-prime to every integer, any state with a self-transition is aperiodic.\n\nConsider a finite\n\nirreducible\n\nMarkov chain\n\nXn\n\nX\n\nn\n\n:\n\n1. If there is a self-transition in the chain (\n   pii\\>0\n   \n   p\n   \n   i\n   \n   i\n   \n   \\>\n   \n   0\n   for some\n   i\n   \n   i\n   ), then the chain is aperiodic.\n2. Suppose that you can go from state\n   i\n   \n   i\n   to state\n   i\n   \n   i\n   in\n   l\n   \n   l\n   steps, i.e.,\n   p(l)ii\\>0\n   \n   p\n   \n   i\n   \n   i\n   \n   (\n   \n   l\n   \n   )\n   \n   \\>\n   \n   0\n   . Also suppose that\n   p(m)ii\\>0\n   \n   p\n   \n   i\n   \n   i\n   \n   (\n   \n   m\n   \n   )\n   \n   \\>\n   \n   0\n   . If\n   gcd(l,m)\\=1\n   \n   gcd\n   \n   (\n   \n   l\n   \n   ,\n   \n   m\n   \n   )\n   \n   \\=\n   \n   1\n   , then state\n   i\n   \n   i\n   is aperiodic.\n3. The chain is aperiodic if and only if there exists a positive integer\n   n\n   \n   n\n   such that all elements of the matrix\n   Pn\n   \n   P\n   \n   n\n   are strictly positive, i.e.,\n***\nExample   \n Consider the Markov chain in [Example 11.6](https:\/\/www.probabilitycourse.com\/chapter11\/chapter11\/11_2_4_classification_of_states.php#example11_6).\n\n1. Is\n   Class 1\\={state 1,state 2}\n   \n   Class 1\n   \n   \\=\n   \n   {\n   \n   state\n   \n   1\n   \n   ,\n   \n   state\n   \n   2\n   \n   }\n   aperiodic?\n2. Is\n   Class 2\\={state 3,state 4}\n   \n   Class 2\n   \n   \\=\n   \n   {\n   \n   state\n   \n   3\n   \n   ,\n   \n   state\n   \n   4\n   \n   }\n   aperiodic?\n3. Is\n   Class 4\\={state 6,state 7,state 8}\n   \n   Class 4\n   \n   \\=\n   \n   {\n   \n   state\n   \n   6\n   \n   ,\n   \n   state\n   \n   7\n   \n   ,\n   \n   state\n   \n   8\n   \n   }\n   aperiodic?\n\n- [**Solution**]()\n  - 1. Class 1\\={state 1,state 2}\n       \n       Class 1\n       \n       \\=\n       \n       {\n       \n       state\n       \n       1\n       \n       ,\n       \n       state\n       \n       2\n       \n       }\n       is\n       aperiodic\n       since it has a self-transition,\n       p22\\>0\n       \n       p\n       \n       22\n       \n       \\>\n       \n       0\n       .\n    2. Class 2\\={state 3,state 4}\n       \n       Class 2\n       \n       \\=\n       \n       {\n       \n       state\n       \n       3\n       \n       ,\n       \n       state\n       \n       4\n       \n       }\n       is\n       periodic\n       with period\n       2\n       \n       2\n       .\n    3. Class 4\\={state 6,state 7,state 8}\n       \n       Class 4\n       \n       \\=\n       \n       {\n       \n       state\n       \n       6\n       \n       ,\n       \n       state\n       \n       7\n       \n       ,\n       \n       state\n       \n       8\n       \n       }\n       is\n       aperiodic\n       . For example, note that we can go from state\n       6\n       \n       6\n       to state\n       6\n       \n       6\n       in two steps (\n       6−7−6\n       \n       6\n       \n       −\n       \n       7\n       \n       −\n       \n       6\n       ) and in three steps (\n       6−7−8−6\n       \n       6\n       \n       −\n       \n       7\n       \n       −\n       \n       8\n       \n       −\n       \n       6\n       ). Since\n       gcd(2,3)\\=1\n       \n       gcd\n       \n       (\n       \n       2\n       \n       ,\n       \n       3\n       \n       )\n       \n       \\=\n       \n       1\n       , we conclude state\n       6\n       \n       6\n       and its class are aperiodic.\n***\n[←]() [previous](https:\/\/www.probabilitycourse.com\/chapter11\/chapter11\/11_2_3_probability_distributions.php)\n\n[next](https:\/\/www.probabilitycourse.com\/chapter11\/chapter11\/11_2_5_using_the_law_of_total_probability_with_recursion.php) [→]()\n\n***\n|   |\n|---|\n| The print version of the book is available on [Amazon](https:\/\/www.amazon.com\/Introduction-Probability-Statistics-Random-Processes\/dp\/0990637204\/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https:\/\/www.probabilitycourse.com\/images\/home\/book_cover.jpg)](https:\/\/www.amazon.com\/Introduction-Probability-Statistics-Random-Processes\/dp\/0990637204\/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) |\n| **Practical uncertainty:** *Useful Ideas in Decision-Making, Risk, Randomness, & AI* [![ractical Uncertaintly Cover](https:\/\/www.probabilitycourse.com\/images\/home\/practical_uncertaintly.PNG)](https:\/\/www.amazon.com\/dp\/B0CH2BHRVH\/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) |\n\nOpen Menu\n\n- [0 Preface](https:\/\/www.probabilitycourse.com\/preface.php)\n- [1 Basic Concepts]()\n  - [1\\.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter1\/1_0_0_introduction.php)\n  - [1\\.1 Introduction]()\n    - [1\\.1.0 What Is Probability?](https:\/\/www.probabilitycourse.com\/chapter1\/1_1_0_what_is_probability.php)\n    - [1\\.1.1 Example](https:\/\/www.probabilitycourse.com\/chapter1\/1_1_1_example.php)\n  - [1\\.2 Review of Set Theory]()\n    - [1\\.2.0 Review](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_0_review_set_theory.php)\n    - [1\\.2.1 Venn Diagrams](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_1_venn.php)\n    - [1\\.2.2 Set Operations](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_2_set_operations.php)\n    - [1\\.2.3 Cardinality](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_3_cardinality.php)\n    - [1\\.2.4 Functions](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_4_functions.php)\n    - [1\\.2.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter1\/1_2_5_solved1.php)\n  - [1\\.3 Random Experiments and Probabilities]()\n    - [1\\.3.1 Random Experiments](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_1_random_experiments.php)\n    - [1\\.3.2 Probability](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_2_probability.php)\n    - [1\\.3.3 Finding Probabilities](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_3_finding_probabilities.php)\n    - [1\\.3.4 Discrete Models](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_4_discrete_models.php)\n    - [1\\.3.5 Continuous Models](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_5_continuous_models.php)\n    - [1\\.3.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter1\/1_3_6_solved2.php)\n  - [1\\.4 Conditional Probability]()\n    - [1\\.4.0 Conditional Probability](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_0_conditional_probability.php)\n    - [1\\.4.1 Independence](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_1_independence.php)\n    - [1\\.4.2 Law of Total Probability](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_2_total_probability.php)\n    - [1\\.4.3 Bayes' Rule](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_3_bayes_rule.php)\n    - [1\\.4.4 Conditional Independence](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_4_conditional_independence.php)\n    - [1\\.4.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter1\/1_4_5_solved3.php)\n  - [1\\.5 Problems]()\n    - [1\\.5.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter1\/1_5_0_chapter1_problems.php)\n- [2 Combinatorics: Counting Methods]()\n  - [2\\.1 Combinatorics]()\n    - [2\\.1.0 Finding Probabilities with Counting Methods](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_0_counting.php)\n    - [2\\.1.1 Ordered with Replacement](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_1_ordered_with_replacement.php)\n    - [2\\.1.2 Ordered without Replacement](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_2_ordered_without_replacement.php)\n    - [2\\.1.3 Unordered without Replacement](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_3_unordered_without_replacement.php)\n    - [2\\.1.4 Unordered with Replacement](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_4_unordered_with_replacement.php)\n    - [2\\.1.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter2\/2_1_5_solved2_1.php)\n  - [2\\.2 Problems]()\n    - [2\\.2.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter2\/2_3_0_chapter2_problems.php)\n- [3 Discrete Random Variables]()\n  - [3\\.1 Basic Concepts]()\n    - [3\\.1.1 Random Variables](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_1_random_variables.php)\n    - [3\\.1.2 Discrete Random Variables](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_2_discrete_random_var.php)\n    - [3\\.1.3 Probability Mass Function](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_3_pmf.php)\n    - [3\\.1.4 Independent Random Variables](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_4_independent_random_var.php)\n    - [3\\.1.5 Special Distributions](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_5_special_discrete_distr.php)\n    - [3\\.1.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter3\/3_1_6_solved3_1.php)\n  - [3\\.2 More about Discrete Random Variables]()\n    - [3\\.2.1 Cumulative Distribution Function](https:\/\/www.probabilitycourse.com\/chapter3\/3_2_1_cdf.php)\n    - [3\\.2.2 Expectation](https:\/\/www.probabilitycourse.com\/chapter3\/3_2_2_expectation.php)\n    - [3\\.2.3 Functions of Random Variables](https:\/\/www.probabilitycourse.com\/chapter3\/3_2_3_functions_random_var.php)\n    - [3\\.2.4 Variance](https:\/\/www.probabilitycourse.com\/chapter3\/3_2_4_variance.php)\n    - [3\\.2.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter3\/3_2_5_solved3_2.php)\n  - [3\\.3 Problems]()\n    - [3\\.3.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter3\/3_3_0_chapter3_problems.php)\n- [4 Continuous and Mixed Random Variables]()\n  - [4\\.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter4\/4_0_0_intro.php)\n  - [4\\.1 Continuous Random Variables]()\n    - [4\\.1.0 Continuous Random Variables and their Distributions](https:\/\/www.probabilitycourse.com\/chapter4\/4_1_0_continuous_random_vars_distributions.php)\n    - [4\\.1.1 Probability Density Function](https:\/\/www.probabilitycourse.com\/chapter4\/4_1_1_pdf.php)\n    - [4\\.1.2 Expected Value and Variance](https:\/\/www.probabilitycourse.com\/chapter4\/4_1_2_expected_val_variance.php)\n    - [4\\.1.3 Functions of Continuous Random Variables](https:\/\/www.probabilitycourse.com\/chapter4\/4_1_3_functions_continuous_var.php)\n    - [4\\.1.4 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter4\/4_1_4_solved4_1.php)\n  - [4\\.2 Special Distributions]()\n    - [4\\.2.1 Uniform Distribution](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_1_uniform.php)\n    - [4\\.2.2 Exponential Distribution](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_2_exponential.php)\n    - [4\\.2.3 Normal (Gaussian) Distribution](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_3_normal.php)\n    - [4\\.2.4 Gamma Distribution](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_4_Gamma_distribution.php)\n    - [4\\.2.5 Other Distributions](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_5_other_distr.php)\n    - [4\\.2.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter4\/4_2_6_solved4_2.php)\n  - [4\\.3 Mixed Random Variables]()\n    - [4\\.3.1 Mixed Random Variables](https:\/\/www.probabilitycourse.com\/chapter4\/4_3_1_mixed.php)\n    - [4\\.3.2 Using the Delta Function](https:\/\/www.probabilitycourse.com\/chapter4\/4_3_2_delta_function.php)\n    - [4\\.3.3 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter4\/4_3_3_solved4_3.php)\n  - [4\\.4 Problems]()\n    - [4\\.4.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter4\/4_4_0_chapter4_problems.php)\n- [5 Joint Distributions]()\n  - [5\\.1 Two Discrete Random Variables]()\n    - [5\\.1.0 Two Random Variables](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_0_joint_distributions.php)\n    - [5\\.1.1 Joint Probability Mass Function (PMF)](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_1_joint_pmf.php)\n    - [5\\.1.2 Joint Cumulative Distribution Function (CDF)](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_2_joint_cdf.php)\n    - [5\\.1.3 Conditioning and Independence](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_3_conditioning_independence.php)\n    - [5\\.1.4 Functions of Two Random Variables](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_4_functions_two_variables.php)\n    - [5\\.1.5 Conditional Expectation](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_5_conditional_expectation.php)\n    - [5\\.1.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter5\/5_1_6_solved_prob.php)\n  - [5\\.2 Two Continuous Random Variables]()\n    - [5\\.2.0 Two Continuous Random Variables](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_0_continuous_vars.php)\n    - [5\\.2.1 Joint Probability Density Function](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_1_joint_pdf.php)\n    - [5\\.2.2 Joint Cumulative Distribution Function](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_2_joint_cdf.php)\n    - [5\\.2.3 Conditioning and Independence](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_3_conditioning_independence.php)\n    - [5\\.2.4 Functions of Two Continuous Random Variables](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_4_functions.php)\n    - [5\\.2.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter5\/5_2_5_solved_prob.php)\n  - [5\\.3 More Topics]()\n    - [5\\.3.1 Covariance and Correlation](https:\/\/www.probabilitycourse.com\/chapter5\/5_3_1_covariance_correlation.php)\n    - [5\\.3.2 Bivariate Normal Distribution](https:\/\/www.probabilitycourse.com\/chapter5\/5_3_2_bivariate_normal_dist.php)\n    - [5\\.3.3 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter5\/5_3_3_solved_probs.php)\n  - [5\\.4 Problems]()\n    - [5\\.4.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter5\/5_4_0_chapter_problems.php)\n- [6 Multiple Random Variables]()\n  - [6\\.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter6\/6_0_0_intro.php)\n  - [6\\.1 Methods for More Than Two Random Variables]()\n    - [6\\.1.1 Joint Distributions and Independence](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_1_joint_distributions_independence.php)\n    - [6\\.1.2 Sums of Random Variables](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_2_sums_random_variables.php)\n    - [6\\.1.3 Moment Generating Functions](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_3_moment_functions.php)\n    - [6\\.1.4 Characteristic Functions](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_4_characteristic_functions.php)\n    - [6\\.1.5 Random Vectors](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_5_random_vectors.php)\n    - [6\\.1.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter6\/6_1_6_solved_probs.php)\n  - [6\\.2 Probability Bounds]()\n    - [6\\.2.0 Probability Bounds](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_0_probability_bounds.php)\n    - [6\\.2.1 Union Bound and Extension](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_1_union_bound_and_exten.php)\n    - [6\\.2.2 Markov Chebyshev Inequalities](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_2_markov_chebyshev_inequalities.php)\n    - [6\\.2.3 Chernoff Bounds](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_3_chernoff_bounds.php)\n    - [6\\.2.4 Cauchy Schwarz Inequality](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_4_cauchy_schwarz.php)\n    - [6\\.2.5 Jensen's Inequality](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_5_jensen's_inequality.php)\n    - [6\\.2.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter6\/6_2_6_solved6_2.php)\n  - [6\\.3 Problems]()\n    - [6\\.3.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter6\/6_3_0_chapter_problems.php)\n- [7 Limit Theorems and Convergence of Random Variables]()\n  - [7\\.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter7\/7_0_0_intro.php)\n  - [7\\.1 Limit Theorems]()\n    - [7\\.1.0 Limit Theorems](https:\/\/www.probabilitycourse.com\/chapter7\/7_1_0_limit_theorems.php)\n    - [7\\.1.1 Law of Large Numbers](https:\/\/www.probabilitycourse.com\/chapter7\/7_1_1_law_of_large_numbers.php)\n    - [7\\.1.2 Central Limit Theorem (CLT)](https:\/\/www.probabilitycourse.com\/chapter7\/7_1_2_central_limit_theorem.php)\n    - [7\\.1.3 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter7\/7_1_3_solved_probs.php)\n  - [7\\.2 Convergence of Random Variables]()\n    - [7\\.2.0 Convergence of Random Variables](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_0_convergence_of_random_variables.php)\n    - [7\\.2.1 Convergence of Sequence of Numbers](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_1_convergence_of_a_seq_of_nums.php)\n    - [7\\.2.2 Sequence of Random Variables](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_2_sequence_of_random_variables.php)\n    - [7\\.2.3 Different Types of Convergence for Sequences of Random Variables](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_3_different_types_of_convergence_for_sequences_of_random_variables.php)\n    - [7\\.2.4 Convergence in Distribution](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_4_convergence_in_distribution.php)\n    - [7\\.2.5 Convergence in Probability](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_5_convergence_in_probability.php)\n    - [7\\.2.6 Convergence in Mean](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_6_convergence_in_mean.php)\n    - [7\\.2.7 Almost Sure Convergence](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_7_almost_sure_convergence.php)\n    - [7\\.2.8 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter7\/7_2_8_solved_probs.php)\n  - [7\\.3 Problems]()\n    - [7\\.3.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter7\/7_3_0_chapter_problem.php)\n- [8 Statistical Inference I: Classical Methods]()\n  - [8\\.1 Introduction]()\n    - [8\\.1.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter8\/8_1_0_intro.php)\n    - [8\\.1.1 Random Sampling](https:\/\/www.probabilitycourse.com\/chapter8\/8_1_1_random_sampling.php)\n  - [8\\.2 Point Estimation]()\n    - [8\\.2.0 Point Estimation](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_0_point_estimation.php)\n    - [8\\.2.1 Evaluating Estimators](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_1_evaluating_estimators.php)\n    - [8\\.2.2 Point Estimators for Mean and Variance](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_2_point_estimators_for_mean_and_var.php)\n    - [8\\.2.3 Maximum Likelihood Estimation (MLE)](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_3_max_likelihood_estimation.php)\n    - [8\\.2.4 Asymptotic Properties of MLEs](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_4_asymptotic_probs_of_MLE.php)\n    - [8\\.2.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter8\/8_2_5_solved_probs.php)\n  - [8\\.3 Interval Estimation (Confidence Intervals)]()\n    - [8\\.3.0 Interval Estimation (Confidence Intervals)](https:\/\/www.probabilitycourse.com\/chapter8\/8_3_0_interval_estimation.php)\n    - [8\\.3.1 The general framework of Interval Estimation](https:\/\/www.probabilitycourse.com\/chapter8\/8_3_1_gen_framework_of_int_estimation.php)\n    - [8\\.3.2 Finding Interval Estimators](https:\/\/www.probabilitycourse.com\/chapter8\/8_3_2_finding_interval_estimators.php)\n    - [8\\.3.3 Confidence Intervals for Normal Samples](https:\/\/www.probabilitycourse.com\/chapter8\/8_3_3_confidence_intervals_for_norm_samples.php)\n    - [8\\.3.4 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter8\/8_3_4_solved_probs.php)\n  - [8\\.4 Hypothesis Testing]()\n    - [8\\.4.1 Introduction](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_1_intro.php)\n    - [8\\.4.2 General Setting and Definitions](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_2_general_setting_definitions.php)\n    - [8\\.4.3 Hypothesis Testing for the Mean](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_3_hypothesis_testing_for_mean.php)\n    - [8\\.4.4 P-Values](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_4_p_vals.php)\n    - [8\\.4.5 Likelihood Ratio Tests](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_5_likelihood_ratio_tests.php)\n    - [8\\.4.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter8\/8_4_6_solved_probs.php)\n  - [8\\.5 Linear Regression]()\n    - [8\\.5.0 Linear Regression](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_0_linear_regression.php)\n    - [8\\.5.1 Simple Linear Regression Model](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_1_simple_linear_regression_model.php)\n    - [8\\.5.2 The First Method for Finding beta](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_2_first_method_for_finding_beta.php)\n    - [8\\.5.3 The Method of Least Squares](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_3_the_method_of_least_squares.php)\n    - [8\\.5.4 Extensions and Issues](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_4_extensions_and_issues.php)\n    - [8\\.5.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter8\/8_5_5_solved_probs.php)\n  - [8\\.6 Problems]()\n    - [8\\.6.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter8\/8_6_0_ch_probs.php)\n- [9 Statistical Inference II: Bayesian Inference]()\n  - [9\\.1 Bayesian Inference]()\n    - [9\\.1.0 Bayesian Inference](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_0_bayesian_inference.php)\n    - [9\\.1.1 Prior and Posterior](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_1_prior_and_posterior.php)\n    - [9\\.1.2 Maximum A Posteriori (MAP) Estimation](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_2_MAP_estimation.php)\n    - [9\\.1.3 Comparison to ML Estimation](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_3_comparison_to_ML_estimation.php)\n    - [9\\.1.4 Conditional Expectation (MMSE)](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_4_conditional_expectation_MMSE.php)\n    - [9\\.1.5 Mean Squared Error (MSE)](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_5_mean_squared_error_MSE.php)\n    - [9\\.1.6 Linear MMSE Estimation of Random Variables](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_6_linear_MMSE_estimat_of_random_vars.php)\n    - [9\\.1.7 Estimation for Random Vectors](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_7_estimation_for_random_vectors.php)\n    - [9\\.1.8 Bayesian Hypothesis Testing](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_8_bayesian_hypothesis_testing.php)\n    - [9\\.1.9 Bayesian Interval Estimation](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_9_bayesian_interval_estimation.php)\n    - [9\\.1.10 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter9\/9_1_10_solved_probs.php)\n  - [9\\.2 Problems]()\n    - [9\\.2.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter9\/9_2_0_ch_probs.php)\n- [10 Introduction to Random Processes]()\n  - [10\\.1 Basic Concepts]()\n    - [10\\.1.0 Basic Concepts](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_0_basic_concepts.php)\n    - [10\\.1.1 PDFs and CDFs](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_1_PDFs_and_CDFs.php)\n    - [10\\.1.2 Mean and Correlation Functions](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_2_mean_and_correlation_functions.php)\n    - [10\\.1.3 Multiple Random Processes](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_3_multiple_random_processes.php)\n    - [10\\.1.4 Stationary Processes](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_4_stationary_processes.php)\n    - [10\\.1.5 Gaussian Random Processes](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_5_gaussian_random_processes.php)\n    - [10\\.1.6 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter10\/10_1_6_solved_probs.php)\n  - [10\\.2 Processing of Random Signals]()\n    - [10\\.2.0 Processing of Random Signals](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_0_processing_of_random_signals.php)\n    - [10\\.2.1 Power Spectral Density](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_1_power_spectral_density.php)\n    - [10\\.2.2 Linear Time-Invariant (LTI) Systems with Random Inputs](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_2_LTI_systems_with_random_inputs.php)\n    - [10\\.2.3 Power in a Frequency Band](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_3_power_in_a_frequency_band.php)\n    - [10\\.2.4 White Noise](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_4_white_noise.php)\n    - [10\\.2.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter10\/10_2_5_solved_probs.php)\n  - [10\\.3 Problems]()\n    - [10\\.3.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter10\/10_3_0_ch_probs.php)\n- [11 Some Important Random Processes]()\n  - [11\\.1 Poisson Processes]()\n    - [11\\.1.0 Introduction](https:\/\/www.probabilitycourse.com\/chapter11\/11_0_0_intro.php)\n    - [11\\.1.1 Counting Processes](https:\/\/www.probabilitycourse.com\/chapter11\/11_1_1_counting_processes.php)\n    - [11\\.1.2 Basic Concepts of the Poisson Process](https:\/\/www.probabilitycourse.com\/chapter11\/11_1_2_basic_concepts_of_the_poisson_process.php)\n    - [11\\.1.3 Merging and Splitting Poisson Processes](https:\/\/www.probabilitycourse.com\/chapter11\/11_1_3_merging_and_splitting_poisson_processes.php)\n    - [11\\.1.4 Nonhomogeneous Poisson Processes](https:\/\/www.probabilitycourse.com\/chapter11\/11_1_4_nonhomogeneous_poisson_processes.php)\n    - [11\\.1.5 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter11\/11_1_5_solved_probs.php)\n  - [11\\.2 Discrete-Time Markov Chains]()\n    - [11\\.2.1 Introduction](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_1_introduction.php)\n    - [11\\.2.2 State Transition Matrix and Diagram](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_2_state_transition_matrix_and_diagram.php)\n    - [11\\.2.3 Probability Distributions](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_3_probability_distributions.php)\n    - [11\\.2.4 Classification of States](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_4_classification_of_states.php)\n    - [11\\.2.5 Using the Law of Total Probability with Recursion](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_5_using_the_law_of_total_probability_with_recursion.php)\n    - [11\\.2.6 Stationary and Limiting Distributions](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_6_stationary_and_limiting_distributions.php)\n    - [11\\.2.7 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_7_solved_probs.php)\n  - [11\\.3 Continuous-Time Markov Chains]()\n    - [11\\.3.1 Introduction](https:\/\/www.probabilitycourse.com\/chapter11\/11_3_1_introduction.php)\n    - [11\\.3.2 Stationary and Limiting Distributions](https:\/\/www.probabilitycourse.com\/chapter11\/11_3_2_stationary_and_limiting_distributions.php)\n    - [11\\.3.3 The Generator Matrix](https:\/\/www.probabilitycourse.com\/chapter11\/11_3_3_the_generator_matrix.php)\n    - [11\\.3.4 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter11\/11_3_4_solved_probs.php)\n  - [11\\.4 Brownian Motion (Wiener Process)]()\n    - [11\\.4.0 Brownian Motion (Wiener Process)](https:\/\/www.probabilitycourse.com\/chapter11\/11_4_0_brownian_motion_wiener_process.php)\n    - [11\\.4.1 Brownian Motion as the Limit of a Symmetric Random Walk](https:\/\/www.probabilitycourse.com\/chapter11\/11_4_1_brownian_motion_as_the_limit_of_a_symmetric_random_walk.php)\n    - [1\\.4.2 Definition and Some Properties](https:\/\/www.probabilitycourse.com\/chapter11\/11_4_2_definition_and_some_properties.php)\n    - [11\\.4.3 Solved Problems](https:\/\/www.probabilitycourse.com\/chapter11\/11_4_3_solved_probs.php)\n  - [11\\.5 Problems]()\n    - [11\\.5.0 End of Chapter Problems](https:\/\/www.probabilitycourse.com\/chapter11\/11_5_0_end_of_chapter_problems.php)\n- [12 Introduction to Simulation Using MATLAB](https:\/\/www.probabilitycourse.com\/chapter12\/chapter12.php)\n- [13 Introduction to Simulation Using R](https:\/\/www.probabilitycourse.com\/chapter13\/chapter13.php)\n- [14 Introduction to Simulation Using Python](https:\/\/www.probabilitycourse.com\/chapter14\/chapter14.php)\n- [15 Recursive Methods](https:\/\/www.probabilitycourse.com\/chapter15\/chapter15.php)\n- [Appendix]()\n  - [Some Important Distributions](https:\/\/www.probabilitycourse.com\/appendix\/some_important_distributions.php)\n  - [Review of the Fourier Transform](https:\/\/www.probabilitycourse.com\/appendix\/review_fourier_transform.php)\n- [Bibliography](https:\/\/www.probabilitycourse.com\/bibliography.php)\n\n[![Creative Commons License](https:\/\/i.creativecommons.org\/l\/by-nc-nd\/3.0\/88x31.png)](https:\/\/creativecommons.org\/licenses\/by-nc-nd\/3.0\/deed.en_US)  \nIntroduction to Probability by  [Hossein Pishro-Nik](https:\/\/websites.umass.edu\/pishro\/) is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License](https:\/\/creativecommons.org\/licenses\/by-nc-nd\/3.0\/deed.en_US)","attrs_readable_markdown":"***\nTo better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state i to state j, we say that state j is *accessible* from state i. In particular, we can provide the following definitions.\n\nWe say that state j is **accessible** from state i, written as i → j, if p i j ( n ) \\> 0 for some n. We assume every state is accessible from itself since p i i ( 0 ) \\= 1.\n\nTwo states i and j are said to **communicate**, written as i ↔ j, if they are **accessible** from each other. In other words,\n\ni ↔ j means i → j and j → i .\n\nCommunication is an *equivalence* relation. That means that\n\n−\n\nevery state communicates with itself,\n\ni\n\n↔\n\ni\n\n;\n\n−\n\nif\n\ni\n\n↔\n\nj\n\n, then\n\nj\n\n↔\n\ni\n\n;\n\n−\n\nif\n\ni\n\n↔\n\nj\n\nand\n\nj\n\n↔\n\nk\n\n, then\n\ni\n\n↔\n\nk\n\n.\n\nTherefore, the states of a Markov chain can be partitioned into communicating *classes* such that only members of the same class communicate with each other. That is, two states i and j belong to the same class if and only if i ↔ j.\n***\nExample  \n Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state i to state j, then p i j \\> 0. Find the equivalence classes for this Markov chain.\n\n![MC-diagram-3](https:\/\/www.probabilitycourse.com\/images\/chapter11\/MC-diagram-3.png)\n\nFigure 11.9 - A state transition diagram.\n\n- [**Solution**]()\n  - There are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states\n    1\n    and\n    2\n    communicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes\n    3\n    and\n    4\n    communicate with each other, but they do not communicate with any other nodes in the graph. State\n    5\n    does not communicate with any other states, so it by itself is a class. Finally, states\n    6\n    ,\n    7\n    , and\n    8\n    construct another class. Thus, here are the classes:\n    Class 1 \\= { state 1 , state 2 } , Class 2 \\= { state 3 , state 4 } , Class 3 \\= { state 5 } , Class 4 \\= { state 6 , state 7 , state 8 } .\n    ![MC-diagram-4](https:\/\/www.probabilitycourse.com\/images\/chapter11\/MC-diagram-4.png)\n    \n    Figure 11.10 - Equivalence classes.\n***\nA Markov chain is said to be *irreducible* if it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior.\n\nA Markov chain is said to be **irreducible** if all states communicate with each other.\n\nLooking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class 4, it will always stay in that class. On the other hand, for other classes this is not true. For example, if X 0 \\= 1, then the Markov chain might stay in Class 1 for a while, but at some point, it will leave that class and it will never return to that class again. The states in Class 4 are called *recurrent*  states, while the other states in this chain are called *transient*.\n\nIn general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition:\n\nFor any state i, we define\n\nf i i \\= P ( X n \\= i , for some n ≥ 1 \\| X 0 \\= i ) .\n\nState i is **recurrent** if f i i \\= 1, and it is **transient** if f i i \\< 1.\n\nIt is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes.\n\nConsider a Markov chain and assume X 0 \\= i. If i is a recurrent state, then the chain will return to state i any time it leaves that state. Therefore, the chain will visit state i an infinite number of times. On the other hand, if i is a transient state, the chain will return to state i with probability f i i \\< 1. Thus, in that case, the total number of visits to state i will be a Geometric random variable with parameter 1 − f i i.\n\nConsider a discrete-time Markov chain. Let V be the total number of visits to state i.\n\n1. If\n   i\n   is a recurrent state, then\n   P ( V \\= ∞ \\| X 0 \\= i ) \\= 1\\.\n2. If\n   i\n   is a transient state, then\n   V \\| X 0 \\= i ∼ G e o m e t r i c ( 1 − f i i ) .\n***\nExample   \n Show that in a finite Markov chain, there is at least one recurrent class.\n\n- [**Solution**]()\n  - Consider a finite Markov chain with\n    r\n    states,\n    S\n    \n    \\=\n    \n    {\n    \n    1\n    \n    ,\n    \n    2\n    \n    ,\n    \n    ⋯\n    \n    ,\n    \n    r\n    \n    }\n    . Suppose that all states are transient. Then, starting from time\n    0\n    , the chain might visit state\n    1\n    several times, but at some point the chain will leave state\n    1\n    and will never return to it. That is, there exists an integer\n    M\n    \n    1\n    \n    \\>\n    \n    0\n    such that\n    X\n    \n    n\n    \n    ≠\n    \n    1\n    , for all\n    n\n    \n    ≥\n    \n    M\n    \n    1\n    . Similarly, there exists an integer\n    M\n    \n    2\n    \n    \\>\n    \n    0\n    such that\n    X\n    \n    n\n    \n    ≠\n    \n    2\n    , for all\n    n\n    \n    ≥\n    \n    M\n    \n    2\n    , and so on. Now, if you choose\n    n ≥ max { M 1 , M 2 , ⋯ , M r } ,\n    then\n    X\n    \n    n\n    cannot be equal to any of the states\n    1\n    \n    ,\n    \n    2\n    \n    ,\n    \n    ⋯\n    \n    ,\n    \n    r\n    . This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class.\n***\n### Periodicity:\nConsider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state 0, we only return to 0 at times n \\= 3 , 6 , ⋯. In other words, p 00 ( n ) \\= 0, if n is not divisible by 3. Such a state is called a *periodic* state with period d ( 0 ) \\= 3.\n\n![MC-diagram-periodic](https:\/\/www.probabilitycourse.com\/images\/chapter11\/MC-diagram-periodic.png)\n\nFigure 11.11 - A state transition diagram.\n\nThe **period** of a state i is the largest integer d satisfying the following property: p i i ( n ) \\= 0, whenever n is not divisible by d. The period of i is shown by d ( i ). If p i i ( n ) \\= 0, for all n \\> 0, then we let d ( i ) \\= ∞.\n\n−\n\nIf\n\nd\n\n(\n\ni\n\n)\n\n\\>\n\n1\n\n, we say that state\n\ni\n\nis **periodic**.\n\n−\n\nIf\n\nd\n\n(\n\ni\n\n)\n\n\\=\n\n1\n\n, we say that state\n\ni\n\nis **aperiodic**.\n\nYou can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic.\n\nIf i ↔ j , then d ( i ) \\= d ( j ) .\n\nWhy is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic.\n\nHow do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers m and l are said to be *co-prime* if their greatest common divisor (gcd) is 1, i.e., gcd ( l , m ) \\= 1. Now, suppose that we can find two co-prime numbers l and m such that p i i ( l ) \\> 0 and p i i ( m ) \\> 0. That is, we can go from state i to itself in l steps, and also in m steps. Then, we can conclude state i is aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number 1 is co-prime to every integer, any state with a self-transition is aperiodic.\n\nConsider a finite\n\nirreducible\n\nMarkov chain X n:\n\n1. If there is a self-transition in the chain (\n   p\n   \n   i\n   \n   i\n   \n   \\>\n   \n   0\n   for some\n   i\n   ), then the chain is aperiodic.\n2. Suppose that you can go from state\n   i\n   to state\n   i\n   in\n   l\n   steps, i.e.,\n   p\n   \n   i\n   \n   i\n   \n   (\n   \n   l\n   \n   )\n   \n   \\>\n   \n   0\n   . Also suppose that\n   p\n   \n   i\n   \n   i\n   \n   (\n   \n   m\n   \n   )\n   \n   \\>\n   \n   0\n   . If\n   gcd\n   \n   (\n   \n   l\n   \n   ,\n   \n   m\n   \n   )\n   \n   \\=\n   \n   1\n   , then state\n   i\n   is aperiodic.\n3. The chain is aperiodic if and only if there exists a positive integer\n   n\n   such that all elements of the matrix\n   P\n   \n   n\n   are strictly positive, i.e.,\n***\nExample   \n Consider the Markov chain in [Example 11.6](https:\/\/www.probabilitycourse.com\/chapter11\/11_2_4_classification_of_states.php#example11_6).\n\n1. Is\n   Class 1\n   \n   \\=\n   \n   {\n   \n   state\n   \n   1\n   \n   ,\n   \n   state\n   \n   2\n   \n   }\n   aperiodic?\n2. Is\n   Class 2\n   \n   \\=\n   \n   {\n   \n   state\n   \n   3\n   \n   ,\n   \n   state\n   \n   4\n   \n   }\n   aperiodic?\n3. Is\n   Class 4\n   \n   \\=\n   \n   {\n   \n   state\n   \n   6\n   \n   ,\n   \n   state\n   \n   7\n   \n   ,\n   \n   state\n   \n   8\n   \n   }\n   aperiodic?\n\n- [**Solution**]()\n  - 1. Class 1\n       \n       \\=\n       \n       {\n       \n       state\n       \n       1\n       \n       ,\n       \n       state\n       \n       2\n       \n       }\n       is\n       aperiodic\n       since it has a self-transition,\n       p\n       \n       22\n       \n       \\>\n       \n       0\n       .\n    2. Class 2\n       \n       \\=\n       \n       {\n       \n       state\n       \n       3\n       \n       ,\n       \n       state\n       \n       4\n       \n       }\n       is\n       periodic\n       with period\n       2\n       .\n    3. Class 4\n       \n       \\=\n       \n       {\n       \n       state\n       \n       6\n       \n       ,\n       \n       state\n       \n       7\n       \n       ,\n       \n       state\n       \n       8\n       \n       }\n       is\n       aperiodic\n       . For example, note that we can go from state\n       6\n       to state\n       6\n       in two steps (\n       6\n       \n       −\n       \n       7\n       \n       −\n       \n       6\n       ) and in three steps (\n       6\n       \n       −\n       \n       7\n       \n       −\n       \n       8\n       \n       −\n       \n       6\n       ). Since\n       gcd\n       \n       (\n       \n       2\n       \n       ,\n       \n       3\n       \n       )\n       \n       \\=\n       \n       1\n       , we conclude state\n       6\n       and its class are aperiodic.\n***\n***\n|   |\n|---|\n| The print version of the book is available on [Amazon](https:\/\/www.amazon.com\/Introduction-Probability-Statistics-Random-Processes\/dp\/0990637204\/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https:\/\/www.probabilitycourse.com\/images\/home\/book_cover.jpg)](https:\/\/www.amazon.com\/Introduction-Probability-Statistics-Random-Processes\/dp\/0990637204\/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) |\n| **Practical uncertainty:** *Useful Ideas in Decision-Making, Risk, Randomness, & AI* [![ractical Uncertaintly Cover](https:\/\/www.probabilitycourse.com\/images\/home\/practical_uncertaintly.PNG)](https:\/\/www.amazon.com\/dp\/B0CH2BHRVH\/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) |","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 day ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.1 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php
Last Crawled	2026-04-13 16:42:21 (1 day ago)
First Indexed	2018-12-15 13:22:05 (7 years ago)
HTTP Status Code	200
Meta Title	Classification of States
Meta Description	null
Meta Canonical	null
Boilerpipe Text	To better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state i to state j , we say that state j is accessible from state i . In particular, we can provide the following definitions. We say that state j is accessible from state i , written as i → j , if p i j ( n ) > 0 for some n . We assume every state is accessible from itself since p i i ( 0 ) = 1 . Two states i and j are said to communicate , written as i ↔ j , if they are accessible from each other. In other words, i ↔ j means i → j and j → i . Communication is an equivalence relation. That means that − every state communicates with itself, i ↔ i ; − if i ↔ j , then j ↔ i ; − if i ↔ j and j ↔ k , then i ↔ k . Therefore, the states of a Markov chain can be partitioned into communicating classes such that only members of the same class communicate with each other. That is, two states i and j belong to the same class if and only if i ↔ j . Example Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state i to state j , then p i j > 0 . Find the equivalence classes for this Markov chain. Figure 11.9 - A state transition diagram. Solution There are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states 1 and 2 communicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes 3 and 4 communicate with each other, but they do not communicate with any other nodes in the graph. State 5 does not communicate with any other states, so it by itself is a class. Finally, states 6 , 7 , and 8 construct another class. Thus, here are the classes: Class 1 = { state 1 , state 2 } , Class 2 = { state 3 , state 4 } , Class 3 = { state 5 } , Class 4 = { state 6 , state 7 , state 8 } . Figure 11.10 - Equivalence classes. A Markov chain is said to be irreducible if it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior. A Markov chain is said to be irreducible if all states communicate with each other. Looking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class 4 , it will always stay in that class. On the other hand, for other classes this is not true. For example, if X 0 = 1 , then the Markov chain might stay in Class 1 for a while, but at some point, it will leave that class and it will never return to that class again. The states in Class 4 are called recurrent states, while the other states in this chain are called transient . In general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition: For any state i , we define f i i = P ( X n = i , for some n ≥ 1 \| X 0 = i ) . State i is recurrent if f i i = 1 , and it is transient if f i i < 1 . It is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes. Consider a Markov chain and assume X 0 = i . If i is a recurrent state, then the chain will return to state i any time it leaves that state. Therefore, the chain will visit state i an infinite number of times. On the other hand, if i is a transient state, the chain will return to state i with probability f i i < 1 . Thus, in that case, the total number of visits to state i will be a Geometric random variable with parameter 1 − f i i . Consider a discrete-time Markov chain. Let V be the total number of visits to state i . If i is a recurrent state, then P ( V = ∞ \| X 0 = i ) = 1. If i is a transient state, then V \| X 0 = i ∼ G e o m e t r i c ( 1 − f i i ) . Example Show that in a finite Markov chain, there is at least one recurrent class. Solution Consider a finite Markov chain with r states, S = { 1 , 2 , ⋯ , r } . Suppose that all states are transient. Then, starting from time 0 , the chain might visit state 1 several times, but at some point the chain will leave state 1 and will never return to it. That is, there exists an integer M 1 > 0 such that X n ≠ 1 , for all n ≥ M 1 . Similarly, there exists an integer M 2 > 0 such that X n ≠ 2 , for all n ≥ M 2 , and so on. Now, if you choose n ≥ max { M 1 , M 2 , ⋯ , M r } , then X n cannot be equal to any of the states 1 , 2 , ⋯ , r . This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class. Periodicity: Consider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state 0 , we only return to 0 at times n = 3 , 6 , ⋯ . In other words, p 00 ( n ) = 0 , if n is not divisible by 3 . Such a state is called a periodic state with period d ( 0 ) = 3 . Figure 11.11 - A state transition diagram. The period of a state i is the largest integer d satisfying the following property: p i i ( n ) = 0 , whenever n is not divisible by d . The period of i is shown by d ( i ) . If p i i ( n ) = 0 , for all n > 0 , then we let d ( i ) = ∞ . − If d ( i ) > 1 , we say that state i is periodic . − If d ( i ) = 1 , we say that state i is aperiodic . You can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic. If i ↔ j , then d ( i ) = d ( j ) . Why is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic. How do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers m and l are said to be co-prime if their greatest common divisor (gcd) is 1 , i.e., gcd ( l , m ) = 1 . Now, suppose that we can find two co-prime numbers l and m such that p i i ( l ) > 0 and p i i ( m ) > 0 . That is, we can go from state i to itself in l steps, and also in m steps. Then, we can conclude state i is aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number 1 is co-prime to every integer, any state with a self-transition is aperiodic. Consider a finite irreducible Markov chain X n : If there is a self-transition in the chain ( p i i > 0 for some i ), then the chain is aperiodic. Suppose that you can go from state i to state i in l steps, i.e., p i i ( l ) > 0 . Also suppose that p i i ( m ) > 0 . If gcd ( l , m ) = 1 , then state i is aperiodic. The chain is aperiodic if and only if there exists a positive integer n such that all elements of the matrix P n are strictly positive, i.e., p i j ( n ) > 0 , for all i , j ∈ S . Example Consider the Markov chain in Example 11.6 . Is Class 1 = { state 1 , state 2 } aperiodic? Is Class 2 = { state 3 , state 4 } aperiodic? Is Class 4 = { state 6 , state 7 , state 8 } aperiodic? Solution Class 1 = { state 1 , state 2 } is aperiodic since it has a self-transition, p 22 > 0 . Class 2 = { state 3 , state 4 } is periodic with period 2 . Class 4 = { state 6 , state 7 , state 8 } is aperiodic . For example, note that we can go from state 6 to state 6 in two steps ( 6 − 7 − 6 ) and in three steps ( 6 − 7 − 8 − 6 ). Since gcd ( 2 , 3 ) = 1 , we conclude state 6 and its class are aperiodic. The print version of the book is available on Amazon . Practical uncertainty: Useful Ideas in Decision-Making, Risk, Randomness, & AI
Markdown	- [HOME](https://www.probabilitycourse.com/) - [VIDEOS](https://www.probabilitycourse.com/videos/videos.php) - [CALCULATOR](https://www.probabilitycourse.com/calculator/calculator.php) - [COMMENTS](https://www.probabilitycourse.com/comments.php) - [COURSES](https://www.probabilitycourse.com/courses.php) - [FOR INSTRUCTOR](https://www.probabilitycourse.com/for_instructors.php) - [LOG IN](https://www.probabilitycourse.com/Login/mobile_login.php) [![](https://www.probabilitycourse.com/images/Template/logo_probability_website.png)](https://www.probabilitycourse.com/) Chapters Menu - [HOME](https://www.probabilitycourse.com/) - [VIDEOS](https://www.probabilitycourse.com/videos/videos.php) - [CALCULATOR](https://www.probabilitycourse.com/calculator/calculator.php) - [COMMENTS](https://www.probabilitycourse.com/comments.php) - [COURSES](https://www.probabilitycourse.com/courses.php) - [FOR INSTRUCTORS](https://www.probabilitycourse.com/for_instructors.php) - [Sign In]() [Forgot password?](https://www.probabilitycourse.com/Login/forgot_password.php) [←]() [previous](https://www.probabilitycourse.com/chapter11/chapter11/11_2_3_probability_distributions.php) [next](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) [→]() *** ## 11\.2.4 Classification of States To better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state i i to state j j , we say that state j j is accessible from state i i . In particular, we can provide the following definitions. We say that state j j is accessible from state i i , written as i→j i → j , if p(n)ij\>0 p i j ( n ) \> 0 for some n n . We assume every state is accessible from itself since p(0)ii\=1 p i i ( 0 ) \= 1 . Two states i i and j j are said to communicate, written as i↔j i ↔ j , if they are accessible from each other. In other words, i↔j means i→j and j→i. i ↔ j means i → j and j → i . Communication is an equivalence relation. That means that − − every state communicates with itself, i↔i i ↔ i ; − − if i↔j i ↔ j , then j↔i j ↔ i ; − − if i↔j i ↔ j and j↔k j ↔ k , then i↔k i ↔ k . Therefore, the states of a Markov chain can be partitioned into communicating classes such that only members of the same class communicate with each other. That is, two states i i and j j belong to the same class if and only if i↔j i ↔ j . * Example Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state i i to state j j , then pij\>0 p i j \> 0 . Find the equivalence classes for this Markov chain. ![MC-diagram-3](https://www.probabilitycourse.com/chapter11/images/chapter11/MC-diagram-3.png) Figure 11.9 - A state transition diagram. - [Solution]() - There are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states 1 1 and 2 2 communicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes 3 3 and 4 4 communicate with each other, but they do not communicate with any other nodes in the graph. State 5 5 does not communicate with any other states, so it by itself is a class. Finally, states 6 6 , 7 7 , and 8 8 construct another class. Thus, here are the classes: Class 1\={state 1,state 2},Class 2\={state 3,state 4},Class 3\={state 5},Class 4\={state 6,state 7,state 8}. Class 1 \= { state 1 , state 2 } , Class 2 \= { state 3 , state 4 } , Class 3 \= { state 5 } , Class 4 \= { state 6 , state 7 , state 8 } . ![MC-diagram-4](https://www.probabilitycourse.com/chapter11/images/chapter11/MC-diagram-4.png) Figure 11.10 - Equivalence classes. * A Markov chain is said to be irreducible if it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior. A Markov chain is said to be irreducible if all states communicate with each other. Looking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class 4 4, it will always stay in that class. On the other hand, for other classes this is not true. For example, if X0\=1 X 0 \= 1, then the Markov chain might stay in Class 1 1 for a while, but at some point, it will leave that class and it will never return to that class again. The states in Class 4 4 are called recurrent states, while the other states in this chain are called transient. In general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition: For any state i i , we define fii\=P(Xn\=i, for some n≥1\\|X0\=i). f i i \= P ( X n \= i , for some n ≥ 1 \\| X 0 \= i ) . State i i is recurrent if fii\=1 f i i \= 1 , and it is transient if fii\<1 f i i \< 1 . It is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes. Consider a Markov chain and assume X0\=i X 0 \= i. If i i is a recurrent state, then the chain will return to state i i any time it leaves that state. Therefore, the chain will visit state i i an infinite number of times. On the other hand, if i i is a transient state, the chain will return to state i i with probability fii\<1 f i i \< 1. Thus, in that case, the total number of visits to state i i will be a Geometric random variable with parameter 1−fii 1 − f i i. Consider a discrete-time Markov chain. Let V V be the total number of visits to state i i . 1. If i i is a recurrent state, then P(V\=∞\\|X0\=i)\=1\. P ( V \= ∞ \\| X 0 \= i ) \= 1\. 2. If i i is a transient state, then V\\|X0\=i∼Geometric(1−fii). V \\| X 0 \= i ∼ G e o m e t r i c ( 1 − f i i ) . * Example Show that in a finite Markov chain, there is at least one recurrent class. - [Solution]() - Consider a finite Markov chain with r r states, S\={1,2,⋯,r} S \= { 1 , 2 , ⋯ , r } . Suppose that all states are transient. Then, starting from time 0 0 , the chain might visit state 1 1 several times, but at some point the chain will leave state 1 1 and will never return to it. That is, there exists an integer M1\>0 M 1 \> 0 such that Xn≠1 X n ≠ 1 , for all n≥M1 n ≥ M 1 . Similarly, there exists an integer M2\>0 M 2 \> 0 such that Xn≠2 X n ≠ 2 , for all n≥M2 n ≥ M 2 , and so on. Now, if you choose n≥max{M1,M2,⋯,Mr}, n ≥ max { M 1 , M 2 , ⋯ , M r } , then Xn X n cannot be equal to any of the states 1,2,⋯,r 1 , 2 , ⋯ , r . This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class. * ### Periodicity: Consider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state 0 0 , we only return to 0 0 at times n\=3,6,⋯ n \= 3 , 6 , ⋯ . In other words, p(n)00\=0 p 00 ( n ) \= 0 , if n n is not divisible by 3 3 . Such a state is called a periodic state with period d(0)\=3 d ( 0 ) \= 3 . ![MC-diagram-periodic](https://www.probabilitycourse.com/chapter11/images/chapter11/MC-diagram-periodic.png) Figure 11.11 - A state transition diagram. The period of a state i i is the largest integer d d satisfying the following property: p(n)ii\=0 p i i ( n ) \= 0 , whenever n n is not divisible by d d . The period of i i is shown by d(i) d ( i ) . If p(n)ii\=0 p i i ( n ) \= 0 , for all n\>0 n \> 0 , then we let d(i)\=∞ d ( i ) \= ∞ . − − If d(i)\>1 d ( i ) \> 1 , we say that state i i is periodic. − − If d(i)\=1 d ( i ) \= 1 , we say that state i i is aperiodic. You can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic. If i↔j i ↔ j , then d(i)\=d(j) d ( i ) \= d ( j ) . Why is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic. How do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers m m and l l are said to be co-prime if their greatest common divisor (gcd) is 1 1, i.e., gcd(l,m)\=1 gcd ( l , m ) \= 1. Now, suppose that we can find two co-prime numbers l l and m m such that p(l)ii\>0 p i i ( l ) \> 0 and p(m)ii\>0 p i i ( m ) \> 0. That is, we can go from state i i to itself in l l steps, and also in m m steps. Then, we can conclude state i i is aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number 1 1 is co-prime to every integer, any state with a self-transition is aperiodic. Consider a finite irreducible Markov chain Xn X n : 1. If there is a self-transition in the chain ( pii\>0 p i i \> 0 for some i i ), then the chain is aperiodic. 2. Suppose that you can go from state i i to state i i in l l steps, i.e., p(l)ii\>0 p i i ( l ) \> 0 . Also suppose that p(m)ii\>0 p i i ( m ) \> 0 . If gcd(l,m)\=1 gcd ( l , m ) \= 1 , then state i i is aperiodic. 3. The chain is aperiodic if and only if there exists a positive integer n n such that all elements of the matrix Pn P n are strictly positive, i.e., * Example Consider the Markov chain in [Example 11.6](https://www.probabilitycourse.com/chapter11/chapter11/11_2_4_classification_of_states.php#example11_6). 1. Is Class 1\={state 1,state 2} Class 1 \= { state 1 , state 2 } aperiodic? 2. Is Class 2\={state 3,state 4} Class 2 \= { state 3 , state 4 } aperiodic? 3. Is Class 4\={state 6,state 7,state 8} Class 4 \= { state 6 , state 7 , state 8 } aperiodic? - [Solution]() - 1. Class 1\={state 1,state 2} Class 1 \= { state 1 , state 2 } is aperiodic since it has a self-transition, p22\>0 p 22 \> 0 . 2. Class 2\={state 3,state 4} Class 2 \= { state 3 , state 4 } is periodic with period 2 2 . 3. Class 4\={state 6,state 7,state 8} Class 4 \= { state 6 , state 7 , state 8 } is aperiodic . For example, note that we can go from state 6 6 to state 6 6 in two steps ( 6−7−6 6 − 7 − 6 ) and in three steps ( 6−7−8−6 6 − 7 − 8 − 6 ). Since gcd(2,3)\=1 gcd ( 2 , 3 ) \= 1 , we conclude state 6 6 and its class are aperiodic. * [←]() [previous](https://www.probabilitycourse.com/chapter11/chapter11/11_2_3_probability_distributions.php) [next](https://www.probabilitycourse.com/chapter11/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) [→]() * \| \| \|---\| \| The print version of the book is available on [Amazon](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https://www.probabilitycourse.com/images/home/book_cover.jpg)](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) \| \| Practical uncertainty:** Useful Ideas in Decision-Making, Risk, Randomness, & AI [![ractical Uncertaintly Cover](https://www.probabilitycourse.com/images/home/practical_uncertaintly.PNG)](https://www.amazon.com/dp/B0CH2BHRVH/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) \| Open Menu - [0 Preface](https://www.probabilitycourse.com/preface.php) - [1 Basic Concepts]() - [1\.0 Introduction](https://www.probabilitycourse.com/chapter1/1_0_0_introduction.php) - [1\.1 Introduction]() - [1\.1.0 What Is Probability?](https://www.probabilitycourse.com/chapter1/1_1_0_what_is_probability.php) - [1\.1.1 Example](https://www.probabilitycourse.com/chapter1/1_1_1_example.php) - [1\.2 Review of Set Theory]() - [1\.2.0 Review](https://www.probabilitycourse.com/chapter1/1_2_0_review_set_theory.php) - [1\.2.1 Venn Diagrams](https://www.probabilitycourse.com/chapter1/1_2_1_venn.php) - [1\.2.2 Set Operations](https://www.probabilitycourse.com/chapter1/1_2_2_set_operations.php) - [1\.2.3 Cardinality](https://www.probabilitycourse.com/chapter1/1_2_3_cardinality.php) - [1\.2.4 Functions](https://www.probabilitycourse.com/chapter1/1_2_4_functions.php) - [1\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter1/1_2_5_solved1.php) - [1\.3 Random Experiments and Probabilities]() - [1\.3.1 Random Experiments](https://www.probabilitycourse.com/chapter1/1_3_1_random_experiments.php) - [1\.3.2 Probability](https://www.probabilitycourse.com/chapter1/1_3_2_probability.php) - [1\.3.3 Finding Probabilities](https://www.probabilitycourse.com/chapter1/1_3_3_finding_probabilities.php) - [1\.3.4 Discrete Models](https://www.probabilitycourse.com/chapter1/1_3_4_discrete_models.php) - [1\.3.5 Continuous Models](https://www.probabilitycourse.com/chapter1/1_3_5_continuous_models.php) - [1\.3.6 Solved Problems](https://www.probabilitycourse.com/chapter1/1_3_6_solved2.php) - [1\.4 Conditional Probability]() - [1\.4.0 Conditional Probability](https://www.probabilitycourse.com/chapter1/1_4_0_conditional_probability.php) - [1\.4.1 Independence](https://www.probabilitycourse.com/chapter1/1_4_1_independence.php) - [1\.4.2 Law of Total Probability](https://www.probabilitycourse.com/chapter1/1_4_2_total_probability.php) - [1\.4.3 Bayes' Rule](https://www.probabilitycourse.com/chapter1/1_4_3_bayes_rule.php) - [1\.4.4 Conditional Independence](https://www.probabilitycourse.com/chapter1/1_4_4_conditional_independence.php) - [1\.4.5 Solved Problems](https://www.probabilitycourse.com/chapter1/1_4_5_solved3.php) - [1\.5 Problems]() - [1\.5.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter1/1_5_0_chapter1_problems.php) - [2 Combinatorics: Counting Methods]() - [2\.1 Combinatorics]() - [2\.1.0 Finding Probabilities with Counting Methods](https://www.probabilitycourse.com/chapter2/2_1_0_counting.php) - [2\.1.1 Ordered with Replacement](https://www.probabilitycourse.com/chapter2/2_1_1_ordered_with_replacement.php) - [2\.1.2 Ordered without Replacement](https://www.probabilitycourse.com/chapter2/2_1_2_ordered_without_replacement.php) - [2\.1.3 Unordered without Replacement](https://www.probabilitycourse.com/chapter2/2_1_3_unordered_without_replacement.php) - [2\.1.4 Unordered with Replacement](https://www.probabilitycourse.com/chapter2/2_1_4_unordered_with_replacement.php) - [2\.1.5 Solved Problems](https://www.probabilitycourse.com/chapter2/2_1_5_solved2_1.php) - [2\.2 Problems]() - [2\.2.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter2/2_3_0_chapter2_problems.php) - [3 Discrete Random Variables]() - [3\.1 Basic Concepts]() - [3\.1.1 Random Variables](https://www.probabilitycourse.com/chapter3/3_1_1_random_variables.php) - [3\.1.2 Discrete Random Variables](https://www.probabilitycourse.com/chapter3/3_1_2_discrete_random_var.php) - [3\.1.3 Probability Mass Function](https://www.probabilitycourse.com/chapter3/3_1_3_pmf.php) - [3\.1.4 Independent Random Variables](https://www.probabilitycourse.com/chapter3/3_1_4_independent_random_var.php) - [3\.1.5 Special Distributions](https://www.probabilitycourse.com/chapter3/3_1_5_special_discrete_distr.php) - [3\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter3/3_1_6_solved3_1.php) - [3\.2 More about Discrete Random Variables]() - [3\.2.1 Cumulative Distribution Function](https://www.probabilitycourse.com/chapter3/3_2_1_cdf.php) - [3\.2.2 Expectation](https://www.probabilitycourse.com/chapter3/3_2_2_expectation.php) - [3\.2.3 Functions of Random Variables](https://www.probabilitycourse.com/chapter3/3_2_3_functions_random_var.php) - [3\.2.4 Variance](https://www.probabilitycourse.com/chapter3/3_2_4_variance.php) - [3\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter3/3_2_5_solved3_2.php) - [3\.3 Problems]() - [3\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter3/3_3_0_chapter3_problems.php) - [4 Continuous and Mixed Random Variables]() - [4\.0 Introduction](https://www.probabilitycourse.com/chapter4/4_0_0_intro.php) - [4\.1 Continuous Random Variables]() - [4\.1.0 Continuous Random Variables and their Distributions](https://www.probabilitycourse.com/chapter4/4_1_0_continuous_random_vars_distributions.php) - [4\.1.1 Probability Density Function](https://www.probabilitycourse.com/chapter4/4_1_1_pdf.php) - [4\.1.2 Expected Value and Variance](https://www.probabilitycourse.com/chapter4/4_1_2_expected_val_variance.php) - [4\.1.3 Functions of Continuous Random Variables](https://www.probabilitycourse.com/chapter4/4_1_3_functions_continuous_var.php) - [4\.1.4 Solved Problems](https://www.probabilitycourse.com/chapter4/4_1_4_solved4_1.php) - [4\.2 Special Distributions]() - [4\.2.1 Uniform Distribution](https://www.probabilitycourse.com/chapter4/4_2_1_uniform.php) - [4\.2.2 Exponential Distribution](https://www.probabilitycourse.com/chapter4/4_2_2_exponential.php) - [4\.2.3 Normal (Gaussian) Distribution](https://www.probabilitycourse.com/chapter4/4_2_3_normal.php) - [4\.2.4 Gamma Distribution](https://www.probabilitycourse.com/chapter4/4_2_4_Gamma_distribution.php) - [4\.2.5 Other Distributions](https://www.probabilitycourse.com/chapter4/4_2_5_other_distr.php) - [4\.2.6 Solved Problems](https://www.probabilitycourse.com/chapter4/4_2_6_solved4_2.php) - [4\.3 Mixed Random Variables]() - [4\.3.1 Mixed Random Variables](https://www.probabilitycourse.com/chapter4/4_3_1_mixed.php) - [4\.3.2 Using the Delta Function](https://www.probabilitycourse.com/chapter4/4_3_2_delta_function.php) - [4\.3.3 Solved Problems](https://www.probabilitycourse.com/chapter4/4_3_3_solved4_3.php) - [4\.4 Problems]() - [4\.4.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter4/4_4_0_chapter4_problems.php) - [5 Joint Distributions]() - [5\.1 Two Discrete Random Variables]() - [5\.1.0 Two Random Variables](https://www.probabilitycourse.com/chapter5/5_1_0_joint_distributions.php) - [5\.1.1 Joint Probability Mass Function (PMF)](https://www.probabilitycourse.com/chapter5/5_1_1_joint_pmf.php) - [5\.1.2 Joint Cumulative Distribution Function (CDF)](https://www.probabilitycourse.com/chapter5/5_1_2_joint_cdf.php) - [5\.1.3 Conditioning and Independence](https://www.probabilitycourse.com/chapter5/5_1_3_conditioning_independence.php) - [5\.1.4 Functions of Two Random Variables](https://www.probabilitycourse.com/chapter5/5_1_4_functions_two_variables.php) - [5\.1.5 Conditional Expectation](https://www.probabilitycourse.com/chapter5/5_1_5_conditional_expectation.php) - [5\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter5/5_1_6_solved_prob.php) - [5\.2 Two Continuous Random Variables]() - [5\.2.0 Two Continuous Random Variables](https://www.probabilitycourse.com/chapter5/5_2_0_continuous_vars.php) - [5\.2.1 Joint Probability Density Function](https://www.probabilitycourse.com/chapter5/5_2_1_joint_pdf.php) - [5\.2.2 Joint Cumulative Distribution Function](https://www.probabilitycourse.com/chapter5/5_2_2_joint_cdf.php) - [5\.2.3 Conditioning and Independence](https://www.probabilitycourse.com/chapter5/5_2_3_conditioning_independence.php) - [5\.2.4 Functions of Two Continuous Random Variables](https://www.probabilitycourse.com/chapter5/5_2_4_functions.php) - [5\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter5/5_2_5_solved_prob.php) - [5\.3 More Topics]() - [5\.3.1 Covariance and Correlation](https://www.probabilitycourse.com/chapter5/5_3_1_covariance_correlation.php) - [5\.3.2 Bivariate Normal Distribution](https://www.probabilitycourse.com/chapter5/5_3_2_bivariate_normal_dist.php) - [5\.3.3 Solved Problems](https://www.probabilitycourse.com/chapter5/5_3_3_solved_probs.php) - [5\.4 Problems]() - [5\.4.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter5/5_4_0_chapter_problems.php) - [6 Multiple Random Variables]() - [6\.0 Introduction](https://www.probabilitycourse.com/chapter6/6_0_0_intro.php) - [6\.1 Methods for More Than Two Random Variables]() - [6\.1.1 Joint Distributions and Independence](https://www.probabilitycourse.com/chapter6/6_1_1_joint_distributions_independence.php) - [6\.1.2 Sums of Random Variables](https://www.probabilitycourse.com/chapter6/6_1_2_sums_random_variables.php) - [6\.1.3 Moment Generating Functions](https://www.probabilitycourse.com/chapter6/6_1_3_moment_functions.php) - [6\.1.4 Characteristic Functions](https://www.probabilitycourse.com/chapter6/6_1_4_characteristic_functions.php) - [6\.1.5 Random Vectors](https://www.probabilitycourse.com/chapter6/6_1_5_random_vectors.php) - [6\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter6/6_1_6_solved_probs.php) - [6\.2 Probability Bounds]() - [6\.2.0 Probability Bounds](https://www.probabilitycourse.com/chapter6/6_2_0_probability_bounds.php) - [6\.2.1 Union Bound and Extension](https://www.probabilitycourse.com/chapter6/6_2_1_union_bound_and_exten.php) - [6\.2.2 Markov Chebyshev Inequalities](https://www.probabilitycourse.com/chapter6/6_2_2_markov_chebyshev_inequalities.php) - [6\.2.3 Chernoff Bounds](https://www.probabilitycourse.com/chapter6/6_2_3_chernoff_bounds.php) - [6\.2.4 Cauchy Schwarz Inequality](https://www.probabilitycourse.com/chapter6/6_2_4_cauchy_schwarz.php) - [6\.2.5 Jensen's Inequality](https://www.probabilitycourse.com/chapter6/6_2_5_jensen's_inequality.php) - [6\.2.6 Solved Problems](https://www.probabilitycourse.com/chapter6/6_2_6_solved6_2.php) - [6\.3 Problems]() - [6\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter6/6_3_0_chapter_problems.php) - [7 Limit Theorems and Convergence of Random Variables]() - [7\.0 Introduction](https://www.probabilitycourse.com/chapter7/7_0_0_intro.php) - [7\.1 Limit Theorems]() - [7\.1.0 Limit Theorems](https://www.probabilitycourse.com/chapter7/7_1_0_limit_theorems.php) - [7\.1.1 Law of Large Numbers](https://www.probabilitycourse.com/chapter7/7_1_1_law_of_large_numbers.php) - [7\.1.2 Central Limit Theorem (CLT)](https://www.probabilitycourse.com/chapter7/7_1_2_central_limit_theorem.php) - [7\.1.3 Solved Problems](https://www.probabilitycourse.com/chapter7/7_1_3_solved_probs.php) - [7\.2 Convergence of Random Variables]() - [7\.2.0 Convergence of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_0_convergence_of_random_variables.php) - [7\.2.1 Convergence of Sequence of Numbers](https://www.probabilitycourse.com/chapter7/7_2_1_convergence_of_a_seq_of_nums.php) - [7\.2.2 Sequence of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_2_sequence_of_random_variables.php) - [7\.2.3 Different Types of Convergence for Sequences of Random Variables](https://www.probabilitycourse.com/chapter7/7_2_3_different_types_of_convergence_for_sequences_of_random_variables.php) - [7\.2.4 Convergence in Distribution](https://www.probabilitycourse.com/chapter7/7_2_4_convergence_in_distribution.php) - [7\.2.5 Convergence in Probability](https://www.probabilitycourse.com/chapter7/7_2_5_convergence_in_probability.php) - [7\.2.6 Convergence in Mean](https://www.probabilitycourse.com/chapter7/7_2_6_convergence_in_mean.php) - [7\.2.7 Almost Sure Convergence](https://www.probabilitycourse.com/chapter7/7_2_7_almost_sure_convergence.php) - [7\.2.8 Solved Problems](https://www.probabilitycourse.com/chapter7/7_2_8_solved_probs.php) - [7\.3 Problems]() - [7\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter7/7_3_0_chapter_problem.php) - [8 Statistical Inference I: Classical Methods]() - [8\.1 Introduction]() - [8\.1.0 Introduction](https://www.probabilitycourse.com/chapter8/8_1_0_intro.php) - [8\.1.1 Random Sampling](https://www.probabilitycourse.com/chapter8/8_1_1_random_sampling.php) - [8\.2 Point Estimation]() - [8\.2.0 Point Estimation](https://www.probabilitycourse.com/chapter8/8_2_0_point_estimation.php) - [8\.2.1 Evaluating Estimators](https://www.probabilitycourse.com/chapter8/8_2_1_evaluating_estimators.php) - [8\.2.2 Point Estimators for Mean and Variance](https://www.probabilitycourse.com/chapter8/8_2_2_point_estimators_for_mean_and_var.php) - [8\.2.3 Maximum Likelihood Estimation (MLE)](https://www.probabilitycourse.com/chapter8/8_2_3_max_likelihood_estimation.php) - [8\.2.4 Asymptotic Properties of MLEs](https://www.probabilitycourse.com/chapter8/8_2_4_asymptotic_probs_of_MLE.php) - [8\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter8/8_2_5_solved_probs.php) - [8\.3 Interval Estimation (Confidence Intervals)]() - [8\.3.0 Interval Estimation (Confidence Intervals)](https://www.probabilitycourse.com/chapter8/8_3_0_interval_estimation.php) - [8\.3.1 The general framework of Interval Estimation](https://www.probabilitycourse.com/chapter8/8_3_1_gen_framework_of_int_estimation.php) - [8\.3.2 Finding Interval Estimators](https://www.probabilitycourse.com/chapter8/8_3_2_finding_interval_estimators.php) - [8\.3.3 Confidence Intervals for Normal Samples](https://www.probabilitycourse.com/chapter8/8_3_3_confidence_intervals_for_norm_samples.php) - [8\.3.4 Solved Problems](https://www.probabilitycourse.com/chapter8/8_3_4_solved_probs.php) - [8\.4 Hypothesis Testing]() - [8\.4.1 Introduction](https://www.probabilitycourse.com/chapter8/8_4_1_intro.php) - [8\.4.2 General Setting and Definitions](https://www.probabilitycourse.com/chapter8/8_4_2_general_setting_definitions.php) - [8\.4.3 Hypothesis Testing for the Mean](https://www.probabilitycourse.com/chapter8/8_4_3_hypothesis_testing_for_mean.php) - [8\.4.4 P-Values](https://www.probabilitycourse.com/chapter8/8_4_4_p_vals.php) - [8\.4.5 Likelihood Ratio Tests](https://www.probabilitycourse.com/chapter8/8_4_5_likelihood_ratio_tests.php) - [8\.4.6 Solved Problems](https://www.probabilitycourse.com/chapter8/8_4_6_solved_probs.php) - [8\.5 Linear Regression]() - [8\.5.0 Linear Regression](https://www.probabilitycourse.com/chapter8/8_5_0_linear_regression.php) - [8\.5.1 Simple Linear Regression Model](https://www.probabilitycourse.com/chapter8/8_5_1_simple_linear_regression_model.php) - [8\.5.2 The First Method for Finding beta](https://www.probabilitycourse.com/chapter8/8_5_2_first_method_for_finding_beta.php) - [8\.5.3 The Method of Least Squares](https://www.probabilitycourse.com/chapter8/8_5_3_the_method_of_least_squares.php) - [8\.5.4 Extensions and Issues](https://www.probabilitycourse.com/chapter8/8_5_4_extensions_and_issues.php) - [8\.5.5 Solved Problems](https://www.probabilitycourse.com/chapter8/8_5_5_solved_probs.php) - [8\.6 Problems]() - [8\.6.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter8/8_6_0_ch_probs.php) - [9 Statistical Inference II: Bayesian Inference]() - [9\.1 Bayesian Inference]() - [9\.1.0 Bayesian Inference](https://www.probabilitycourse.com/chapter9/9_1_0_bayesian_inference.php) - [9\.1.1 Prior and Posterior](https://www.probabilitycourse.com/chapter9/9_1_1_prior_and_posterior.php) - [9\.1.2 Maximum A Posteriori (MAP) Estimation](https://www.probabilitycourse.com/chapter9/9_1_2_MAP_estimation.php) - [9\.1.3 Comparison to ML Estimation](https://www.probabilitycourse.com/chapter9/9_1_3_comparison_to_ML_estimation.php) - [9\.1.4 Conditional Expectation (MMSE)](https://www.probabilitycourse.com/chapter9/9_1_4_conditional_expectation_MMSE.php) - [9\.1.5 Mean Squared Error (MSE)](https://www.probabilitycourse.com/chapter9/9_1_5_mean_squared_error_MSE.php) - [9\.1.6 Linear MMSE Estimation of Random Variables](https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_estimat_of_random_vars.php) - [9\.1.7 Estimation for Random Vectors](https://www.probabilitycourse.com/chapter9/9_1_7_estimation_for_random_vectors.php) - [9\.1.8 Bayesian Hypothesis Testing](https://www.probabilitycourse.com/chapter9/9_1_8_bayesian_hypothesis_testing.php) - [9\.1.9 Bayesian Interval Estimation](https://www.probabilitycourse.com/chapter9/9_1_9_bayesian_interval_estimation.php) - [9\.1.10 Solved Problems](https://www.probabilitycourse.com/chapter9/9_1_10_solved_probs.php) - [9\.2 Problems]() - [9\.2.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter9/9_2_0_ch_probs.php) - [10 Introduction to Random Processes]() - [10\.1 Basic Concepts]() - [10\.1.0 Basic Concepts](https://www.probabilitycourse.com/chapter10/10_1_0_basic_concepts.php) - [10\.1.1 PDFs and CDFs](https://www.probabilitycourse.com/chapter10/10_1_1_PDFs_and_CDFs.php) - [10\.1.2 Mean and Correlation Functions](https://www.probabilitycourse.com/chapter10/10_1_2_mean_and_correlation_functions.php) - [10\.1.3 Multiple Random Processes](https://www.probabilitycourse.com/chapter10/10_1_3_multiple_random_processes.php) - [10\.1.4 Stationary Processes](https://www.probabilitycourse.com/chapter10/10_1_4_stationary_processes.php) - [10\.1.5 Gaussian Random Processes](https://www.probabilitycourse.com/chapter10/10_1_5_gaussian_random_processes.php) - [10\.1.6 Solved Problems](https://www.probabilitycourse.com/chapter10/10_1_6_solved_probs.php) - [10\.2 Processing of Random Signals]() - [10\.2.0 Processing of Random Signals](https://www.probabilitycourse.com/chapter10/10_2_0_processing_of_random_signals.php) - [10\.2.1 Power Spectral Density](https://www.probabilitycourse.com/chapter10/10_2_1_power_spectral_density.php) - [10\.2.2 Linear Time-Invariant (LTI) Systems with Random Inputs](https://www.probabilitycourse.com/chapter10/10_2_2_LTI_systems_with_random_inputs.php) - [10\.2.3 Power in a Frequency Band](https://www.probabilitycourse.com/chapter10/10_2_3_power_in_a_frequency_band.php) - [10\.2.4 White Noise](https://www.probabilitycourse.com/chapter10/10_2_4_white_noise.php) - [10\.2.5 Solved Problems](https://www.probabilitycourse.com/chapter10/10_2_5_solved_probs.php) - [10\.3 Problems]() - [10\.3.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter10/10_3_0_ch_probs.php) - [11 Some Important Random Processes]() - [11\.1 Poisson Processes]() - [11\.1.0 Introduction](https://www.probabilitycourse.com/chapter11/11_0_0_intro.php) - [11\.1.1 Counting Processes](https://www.probabilitycourse.com/chapter11/11_1_1_counting_processes.php) - [11\.1.2 Basic Concepts of the Poisson Process](https://www.probabilitycourse.com/chapter11/11_1_2_basic_concepts_of_the_poisson_process.php) - [11\.1.3 Merging and Splitting Poisson Processes](https://www.probabilitycourse.com/chapter11/11_1_3_merging_and_splitting_poisson_processes.php) - [11\.1.4 Nonhomogeneous Poisson Processes](https://www.probabilitycourse.com/chapter11/11_1_4_nonhomogeneous_poisson_processes.php) - [11\.1.5 Solved Problems](https://www.probabilitycourse.com/chapter11/11_1_5_solved_probs.php) - [11\.2 Discrete-Time Markov Chains]() - [11\.2.1 Introduction](https://www.probabilitycourse.com/chapter11/11_2_1_introduction.php) - [11\.2.2 State Transition Matrix and Diagram](https://www.probabilitycourse.com/chapter11/11_2_2_state_transition_matrix_and_diagram.php) - [11\.2.3 Probability Distributions](https://www.probabilitycourse.com/chapter11/11_2_3_probability_distributions.php) - [11\.2.4 Classification of States](https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php) - [11\.2.5 Using the Law of Total Probability with Recursion](https://www.probabilitycourse.com/chapter11/11_2_5_using_the_law_of_total_probability_with_recursion.php) - [11\.2.6 Stationary and Limiting Distributions](https://www.probabilitycourse.com/chapter11/11_2_6_stationary_and_limiting_distributions.php) - [11\.2.7 Solved Problems](https://www.probabilitycourse.com/chapter11/11_2_7_solved_probs.php) - [11\.3 Continuous-Time Markov Chains]() - [11\.3.1 Introduction](https://www.probabilitycourse.com/chapter11/11_3_1_introduction.php) - [11\.3.2 Stationary and Limiting Distributions](https://www.probabilitycourse.com/chapter11/11_3_2_stationary_and_limiting_distributions.php) - [11\.3.3 The Generator Matrix](https://www.probabilitycourse.com/chapter11/11_3_3_the_generator_matrix.php) - [11\.3.4 Solved Problems](https://www.probabilitycourse.com/chapter11/11_3_4_solved_probs.php) - [11\.4 Brownian Motion (Wiener Process)]() - [11\.4.0 Brownian Motion (Wiener Process)](https://www.probabilitycourse.com/chapter11/11_4_0_brownian_motion_wiener_process.php) - [11\.4.1 Brownian Motion as the Limit of a Symmetric Random Walk](https://www.probabilitycourse.com/chapter11/11_4_1_brownian_motion_as_the_limit_of_a_symmetric_random_walk.php) - [1\.4.2 Definition and Some Properties](https://www.probabilitycourse.com/chapter11/11_4_2_definition_and_some_properties.php) - [11\.4.3 Solved Problems](https://www.probabilitycourse.com/chapter11/11_4_3_solved_probs.php) - [11\.5 Problems]() - [11\.5.0 End of Chapter Problems](https://www.probabilitycourse.com/chapter11/11_5_0_end_of_chapter_problems.php) - [12 Introduction to Simulation Using MATLAB](https://www.probabilitycourse.com/chapter12/chapter12.php) - [13 Introduction to Simulation Using R](https://www.probabilitycourse.com/chapter13/chapter13.php) - [14 Introduction to Simulation Using Python](https://www.probabilitycourse.com/chapter14/chapter14.php) - [15 Recursive Methods](https://www.probabilitycourse.com/chapter15/chapter15.php) - [Appendix]() - [Some Important Distributions](https://www.probabilitycourse.com/appendix/some_important_distributions.php) - [Review of the Fourier Transform](https://www.probabilitycourse.com/appendix/review_fourier_transform.php) - [Bibliography](https://www.probabilitycourse.com/bibliography.php) [![Creative Commons License](https://i.creativecommons.org/l/by-nc-nd/3.0/88x31.png)](https://creativecommons.org/licenses/by-nc-nd/3.0/deed.en_US) Introduction to Probability by [Hossein Pishro-Nik](https://websites.umass.edu/pishro/) is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License](https://creativecommons.org/licenses/by-nc-nd/3.0/deed.en_US)
Readable Markdown	*** To better understand Markov chains, we need to introduce some definitions. The first definition concerns the accessibility of states from each other: If it is possible to go from state i to state j, we say that state j is accessible from state i. In particular, we can provide the following definitions. We say that state j is accessible from state i, written as i → j, if p i j ( n ) \> 0 for some n. We assume every state is accessible from itself since p i i ( 0 ) \= 1. Two states i and j are said to communicate, written as i ↔ j, if they are accessible from each other. In other words, i ↔ j means i → j and j → i . Communication is an equivalence relation. That means that − every state communicates with itself, i ↔ i ; − if i ↔ j , then j ↔ i ; − if i ↔ j and j ↔ k , then i ↔ k . Therefore, the states of a Markov chain can be partitioned into communicating classes such that only members of the same class communicate with each other. That is, two states i and j belong to the same class if and only if i ↔ j. * Example Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow from state i to state j, then p i j \> 0. Find the equivalence classes for this Markov chain. ![MC-diagram-3](https://www.probabilitycourse.com/images/chapter11/MC-diagram-3.png) Figure 11.9 - A state transition diagram. - [Solution]() - There are four communicating classes in this Markov chain. Looking at Figure 11.10, we notice that states 1 and 2 communicate with each other, but they do not communicate with any other nodes in the graph. Similarly, nodes 3 and 4 communicate with each other, but they do not communicate with any other nodes in the graph. State 5 does not communicate with any other states, so it by itself is a class. Finally, states 6 , 7 , and 8 construct another class. Thus, here are the classes: Class 1 \= { state 1 , state 2 } , Class 2 \= { state 3 , state 4 } , Class 3 \= { state 5 } , Class 4 \= { state 6 , state 7 , state 8 } . ![MC-diagram-4](https://www.probabilitycourse.com/images/chapter11/MC-diagram-4.png) Figure 11.10 - Equivalence classes. * A Markov chain is said to be irreducible if it has only one communicating class. As we will see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the limiting behavior. A Markov chain is said to be irreducible if all states communicate with each other. Looking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any time the Markov chain enters Class 4, it will always stay in that class. On the other hand, for other classes this is not true. For example, if X 0 \= 1, then the Markov chain might stay in Class 1 for a while, but at some point, it will leave that class and it will never return to that class again. The states in Class 4 are called recurrent states, while the other states in this chain are called transient. In general, a state is said to be recurrent if, any time that we leave that state, we will return to that state in the future with probability one. On the other hand, if the probability of returning is less than one, the state is called transient. Here, we provide a formal definition: For any state i, we define f i i \= P ( X n \= i , for some n ≥ 1 \\| X 0 \= i ) . State i is recurrent if f i i \= 1, and it is transient if f i i \< 1. It is relatively easy to show that if two states are in the same class, either both of them are recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the states are transient, the class is called transient. In general, a Markov chain might consist of several transient classes as well as several recurrent classes. Consider a Markov chain and assume X 0 \= i. If i is a recurrent state, then the chain will return to state i any time it leaves that state. Therefore, the chain will visit state i an infinite number of times. On the other hand, if i is a transient state, the chain will return to state i with probability f i i \< 1. Thus, in that case, the total number of visits to state i will be a Geometric random variable with parameter 1 − f i i. Consider a discrete-time Markov chain. Let V be the total number of visits to state i. 1. If i is a recurrent state, then P ( V \= ∞ \\| X 0 \= i ) \= 1\. 2. If i is a transient state, then V \\| X 0 \= i ∼ G e o m e t r i c ( 1 − f i i ) . * Example Show that in a finite Markov chain, there is at least one recurrent class. - [Solution]() - Consider a finite Markov chain with r states, S \= { 1 , 2 , ⋯ , r } . Suppose that all states are transient. Then, starting from time 0 , the chain might visit state 1 several times, but at some point the chain will leave state 1 and will never return to it. That is, there exists an integer M 1 \> 0 such that X n ≠ 1 , for all n ≥ M 1 . Similarly, there exists an integer M 2 \> 0 such that X n ≠ 2 , for all n ≥ M 2 , and so on. Now, if you choose n ≥ max { M 1 , M 2 , ⋯ , M r } , then X n cannot be equal to any of the states 1 , 2 , ⋯ , r . This is a contradiction, so we conclude that there must be at least one recurrent state, which means that there must be at least one recurrent class. * ### Periodicity: Consider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. Starting from state 0, we only return to 0 at times n \= 3 , 6 , ⋯. In other words, p 00 ( n ) \= 0, if n is not divisible by 3. Such a state is called a periodic state with period d ( 0 ) \= 3. ![MC-diagram-periodic](https://www.probabilitycourse.com/images/chapter11/MC-diagram-periodic.png) Figure 11.11 - A state transition diagram. The period of a state i is the largest integer d satisfying the following property: p i i ( n ) \= 0, whenever n is not divisible by d. The period of i is shown by d ( i ). If p i i ( n ) \= 0, for all n \> 0, then we let d ( i ) \= ∞. − If d ( i ) \> 1 , we say that state i is periodic. − If d ( i ) \= 1 , we say that state i is aperiodic. You can show that all states in the same communicating class have the same period. A class is said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic. If i ↔ j , then d ( i ) \= d ( j ) . Why is periodicity important? As we will see shortly, it plays a role when we discuss limiting distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, and we need to check if it is aperiodic. How do we check that a Markov chain is aperiodic? Here is a useful method. Remember that two numbers m and l are said to be co-prime if their greatest common divisor (gcd) is 1, i.e., gcd ( l , m ) \= 1. Now, suppose that we can find two co-prime numbers l and m such that p i i ( l ) \> 0 and p i i ( m ) \> 0. That is, we can go from state i to itself in l steps, and also in m steps. Then, we can conclude state i is aperiodic. If we have an irreducible Markov chain, this means that the chain is aperiodic. Since the number 1 is co-prime to every integer, any state with a self-transition is aperiodic. Consider a finite irreducible Markov chain X n: 1. If there is a self-transition in the chain ( p i i \> 0 for some i ), then the chain is aperiodic. 2. Suppose that you can go from state i to state i in l steps, i.e., p i i ( l ) \> 0 . Also suppose that p i i ( m ) \> 0 . If gcd ( l , m ) \= 1 , then state i is aperiodic. 3. The chain is aperiodic if and only if there exists a positive integer n such that all elements of the matrix P n are strictly positive, i.e., * Example Consider the Markov chain in [Example 11.6](https://www.probabilitycourse.com/chapter11/11_2_4_classification_of_states.php#example11_6). 1. Is Class 1 \= { state 1 , state 2 } aperiodic? 2. Is Class 2 \= { state 3 , state 4 } aperiodic? 3. Is Class 4 \= { state 6 , state 7 , state 8 } aperiodic? - [Solution]() - 1. Class 1 \= { state 1 , state 2 } is aperiodic since it has a self-transition, p 22 \> 0 . 2. Class 2 \= { state 3 , state 4 } is periodic with period 2 . 3. Class 4 \= { state 6 , state 7 , state 8 } is aperiodic . For example, note that we can go from state 6 to state 6 in two steps ( 6 − 7 − 6 ) and in three steps ( 6 − 7 − 8 − 6 ). Since gcd ( 2 , 3 ) \= 1 , we conclude state 6 and its class are aperiodic. * * \| \| \|---\| \| The print version of the book is available on [Amazon](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik). [![Book Cover](https://www.probabilitycourse.com/images/home/book_cover.jpg)](https://www.amazon.com/Introduction-Probability-Statistics-Random-Processes/dp/0990637204/ref=sr_1_1?ie=UTF8&qid=1408880878&sr=8-1&keywords=pishro-nik) \| \| Practical uncertainty:** Useful Ideas in Decision-Making, Risk, Randomness, & AI [![ractical Uncertaintly Cover](https://www.probabilitycourse.com/images/home/practical_uncertaintly.PNG)](https://www.amazon.com/dp/B0CH2BHRVH/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693837152&sr=8-1) \|
Shard	115 (laksa)
Root Hash	8531609719846996715
Unparsed URL	com,probabilitycourse!www,/chapter11/11_2_4_classification_of_states.php s443