10.2: Applications of Markov Chains

Learning Objectives

In this section you will examine some ways in which Markov Chains models are used in business, finance, public health and other fields of application.

In the previous section, we examined several applications of Markov chains. Before we proceed further in our investigations of the mathematics of Markov chains in the next sections, we take the time in this section to examine how Markov chains are used in real world applications.

In our bike share program example, we modeled the distribution of the locations of bicycles at bike share stations using a Markov chain. Markov chains have been proposed to model locations of cars distributed among multiple car rental locations for a car rental company, and locations of cars in car share programs. Markov chains models analyze package delivery schedules when packages are transported between several intermediate transport and storage locations on their way to their final destination. In these situations, Markov chains are often one part of larger mathematical models using a combination of other techniques, such as optimization to maximize profit or revenue or minimize cost using linear programming.

In our cable TV example, we modeled market share in a simple example of two cable TV providers. Markov chains can be similarly used in market research studies for many types of products and services, to model brand loyalty and brand transitions as we did in the cable TV model. In the field of finance, Markov chains can model investment return and risk for various types of investments.

Markov chains can model the probabilities of claims for insurance, such as life insurance and disability insurance, and for pensions and annuities. For example, for disability insurance, a much simplified model might include states of healthy, temporarily disabled, permanently disabled, recovered, and deceased; additional refinements could distinguish between disabled policyholders still in the waiting period before collecting benefits and claims actively collecting benefits.

Markov chains have been used in the fields of public health and medicine. Markov chains models of HIV and AIDS include states to model HIV transmission, progression to AIDs, and survival (living with HIV or AIDS) versus death due to AIDS. Comparing Markov chain models of HIV transmission and AIDs progression for various risk groups and ethnic groups can guide public health organizations in developing strategies for reducing risk and managing care for these various groups of people. In general, modeling transmission of various infectious diseases with Markov chains can help in determination of appropriate public health responses to monitor and slow or halt the transmission of these diseases and to determine the most efficient ways to approach treating the disease.

Markov chains have many health applications besides modeling spread and progression of infectious diseases. When analyzing infertility treatments, Markov chains can model the probability of successful pregnancy as a result of a sequence of infertility treatments. Another medical application is analysis of medical risk, such as the role of risk in patient condition following surgery; the Markov chain model quantifies the probabilities of patients progressing between various states of health.

Markov chains are used in ranking of websites in web searches. Markov chains model the probabilities of linking to a list of sites from other sites on that list; a link represents a transition. The Markov chain is analyzed to determine if there is a steady state distribution, or equilibrium, after many transitions. Once equilibrium is identified, the pages with the probabilities in the equilibrium distribution determine the ranking of the webpages. This is a very simplified description of how Google uses Markov chains and matrices to determine “Page rankings” as part of their search algorithms.

Of course, a real world use of such a model by Google would involve immense matrices with thousands of rows and columns. The size of such matrices requires some modifications and use of more sophisticated techniques than we study for Markov chains in this course. However the methods we study form the underlying basis for this concept. It is interesting to note that the term Page ranking does not refer to the fact that webpages are ranked, but instead is named after Google founder Larry Page, who was instrumental in developing this application of Markov chains in the area of web page search and rankings.

Markov chains are also used in quality analysis of cell phone and other communications transmissions. Transition matrices model the probabilities of certain types of signals being transmitted in sequence. Certain sequences of signals are more common and expected, having higher probabilities; on the other hand, other sequences of signals are rare and have low probabilities of occurrence. If certain sequences of signals that are unlikely to occur actually do occur, that might be an indication of errors in transmissions; Markov chains help identify the sequences that represent likely transmission errors.