4.5: Conditional Probability and Multiplication Rules
- Page ID
- 105831
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)Section 1: Introduction to Conditional Probability
Sometimes the occurrence of one event may affect the probability of the other event. For example, the chances of a student passing a class depend on whether they ace the final exam or not. This relation between two events is called dependency, and to quantify such dependency we introduce the concept of conditional probability.
Conditional probability \(P(A|B)\) (probability of A given B) is the probability of event \(A\) knowing that \(B\) has occurred. In the context of conditional probability, \(P(A)\) is called a prior probability, \(P(A|B)\) is called posterior probability.
In general, the \(P(A|B)\) is not the same as the \(P(B|A)\).
Consider rolling a die. Let \(A\) be an event whose outcomes are 1,3,5. The probability of A is known and is equal to 0.5. Let’s consider event \(B\) whose outcomes are {4,5,6}. Now, we are ready to roll a die and we are interested in the event \(A\). As of right now, prior to rolling a die, the probability of \(A\) is ½ but imagine that before the outcome of the experiment is announced to us someone whispers that event \(B\) has occurred. Does the knowledge of \(B\) occurring change how we evaluate the probability of \(A\)? What exactly changed once we learned that \(B\) occurred?
To understand what changes, we must look into how the probabilities are computed! The prior probability of A is computed by dividing the number of outcomes in A by the size of the sample size.
\(P(A)=\frac{|\{1,3,5\}|}{|\{1,2,3,4,5,6\}|}=\frac{3}{6}=\frac{1}{2}\)
So what exactly changes when we know that \(B\) has occurred. We may not know what exact outcome has happened but we know that it must be one of the three that describe event \(B\), so event \(B\) is the new sample space for event \(A\). Since 1 and 3 are not in the list of possible outcomes anymore we remove them so event \(A\) can only occur if 5 was the outcome of the experiment. In other words, when \(B\) occurred the event \(A\) may only occur on one way out the remaining three outcomes, so the probability of \(A\) given \(B\):
\(P(A|B)=\frac{|\{5\}|}{|\{4,5,6\}|}=\frac{1}{3}\)
In general, any conditional probability can be found by finding the restricted sample space, and then finding the number of ways the event can occur in the restricted sample space.
Let’s consider for the same experiment the events \(C=\{1,2\}\) and \(D=\({5,6\}\) in addition to previously defined \(A\) and \(B\). Let’s find some prior and posterior probabilities.
The prior probability of \(B\) can be computed by dividing the total number of outcomes that describe \(B\) by the total number of outcomes in the sample space. The prior probability of \(D\) can be computed by dividing the total number of outcomes that describe \(D\) by the total number of outcomes in the sample space. The probability of \(B\) given \(D\) can be computed by dividing the total number of outcomes in \(D\) that describe \(B\) by the total number of outcomes in \(D\). The probability of \(D\) given \(B\) can be computed by dividing the total number of outcomes in \(B\) that describe \(D\) by the total number of outcomes in \(B\).
|
Prior |
Posterior |
|---|---|
|
\(P(B)=\frac{|\{4,5,6\}|}{|\{1,2,3,4,5,6\}|}=\frac{3}{6}=\frac{1}{2}\) |
\(P(B|D)=\frac{|\{5,6\}|}{|\{5,6\}|}=\frac{2}{2}=1\) |
| \(P(D)=\frac{|\{5,6\}|}{|\{1,2,3,4,5,6\}|}=\frac{2}{6}=\frac{1}{3}\) | \(P(D|B)=\frac{|\{5,6\}|}{|\{4,5,6\}|}=\frac{2}{3}\) |
The prior probability of \(A\) can be computed by dividing the total number of outcomes that describe \(A\) by the total number of outcomes in the sample space. The prior probability of \(C\) can be computed by dividing the total number of outcomes that describe \(C\) by the total number of outcomes in the sample space. The probability of \(A\) given \(C\) can be computed by dividing the total number of outcomes in \(C\) that describe \(A\) by the total number of outcomes in \(C\).
|
Prior |
Posterior |
|---|---|
|
\(P(A)=\frac{|\{1,3,5\}|}{|\{1,2,3,4,5,6\}|}=\frac{3}{6}=\frac{1}{2}\) |
\(P(A|C)=\frac{|\{1\}|}{|\{1,2\}|}=\frac{1}{2}\) |
| \(P(C)=\frac{|\{1,2\}|}{|\{1,2,3,4,5,6\}|}=\frac{2}{6}=\frac{1}{3}\) | \(P(C|A)=\frac{|\{1\}|}{|\{1,3,5\}|}=\frac{1}{3}\) |
If we write out the results of prior and conditional probabilities, we will be able to see an interesting phenomenon. The probability of \(A\) didn’t change after we learned that \(C\) occurred, so as the probability of \(C\) didn’t change after we learned that \(A\) has occurred. Such events are called independent. On the other hand, the probability of \(B\) changed after we learned that \(D\) occurred, so as the probability of \(D\) changed after we learned that \(B\) has occurred. Such events are called dependent.
If \(A\) and \(B\) are such that occurrence of one doesn't affect the probability of another, then we call \(A\) and \(B\) independent, otherwise we call such events dependent.
Symbolically we can express the definition of two independent events by the following statements: \(P(A|B)=P(A)\) and \(P(B|A)=P(B)\). If one of them is true, the other one must be true as well. So if anyone wants to check whether two events are independent one must find the prior probability and compare it to the conditional probability to see if they are equal.
Note that frequently students confuse mutually exclusive and independent events. Mutually exclusive events have no common outcomes that is \(P(A\text{ and }B)=0\). With independent events, the occurrence of one does not affect the probability of another that is \(P(A|B)=P(A)\) and \(P(B|A)=P(B)\).
Consider the birth of a child. Before the sex of the child is known, there are two possible outcomes \(B\) or \(G\). Normally, they are considered to be mutually exclusive outcomes. The prior probability that the child is a boy is 0.5 If we ask ourselves what is the probability that the child is a boy given that she is a girl we will find the conditional probability of \(P(B|G)\) which is obviously 0, so the two events are dependent because the probability of \(B\) went down to zero once we learned that event \(G\) has occurred. So the only relation between the two concepts is that the two mutually exclusive events are always dependent!
Section 2: Multiplication Rules
Previously, we set the goal to be able to find the probabilities of compound events if the probabilities of the original events are given. For finding the probability of the complementary events we developed the complementary rule, for finding the probability of the union of two events we developed the special and general addition rules. The next goal is to develop the rule for finding the probability of the intersection of two events.
First, let’s formalize the approach for finding the conditional probability. For any two events \(A\) and \(B\), the way we found the probability of \(A\) given \(B\) was to divide the number of outcomes in \(B\) that describe \(A\) by the number of outcomes in \(B\):
\(P(A|B)=\frac{\text{outcomes of B that describe A}}{\text{outcomes of B}}\)
This process can be easily converted into the following formula
\(P(A|B)=\frac{P(A\text{ and }B)}{P(B)}\)
which we call the conditional probability formula.
Contingency tables are the best playground for practicing the conditional probability formula. Let’s recall the following contingency table with events defined as shown.
|
Dropped (R1) |
Transferred (R2) |
Graduated (R3) |
Total |
|
|---|---|---|---|---|
|
0-2 (Y1) |
0.08 |
0.16 |
0.03 |
0.27 |
|
2-4 (Y2) |
0.14 |
0.13 |
0.15 |
0.42 |
|
4-6 (Y3) |
0.10 |
0.04 |
0.06 |
0.20 |
|
6+ (Y4) |
0.08 |
0.02 |
0.01 |
0.11 |
|
Total |
0.40 |
0.35 |
0.25 |
1.00 |
Let’s find the conditional probability of Y3 given R2 using the conditional probability formula where the numerator is the joint probability of Y3 and R2 and the denominator is the marginal probability of R2:
\(P(Y_3|R_2)=\frac{P(Y_3\text{ and }R_2)}{P(R_2)}=\frac{0.04}{0.35}=0.11\)
Let’s find another conditional probability of R1 given Y4 using the conditional probability formula where the numerator is the joint probability of R1 and Y4 and the denominator is the marginal probability of Y4:
\(P(R_1|Y_4)=\frac{P(R_1\text{ and }Y_4)}{P(Y_4)}=\frac{0.08}{0.11}=0.73\)
Consider the conditional probability formula again.
\(P(A|B)=\frac{P(A\text{ and }B)}{P(B)}\)
Let’s multiply both sides by \(P(B)\) and swap the sides of the equation.
\(P(A\text{ and }B)=P(A|B) \cdot P(B)\)
The result is known as the general multiplication rule that allows to find the probability of the intersection of two events.
When A and B are independent, in other words when the probability of \(A\) given \(B\) is the same as the probability of \(A\) in the general multiplication rule, we can replace \(P(A|B)\) with \(P(A)\) and get the following result
\(P(A\text{ and }B)=P(A) \cdot P(B)\)
which is known as the special multiplication rule. What makes it special is the fact that it can only be applied for two independent events.
if A and B are independent and the probability of A is 0.4 and the probability of B is 0.5 then the probability of A and B is 0.2. In this particular example, we can also use the general addition rule and find the probability of the union of two independent events A and B.
Using the special multiplication rule we can verify whether two events are independent or not. For example in the following contingency table consider events Y2 and R3 and let’s check whether they are independent or not?
|
Dropped (R1) |
Transferred (R2) |
Graduated (R3) |
Total |
|
|---|---|---|---|---|
|
0-2 (Y1) |
0.08 |
0.16 |
0.03 |
0.27 |
|
2-4 (Y2) |
0.14 |
0.13 |
0.15 |
0.42 |
|
4-6 (Y3) |
0.10 |
0.04 |
0.06 |
0.20 |
|
6+ (Y4) |
0.08 |
0.02 |
0.01 |
0.11 |
|
Total |
0.40 |
0.35 |
0.25 |
1.00 |
If they were independent then the special multiplication rule will be true that is the
\(P(Y_2\text{ and }R_3)=P(Y_2) \cdot P(R_3)\)
On the left-hand side there is the joint probability 0.15 and on the right hand side there is the product of the marginal probabilities of Y2 and R3. It is easy to verify that the two sides are not equal, therefore the events are dependent!
\(0.15=P(Y_2\text{ and }R_3)\neq P(Y_2)\cdot P(R_3)=0.42\cdot0.25\)
We finally developed a complete set of probability rules that we can use to find the probabilities of the complement of an event, and the union and intersection of any two events.