Consider the throwing of a many-headed die. If the die has n sides, the throw can result in the showing of exactly one of the sides. This is called an outcome. (In the textbook, CLR, an outcome is called an elementary event.) Each outcome has a probability which is the likelihood of the outcome, expressed as a fraction from 0 to 1. If an outcome never occurs, it is impossible, its probability is 0. If the outcome always occur, it is certain, its probability is 1.
We refer to the throwing of the die as an experiment. Denote that the experiment X resulted in outcome x by the symbol X=x. It is usual that experiments are named with uppercase letters and outcomes with lower case letters. The probability that experiement X results in outcome x is denoted Prob(X=x).
Rule: Since one and exactly one of the outcomes must occur, the probilities must sum to 1.
Prob(X=1) + Prob(X=2) + ... + Prob(X=n) = 1
Example: A coin is a 2-header die and we generally speak of heads H or tails T, rather than 1 or 2. The outcomes heads and tails are equally likely, so,
Prob(X=H) = Prob(X=T) = 1/2
Probability models chance occurances, or other situations of uncertainty. The philosophical underpinnings of chance and belief is complicated, but probability simplifies all issues into a set, called the probability space, subsets of the space, called events, and a measurement function from events to real numbers in the range 0 to 1, called the probability (or measure). The collection of events and the probability function must satisfy some formal properties, called the axioms of probability. As a piece of mathematics, probability theory is certainly true. However, if its correspondence to reality can only be judged empirically: does it give useful results? Can I use probability to play the odds and win.
The collection of all possible outcomes of an experiment is the probability space. In each experiement one and only one outcome occurs. A set of outcomes is called an event. The probability of an event is the sum of the probabilities of all outcomes in the event. It is the likelihood the an experiement will give an outcome that is in the event.
Example: We can list all the possible events of a coin toss. They are,
If F and G are events, then the intersection and union of these sets are also events. The intersection F ∩ G is the event that an experiment has outcome in both F and G; the union F ∪ G is the event that an experiment has outcome in either F or G.
Rule: If F and G are events with no common element, that is, no outcome is both F and G, then the probability of F ∪ G is the sum of the probabilities of F and G individually,
Prob( F ∪ G ) = Prob(F) + Prob(G) ( if F ∩ G = ∅ )If !E is the complement of event E, then Prob(!E ∪ E ) =1, because it is certain that E either happens or it doesn't. Since,
Prob( !E ∪ E ) = Prob(!E) + Prob(E) = 1,then,
Prob(E) = 1 - Prob(!E).
Example: Consider two dice. The probability space is all pairs of numbers (i,j), i and j a number 1 through 6. There are 36 different outcomes, (1,1), (1,2), ..., (6,6). Each is equally likely so,
Prob(X=(1,1)) = Prob(X=(1,2)) = ... = Prob((6,6)) = 1/36.A possible event is "a seven is rolled". This event is the set of outcomes,
E = {(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)}.
The probability of the event is,
Prob(E) = Prob({(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)})
= Prob(X=(1,6)) + Prob(X=(2,5)) + ... + Prob(X=(6,1))
= 6/36 = 1/6.
Exercise: What is the event that two die role a even total? What is the probability of this event?
Experiment: Roll two dice 120 times and count the number of sevens. Is the count close to 20? How could I predict this? How come it was not exactly 20? How could I predict that it would not be exactly 20?
Given two events F and G, the probability of event G might depend on whether F has also occured. For instance, the probability of a die being six is related to the probability that the die rolls and even. Given that a six has been rolled, it is then certain that an even number has been rolled. Given that an even has been rolled, it is now more likely that a six has been rolled.
The conditional probability of even F given that event G has happened is denoted Prob(F|G). It is the likelihood an outcome being in F∩G adjusted for the fact that it is certain that the outcome is something in G,
Prob(F|G) = Prob(F∩G)/Prob(G)
However if Prob(G)=0 then Prob(F|G)=0 as well.
Intuitively, independence of two events is that the likelihood of one event is not changed by the occurance of the other event. We learn nothing about the first having observed the second. So Prob(F|G)=Prob(F). This implies,
Prob(F∩G) = Prob(F) Prob(G) (F and G are independent)This allows for a rule concerning the calculation of probabilities for intersection of events given that the events are independent. It is also the definition of indenpendence. If F and G are events such that Prob(F∩G) = Prob(F) Prob(G) then the events are independent.
Example: What is the probability that two die roll a total of 9 given that the first die rolls a 6? Are the events "first die is a 6" and "total is 9" dependent or independent?
The event E = "total is 9" contains {(3,6), (4,5), (5,4), (6,3)}. Hence Prob(E)=4/36=1/9. Let F be the event "first die rolls a 6". F contains 6 events, {(6,1), (6,2), ..., (6,6)}. Hence Prob(F)=6/36=1/6. Then E∩F={(6,3)} and prob(E∩F)=1/36.
We can now calculate:
Prob(E|F) = Prob(E∩F)/Prob(F) = (1/36)/(1/6) = 1/6.Since Prob(E)<Prof(E|F) the events are dependent. It is less likely to have a total of 9 if it becomes known that the first die rolled a 6.
Exercise: What is the probability that the second of two die rolls a 3 given that the first rolls a 6. Are the events "first die is a 6" and "second die is a 4" dependent or independent?
The definition of indepence is philosophically strange. Intuitively, it should be that F and G are independent if there is no causal connection between them, and dependent otherwise. In this view, dependence of events has a deeper meaning, the presence of a mechanism by which the events affect each other. This may be the scientific understanding, but it is not the mathematical understanding. The mathematical understanding is about predictability, not causality. Even if there is a causal connection between F and G, they are mathematically independent if this connection doesn't change the predictability of one event by the other. Conversely, if observation of one event biases the probability of another event, they are dependent, and it is not necessary to wonder about whatever mechanism causes the dependence.
Example: What is the probability that two die roll an even total given that the first die rolls a 6? Are the events "first die is a 6" and "total is even" dependent or independent? Does the sum "depend" upon the value of the first die?
Call F the even the first die rolls a 6, and G that the total is even. Of 36 possible rolls of both die, 6 are events in F. So Prob(F)=6/26=1/6. There a three ways, each equally likely, that the total is even and the first roll is a six. So Prob(F∩G)=3/36=1/12.
Prob(G|F)=Prob(F∩G)/Prob(F) = (1/12)/(1/6) = 1/2The Prob(G) is the number of ways to roll an even divided by the 36 different possible rolls. Doing the counting, we have Prob(G)=1/2. So Prob(G|F)=Prob(G).
The events F and G are independent, even though the total does certainly "depend" on the outcome of the first roll.
If there is a dependence between two events, F and G, it is often easier to calculate the probability of F assuming G than the probablity of G or of F∩G. In that case, the law of conditional probabilty can be used in this form,
Prob(G|F) Prob (F) = Prob( F∩G )