8 Probabilistic AI agents
\(\DeclarePairedDelimiter{\set}{\{}{\}}\) \(\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}\)
In most engineering and data-science problems we don’t know the truth or falsity of outcomes and hypotheses that interest us. But this doesn’t mean that nothing can be said or done in such situations. Now we shall finally see how to draw uncertain inferences, that is, how to calculate the probability of something that interests us, given particular data, information, and assumptions.
So far we have used the term “probability” somewhat informally and intuitively. It’s time to make it more precise and to emphasize some of its most important aspects, especially for Artificial Intelligence. Then we’ll dive into the rules of probability-inference.
8.1 When truth isn’t known: beliefs and probability
If we want to design an AI agent that can make decisions and act in uncertain situations, we need to equip it with probabilities and utilities. This is an inescapable necessity, as discussed in § 2.5. It is intuitively clear what utilities are: they quantify how desirable different outcomes are to the agent. What do probabilities quantify?
A first tentative answer could be this: the probability of an outcome quantifies how often the agent has seen that outcome. That is, it represents the observed frequency of that outcome.
Although it seems reasonable in many respects, this tentative answer still doesn’t capture several elements that appear in the way we make rational inferences, and that are extremely important in designing an AI agent that can operate in uncertain conditions.
To see what’s missing, consider the following scenario:
Someone shows you and a friend of yours, called Aisha, a mechanical device designed to toss a coin. The device looks complicated; neither of you has a full grasp of its design. A normal coin is placed on the device, and your friend Aisha is required to place a bet on whether the tossed coin will land
headsortails. If she guesses correctly she wins $1; nothing otherwise. Note that not-betting is not an option: feel free to imagine circumstances (possibly very nasty) where your friend cannot refuse to bet. Let’s also make clear that no cheating is taking place: for instance, the outcome does not depend on the betting choice (you can imagine that the coin is tossed right before Aisha bets, but the outcome hidden from her until she bets).From now on you can only observe Aisha’s choices and hear what she says, but you cannot talk with her. In this situation your friend says that it does not matter whether she bets on
headsortails; she has no more belief in one outcome than the other. In fact, she decides by tossing a coin she had in her pocket.
Do you think Aisha’s judgement and choice are rational?
Aisha’s conclusion comes not only from the fact that the win is the same in either outcome, but also from the fact that her beliefs in the heads outcome and in the tails outcome are equal. If one belief were stronger than the other, she would definitely bet on that outcome instead. And from this initial scenario we gather an important insight:
This equality of beliefs does not come from an equality of frequencies. In fact neither you or your friend have observed any frequencies whatsoever! You have never seen a similar tossing device in operation before. For all you know, the device might be designed to always produce heads, or always tails, or maybe one or the other with peculiar frequencies not equal to 50%/50%.
Let’s continue:
Now you and your friend observe the toss outcome. It’s
tails. The device is prepared again, exactly as in the first toss, and Aisha is required to bet again. However, now they propose Aisha two different bets: bettingtailsand guessing correctly, she wins $1; bettingheadsand guessing correctly, she wins $2. (And remember: the outcome won’t depend on her bet; no cheating is taking place).Aisha says the following: If she had been proposed a $1-win on either
headsortails, then she would have bet ontails, just because the device had showntailsonce. But she still has no much more belief intailsthanheads; so given the double win onheads, she now betsheads.
Do you think Aisha’s judgement and choice are rational?
Clearly Aisha’s beliefs in the two possible outcomes are almost equal; and this judgement is not irrational. So we gather another important insight:
This near-equality, 50%/50%, of beliefs is different from the observed frequency: the frequency right now is 100% tails. The frequency does affect Aisha’s belief a little (she’d choose tails if offered an equal bet), but is different from her belief.
Fast-forward in time:
The device is repeatedly used, say for 1000 times (plus some extra times until you or your friend says “stop”, if you like). In these repetitions you and Aisha count that
headsoccurred 781 times, andtails219 times, without any recognizable pattern. Now Aisha is asked again to place a bet on the next toss.Aisha says that she believes more strongly that the coin will land
headsthantails. She even quantifies her belief to around 78% forheadsand 22% fortails. It’s very close or identical to the frequency you both observed. Aisha bets accordingly.
Do you think Aisha’s judgement and choice are rational?
Now we can say that her beliefs and the observed frequencies are aligned. This is another insight:
When frequencies are available, and come from many repetitions, then beliefs and frequencies tend to align.
Final part of this scenario:
Finally, another friend of yours, called Aiden, is brought in, and shown the device. Aiden cannot communicate with you or Aisha; he’s in the same situation she and you were around 1000 tosses ago. Aiden is required to place a bet, and he says that he has no preferences for betting on
headsortails– with exactly the same reasoning Aisha did 1000 tosses ago.
Do you think Aiden’s judgement and choice are rational?
They must be, as Aiden is now in the same position, situation, and state of knowledge as Aisha was before. We gather another insight:
Two agents may have different beliefs about the same situation; and yet both beliefs may be rational at the same time.
If you think that Aisha’s and Aiden’s behaviour and choices were rational, consider a situation in which two AI agents were in their place. It is apparent that we need to equip an AI agent with a sort of quantified, rational “degree of belief” in order for it to take rational action in uncertain situations. This is also the result discussed in § 2.5. This degree of belief can have connections with an observed frequency, but is generally different from it. The two notions must therefore be kept separate, and quantified separately. In fact, we may think of situations in which an agent may be uncertain about a frequency and needs to quantify its own belief about the frequency; we’ll meet these situations in the Inference III part.
What we call probability is this quantified degree of belief:
We shall take the notion of degree of belief as intuitively understood, just as we did with the notion of truth. We shall use the terms probability, degree of belief, belief, plausibility, credibility1 as synonyms.
1 credibility literally means “believability” (from Latin credo = I believe).
Probabilities are quantified between \(0\) and \(1\), or equivalently between \(0\%\) and \(100\%\). Assigning to a sentence a probability 1 is the same as saying that it is true; and a probability 0, that it is false. A probability of 0.5 represents a belief completely symmetric with respect to truth and falsity.
Said otherwise, if an agent assigns to a sentence a probability 1, it means that the agent is completely certain that the sentence is true. If the agent assigns a probability 0, it means that the agent is completely certain that the sentence is false. If the agent assigns a probability 0.5, it means that the agent is equally uncertain about the truth as about the falsity of the sentence.
8.2 Important aspects of probabilities
From our discussion of the tossing-device scenario above we can gather several important aspects and facts about probabilities:
Probabilities are not frequencies
This fact is clear by now. Indeed:
- there can be a degree of belief even when no frequencies are available;
- a degree of belief can be very different from an observed frequency;
- a degree of belief can be equal to an observed frequency.
One remarkable feature of the framework that we’re going to study is that the quantitative connection between these two notions is taken care of automatically! We shall see how this connection works in the Inference III part.
Note also the different status of the notions “frequency” and “probability/belief” from an epistemological point of view.2 Frequencies can be unknown to some agents. Degrees of belief cannot be “unknown”: the agent must have them in order to act. At worst, degrees of belief can be difficult to calculate.
2 That is, from the point of view of an agent’s knowledge.
Probabilities are agent- and knowledge-dependent
The tossing-device scenario shows that different agents can have different probabilities, that is, degrees of belief, about the same situation.
This happened when Aiden entered the scene. Aisha had beliefs around 78% for heads and 22% for tails; but Aiden had 50%/50% beliefs for the same toss. Yet both sets of beliefs were rational and appropriate to their respective situations. They simply reflected the different states of knowledge of the agents that held them.
An omniscient agent would know the truth or falsity of every sentence, and assign only probabilities 0 or 1. Some literature speaks of “actual (but unknown) probabilities”. If there were “actual” probabilities, they would be all 0 or 1, and it would be pointless to speak about probabilities at all – every inference would be a truth-inference.
Probabilities are not physical properties
The fact that two agents can hold different probabilities in the same situation also shows that probabilities are not physical properties, which could be objectively measured with some meter.
Whether a tossed coin lands heads or tails is fully determined by the initial conditions (position, orientation, momentum, rotational momentum) of the toss and the boundary conditions (air velocity and pressure) during the flight. The same is true for all macroscopic engineering phenomena (even quantum phenomena have never been proved to be non-deterministic, and there are deterministic and experimentally consistent mathematical representations of quantum theory). So we cannot measure a probability using some physical apparatus.
We can objectively measure frequencies, in several instances of a phenomenon. Frequencies, as opposed to probabilities, are physically measurable quantities. This shows again the difference between probabilities and frequencies.
Probabilities are assigned to sentences
We already discussed this point in § 6.3, but let’s reiterate it. Consider an engineer working on a problem of electric-power distribution in a specific geographical region. At a given moment the engineer may believe with \(75\%\) probability that the measured average power output in the next hour will be 100 MW. The \(75\%\) probability is assigned not to the quantity “100 MW”, but to the sentence
\[ \textsf{\small`The measured average power output in the next hour will be 100\,MW'} \]
This difference is extremely important. Consider the alternative sentence
\[ \textsf{\small`The average power output in the next hour will be \emph{set} to 100\,MW'} \]
the numerical quantity is the same, but the meaning is very different. The probability can therefore be very different. If the engineer is the person who decides how to set that output, and has decided to set it to 100 MW, then the probability is obviously \(100\%\) (or very close to), because the engineer already knows what the output will be. The probability depends not only on a number, but on what it’s being done with that number: measuring, setting, third-party reporting, and so on. Often we write simply “\(O \mathclose{}\mathord{\nonscript\mkern 0mu\textrm{\small=}\nonscript\mkern 0mu}\mathopen{}10\,\mathrm{W}\)”, provided that the full sentence behind this shorthand is understood.
The points listed above are not just a matter of principle. They have important practical consequences. A data scientist who is not attentive to the source of the data (measured? set? reported, and so maybe less trustworthy?), or who does not carefully assess the context of a probability, or who mixes a probability with a frequency, or who does not take advantage (when possible) of the physics involved in the a problem – such data scientist will design systems with sub-optimal performance3 – or even cause deaths.
3 This fact can be mathematically proven.
8.3 The many uses of the word “probability”
The terms “degree of belief” and “frequency” are quite distinct and are used more or less consistently in the literature. Whenever you encounter these terms you more or less know what’s intended.
Sadly the situation is completely different with the term “probability”, which is used in the literature in incompatible ways. Some literature uses this term as a synonym of “frequency”. Other literature uses it as a synonym of “degree of belief”, as we do; use of “probability” as “degree of belief” is called Bayesian probability theory.
Some recent literature in machine learning uses “probability” in yet another way, to denote the numeric output of some machine-learning algorithms. This numeric output is neither a frequency or a degree of belief, and only has vague associations with them; we’ll discuss this in ch. 42.
There are also a couple more different uses of the term “probability” in the literature. It’s a mess.
In this course we could have stuck to the terms “degree of belief” and “frequency” only, avoiding the problematic “probability”. But such a choice would not be very helpful to you, because you will nevertheless encounter this term in your readings and scientific discussions. In the AI literature the most common use is as “degree of belief”, so we also adopt it, sometimes using “(degree of) belief” and sometimes “probability”, interchangeably.
But beware of this term when you read the literature, or in your scientific discussions. You must try to understand the intended meaning from the context. You’re also free to choose (preferably consistently) the terminology you like most. What’s important is that the notions underlying these words are clear to you.