Probability and statistics
Bayes' rule
Update a belief by multiplying what you thought before by how much better the evidence fits one explanation than the other.
Thomas Bayes, in an essay that Richard Price sent to the Royal Society after Bayes's death; it appeared in the Philosophical Transactions for 1763, printed in 1764. Laplace adopted the result in 1781 and built the general theory.
In words: the probability of a hypothesis after seeing some evidence equals how probable the hypothesis was before, scaled by how much more likely the evidence is if the hypothesis is true than if it is false. The odds form is the one to use by hand. Start with the prior odds, multiply by the likelihood ratio, and you have the posterior odds. The Stanford Encyclopedia of Philosophy puts the content in one line: a hypothesis is confirmed by any evidence that its truth makes more probable. Bayes proved a special case for an unknown proportion; Laplace generalised it, and the rule is now the formal statement of what it means to learn from evidence.
A worked case from a build pipeline. Suppose a real bug is present in 5 percent of commits, a test fails 90 percent of the time when the bug is there, and it also fails 2 percent of the time when the code is fine (it is flaky). Of 1,000 commits, 50 carry the bug and 45 of those fail the test; 950 are fine and 19 of them fail anyway. A single failure therefore means a real bug 45 times in 64, about 70 percent. Rerun it. If reruns are independent, 90 percent of the 45 bugged failures fail again (40.5) while only 2 percent of the 19 flaky ones do (0.38), so two failures in a row point to a bug with probability 99 percent. Each failure multiplied the odds by 45, which is 0.9 divided by 0.02; the base rate decided where you started.
The rule is only as good as its inputs. The likelihood ratio must compare the evidence under both explanations; computing how unlikely the evidence is under one of them and stopping is the prosecutor's fallacy, and it has sent people to prison. Priors are often guesses, and two people with different priors can read the same evidence and move in the same direction to different places, which is fine so long as both keep updating. What the rule never licenses is certainty: a posterior of 99 percent still leaves one case in a hundred, and when that case is expensive the number matters more than the word 'almost'.
Example
At the trial of Sally Clark, a paediatrician told the jury that the chance of two cot deaths in a family like hers was 1 in 73 million, a figure made by squaring a single-death rate. On 23 October 2001 the Royal Statistical Society issued a statement that the figure had no statistical basis, because squaring assumes the two deaths are independent when genetic or environmental factors can make a second case much more likely, and that reading it as the chance the deaths were accidental was the prosecutor's fallacy. What a jury needs is the relative likelihood of the deaths under the two explanations, double cot death or double murder, both of which are rare.
Where it applies
Treat every alert, test result and model score as a likelihood ratio applied to a prior rather than as a verdict. A spam filter, a retrieval reranker and a medical triage model all compute this, and a well-calibrated model is one whose posteriors can be fed into the next decision as priors.
Think about it
Pick a belief you changed this month: what was your prior, what was the evidence, and how much more likely was that evidence if you had been wrong?
Applied it to something real? Log it.
Test yourselfA flaky test fails 2 percent of the time on good code and 90 percent of the time when there is a bug; bugs are in 5 percent of commits. What does one failure tell you?
Per 1,000 commits, 45 failures come from the 50 bugged commits and 19 from the 950 clean ones, so one failure means a bug about 70 percent of the time (odds 45 to 19).
Related models
- Base ratesHow common something is to begin with matters as much as the evidence in front of you, and people routinely leave it out.
- Probabilistic thinkingThink in odds rather than certainties: estimate how likely each outcome is, and judge a decision by those odds rather than by the single result that happened.
- Strong opinions, weakly heldReach a definite conclusion early so you have something to test, then try hard to prove it wrong and let it go when the evidence turns.
- Confirmation biasPeople look for, notice and remember evidence that fits what they already believe, and rarely test what would prove them wrong.