10.8Luck and skill

Are they good, or just lucky?. Every result mixes skill and luck. The best results are partly the best luck, and luck does not repeat, so the stars of one season tend to land nearer the middle the next. Mistake that for cause and effect and you will hire, promote, praise and scold for the wrong reasons. Turn the dial and watch it happen to people who never changed.

Runs in your browser · the players are simulated · settings and seeds are saved in the link, so a shared link replays the same seasons

1Read
Michael Mauboussin, an investment strategist who writes about skill and luck, places every activity on a line “that has all skill and no luck on one side, and no skill and all luck on the other.” Chess and running races sit near the skill end, roulette and the lottery at the luck end, and most of life in between.
He also shows how to measure where an activity sits. Work out how much results would vary if every game were a coin toss, and compare it with how much they actually vary. In the 2009 to 2010 National Basketball Association (NBA) season, luck accounted for about 11% of the spread in teams' win records. For American football teams, he reports Brian Burke's estimate that it is more than half. The model below uses the same measure: the dial is luck's share of the spread in results.
2Set
A league of 100 players. Each has a hidden skill, fixed for good. Each season their result is part skill and part luck, and the luck is drawn fresh every season. Set how much of the spread in results comes from luck.
The mix: 50% luck
Skill's share
50%
Luck's share
50%
How alike two seasons are
0.5
The last readout is the correlation between one season's results and the next: 1 means the same order every time, 0 means no relation.
3Run
Season one. These are the ten best results, the ones a scout, a newspaper or a fund ranking would notice. The last column is what nobody watching can see: where each player's hidden skill ranks.
League 1
Season one, top 10, at 50% luck
Season one Player Skill rank
1 Player 82 6
2 Player 80 26
3 Player 37 2
4 Player 23 9
5 Player 27 23
6 Player 89 8
7 Player 88 10
8 Player 41 4
9 Player 77 12
10 Player 68 38
Of the top 10, truly top 10 by skill
6

Season one, top 10, at 50% luck
Season one	Player	Skill rank
1	Player 82	6
2	Player 80	26
3	Player 37	2
4	Player 23	9
5	Player 27	23
6	Player 89	8
7	Player 88	10
8	Player 41	4
9	Player 77	12
10	Player 68	38

4Rerun

Season two: same players, same skill, new luck. Each line joins a player's rank in season one (left) to season two (right). Last season's top 10 are drawn in ink.

Last season's top 10, average rank now: 34
Still in the top 10: 1 of 10
The lowest of them now: 82

Last season's top 10, and where they finished this season
Player	Season one	Season two
Player 82	1	19
Player 80	2	55
Player 37	3	14
Player 23	4	30
Player 27	5	82
Player 89	6	28
Player 88	7	7
Player 41	8	41
Player 77	9	27
Player 68	10	37

Last season's top 10, average rank next season, for every mix (averaged over 60 leagues)

Show as a table

Average rank next season of last season's top 10, by luck share
Mix	Average rank next season
pure skill	5.5
10% luck	9.1
20% luck	12.6
30% luck	16.8
40% luck	21.6
50% luck	26.5
60% luck	30.6
70% luck	35
80% luck	40
90% luck	45.1
pure luck	51.2

Note

Nobody got worse. Skill did not change between seasons; only the luck was redrawn. Last season's top 10 were, on average, both good and lucky, and the luck does not come back. At 50% luck, across many leagues they average rank 26.5 next season, against 5.5 if nothing moved and 50.5 for a pick at random.

Michael Mauboussin, who studies skill and luck in sport and investing, puts it this way: “the outcomes of activities laden with luck revert to the mean faster than activities with little luck.”

5Flip
Now set skill to zero. 100 people each call 20 coin tosses, heads or tails, like 20 years of beating the market or not. Here are the calls of whoever had the longest run of right answers.
How many callers
Player 83's 20 calls
21. 8 in a row
Filled: a right call. Open: a wrong one.
Longest streak anyone had
8
Typical caller's longest
3
Skill involved
None
Note
Nobody here has any skill: every call is a fair coin. Yet out of 100 callers, Player 83 got 8 in a row, while the typical caller's best run was 3. Look only at the winner and it looks like a gift. Look at the whole group and it is what chance produces when enough people try. The bigger the group, the longer the streak someone will have.
6Judge
A manager watches this league for 10 rounds. After each one they praise the top 10 and scold the bottom 10. In this model the feedback has no effect at all: results are still hidden skill plus fresh luck, at 50% luck (the dial in step 2).
Praised: average rank before, then after
5.5 → 31.9
Scolded: average rank before, then after
95.5 → 75.6
What does the manager conclude?
Note
Daniel Kahneman met this manager while giving a talk to flight instructors. One of the most seasoned of them told him: “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time.”
The instructor saw something real and explained it wrongly. As Kahneman wrote in 2002, “it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them.”
7Go
Before you credit or blame anyone for a change, ask how much of the last result was luck. Read on.

10.8.1How the model works

Each of the 100 players gets a hidden skill, drawn once from a bell curve. Each season their result is √(1 − L) × skill + √L × luck, where luck is a new draw from the same bell curve every season and L is the dial. The square roots make L exactly luck's share of the spread in results, Mauboussin's measure, and they make the correlation between two seasons 1 − L. A player who finished well above average is expected to finish (1 − L) times as far above it next time.

In the streak step nobody has skill: each call is right with chance one half. In the last step the manager's praise and scolding are not in the model at all, which is the point. Every draw comes from a seeded random number generator, so the seed in the link replays the same league.

It is a toy. Real skill changes over time, players learn and age, and luck is rarely a neat bell curve. The shape it shows (extreme results drift back, faster the more luck is involved) holds whenever results are partly noise.

10.8Luck and skill

10.8.1How the model works

10.8.2Sources