Skip to content
Rabin Poudyal
Page 6-1
6-1 Models

6.9Automation bias

Psychology and biases

People tend to do what an automated system tells them and to miss what it fails to tell them, even when other evidence in front of them says otherwise.

Kathleen Mosier, Linda Skitka and colleagues, in studies of airline pilots in the 1990s; Raja Parasuraman and Victor Riley's review Humans and Automation: Use, Misuse, Disuse, Abuse (Human Factors, 1997) placed it in a wider account of over-reliance on automation.

  1. 1Read

    Mosier and her colleagues described automation bias as the errors that follow when people use an automated cue as a shortcut in place of looking for and weighing the information themselves. It comes in two forms. An error of commission is doing what the aid says when other evidence says it is wrong. An error of omission is missing a problem because the aid did not flag it. In their simulator studies, pilots who felt accountable for their performance checked the automation against other cues more often and made fewer errors. The prediction is uncomfortable: the more reliable an aid usually is, the less people check it, and the bigger the surprise on the day it is wrong.

    It is a reasonable habit gone too far. Checking takes effort, and an aid that has been right a hundred times teaches people to stop. Parasuraman and Riley call over-reliance misuse, and list what drives it: workload, how reliable and consistent the automation is, and how visible its state is. They add a fourth term, abuse: designers and managers automating a job "without due regard for the consequences for human performance", which leaves the operator as a by-product of the automation and itself breeds misuse. Language models sharpen the problem, because their output reads as fluent and confident whether it is right or wrong, so the surface gives no hint that it needs checking.

    The opposite failure is real too. Parasuraman and Riley call it disuse: people ignore an aid after it raises too many false alarms. Automation is often more accurate than the person beside it, so the cure is not distrust but checks built into the work. Make the automation's state and confidence visible, send high-stakes decisions to a person who is actually accountable for them, and measure how often reviewers overrule the machine. A reviewer who never overrules it is not reviewing, and a human in the loop who signs everything adds a signature, not a check.

  2. 2Note

    Example

    On 9 June 1995 the cruise ship Royal Majesty left Bermuda for Boston with 1,509 people on board. Within an hour the cable to its GPS (Global Positioning System) antenna came apart. The receiver switched to dead reckoning and showed the small codes SOL and DR, with an audible alarm that lasted one second, and the autopilot went on steering by its position. For more than 34 hours none of the three watch officers or the master noticed. They did not check the GPS against the ship's Loran-C receiver; the chief officer took a radar echo for the buoy marking the Boston traffic lanes, because the GPS said the ship was on track; and when a lookout reported blue and white water dead ahead, the second officer took no action. On the evening of 10 June the ship grounded on Rose and Crown Shoal, about 17 miles from where the officers thought it was. No one was hurt, and damage and lost revenue came to about 7 million dollars. The National Transportation Safety Board (NTSB) found the probable cause to be the watch officers' overreliance on the automated features of the integrated bridge system.

  3. 3Apply

    Where it applies

    When an AI system drafts work that goes out under your name, such as support replies, code or summaries, decide which kinds of output a person must check before they are sent, and make that person accountable for the result. Label machine-written text so its readers know to weigh it. Track how often reviewers change what the machine produced; if the answer is almost never, test them with a known-bad draft. Review an AI-generated diff the way you would review a stranger's.

  4. 4Think

    Think about it

    Which automated output do you accept without checking because it is usually right, and when did you last compare it with the raw evidence?

  5. 5Log

    Applied it to something real? Log it.

  6. 6Check
    Test yourselfWhat are the two kinds of error that make up automation bias?

    Commission errors, doing what the automation says even when other evidence contradicts it, and omission errors, missing a problem because the automation did not flag it.

  7. 7Go

    Related models

    • Confirmation biaspage 6-21

      People look for, notice and remember evidence that fits what they already believe, and rarely test what would prove them wrong.

    • Anchoringpage 6-7

      Any number you see before estimating pulls your estimate towards it, even when you know the number is random.

    • The map is not the territorypage 6-98

      Every model, metric or plan is a simplification, and the parts it leaves out can matter more than the parts it shows.

    • Swiss cheese modelpage 6-86

      Accidents happen when the holes in several layers of defence line up at once, so look for the holes, not the last person to touch the system.

  8. 8See

    Seen in an incident

  9. 9Source

    Sources and further reading