Open the Pod Bay Doors · Episode 6 of 10 · 23 min
Therac-25, MCAS and the Machines Already Overruling Humans
Long before anyone talked about alignment, engineers decided which conflicts a machine would win. The episode looks at the Therac-25 radiation overdoses, Airbus protections that saved one flight and failed on another, the 737 MAX's MCAS, and a drug alert stopped early in a trial. It contrasts them with the host's own neonatal nutrition software, whose limits had a named author.
Who's who
- Nancy LevesonCo-author of the authoritative Therac-25 account
- Clark S. TurnerCo-author of the Therac-25 account
- Brian L. StromLed the 2010 Pennsylvania trial of a drug-interaction alert
Where it happens
- Hudson River, New YorkUS Airways 1549 ditched here on January 15, 2009
- Marietta, GeorgiaSite of the first Therac-25 overdose in June 1985
- Tyler, TexasTwo Therac-25 overdoses in 1986
- Philadelphia, Pennsylvania+6 more episodesHospital where the drug-alert trial was stopped early
Timeline
- June 1985First Therac-25 overdose, at Kennestone in Marietta, Georgia
- January 15, 2009US Airways 1549 ditches in the Hudson with all 155 aboard surviving
- June 1, 2009Air France 447 is lost with 228 people
- 2010Strom trial of a near hard-stop drug alert is published, stopped early by an ethics board
- October 29, 2018Lion Air 610 is lost, 189 dead
- March 10, 2019Ethiopian Airlines 302 is lost, 157 dead
Still debated
Whether designed authority and machine refusal differ in kind or only in origin is argued, as is whether hard stops in medicine save more than they cost. The Airbus-versus-Boeing comparison is also contested, since each design has its own signature failure.