One of the tools Elon Musk and Tesla Motors have used to defend the safety of its Autopilot software is the dark and perplexing art of statistics.
After a fatal Autopilot-related crash in May, a company blog post pointed out that the crash was “the first known fatality in just over 130 million miles where Autopilot was activated.”
The post then noted that “Among all vehicles, in the US, there is a fatality every 94 million miles.”
The clear implication: driving with Autopilot activated is 38 percent safer than non-Autopilot driving.
In July, Musk sent an e-mail to a Fortune magazine reporter about Autopilot safety, and he didn’t just imply that Autopilot saves lives. He explicitly stated it.
Wrote Musk, “Indeed, if anyone bothered to do the math (obviously you did not) they would realize that of the over 1M auto deaths per year worldwide, approximately half a million people would have been saved if the Tesla Autopilot was universally available."
Tesla Motors CEO Elon Musk at Tesla Store opening in Westfield Mall, London, Oct 2013
"Please take five minutes to do the bloody math….” he ended.
Unfortunately, it takes a lot more than five minutes to sort out the statistics of Autopilot safety.
Let’s take a look at some of the complexities that undermine Tesla’s simplistic approach. And yes, it will be bloody.
Last week, by the way, Musk tweeted “Autopilot miles now at 222 million.”
The first—and probably biggest—flaw in Tesla’s Autopilot-is-statistically safer claim is the sample size.
A general principle of statistics says that the larger the sample size, the more reliable the statistic.
Cumulative Tesla Autopilot miles now at 222 million— Elon Musk (@elonmusk) October 7, 2016
We’re talking here about the smallest possible sample size: one fatality.
Consider what would happen to Autopilot’s fatality rate if there happened to be a second Autopilot fatality.
The Autopilot fatality rate would double overnight, and those half a million lucky folks around the world allegedly saved by universal Autopilot would suddenly all be dead again.
In fact, back in January there was a fatal Autopilot crash in China that had not yet come to light when Tesla and Musk made their statistical claims.
Counting the Chinese crash, and using Musk’s latest figure for Autopilot miles driven, the Autopilot fatality rate is now one per 111 million miles—only a smidgen better than the overall U.S. number.
2014 Tesla Model S in China
And, just blue-skying here, what if the car that crashed in Florida had been carrying three passengers? The Autopilot fatality rate would now be one per 44 million miles—more than double that of non-Autopilot U.S. driving.
Based on that number, critics might well call for Autopilot to be banned immediately in the interests of public safety.
Obviously, such extrapolations and conclusions are nonsense.
Billions of miles
A Rand Corporation study last April concluded that “Autonomous vehicles would have to be driven hundreds of millions of miles, and sometimes hundreds of billions of miles, to demonstrate their reliability in terms of of fatalities and injuries.”
Tesla Model S owner talks about some do's and don't with the Autopilot system
The report continued, “…for fatalities and injuries, test-driving alone cannot provide sufficient evidence for demonstrating autonomous vehicle safety and reliability.”
Bottom line: it will be a long time before autonomous vehicles, Autopilot included, accumulate a sample size big enough to prove they’re safer in a statistically valid way.
Apples vs oranges
Sample size notwithstanding, Tesla’s statistical claims also suffer from the old apples-vs-oranges conundrum.
The NHTSA number that Musk presumably used to derive his one-fatality-every-94 million-mile benchmark is the Fatality Rate per 100 Million VMT (Vehicle Miles Traveled).
For the last few years, that number has hovered a bit above 1.00, which translates to a miles-per-fatality number a bit under 100 million.