RAND study demonstrates difficulties of using test driving to prove reliability of self-driving cars.

A recent report by the RAND Corporation suggests that self-driving cars may require hundreds of years of test driving to proven safe.
RAND researchers analysed the ability to demonstrate the failure rate of autonomous vehicles to a certain confidence level using the number of miles driven without a failure and a statistical analysis based on binomial distribution.
Test-Driving Autonomous Cars for Safety
In order to demonstrate a failure rate of 1.09 fatalities per 100 million miles with 95 percent confidence, the researchers found that autonomous vehicles would need to be test-driven for 275 million miles without failure. That’s equivalent to a fleet of 100 self-driving cars being driven continuously for 12.5 years.

A second analysis sought to discern the failure rate to a 20 percent error tolerance within a two-sided confidence interval of 95 percent on a standard distribution. This analysis found that autonomous vehicles would need to drive over 8.8 billion miles—the equivalent of 100 self-driving cars being test-driven continuously for 400 years.

Comparing Self-Driving Cars with Human Drivers
Practically, however, the benchmark for autonomous vehicle safety is set relative to the safety of human drivers. The researchers found that 500 billion miles of test-driving, or 225 years, would be required to demonstrate with 95 percent confidence that the failure rate of autonomous vehicles is 20 percent lower than an assumed benchmark of 1.09 fatalities per 100 million miles.

In order to prove this with a power of 80 percent—that is, an 80 percent probability of rejecting the null hypothesis—a fleet of 100 autonomous cars would need to be continuously test-driven for 11 billion miles, or 518 years.
Can Self-Driving Cars Be Proven Safe?
It’s worth noting that these estimates are optimistic, since they assume that the performance of human drivers is a known and constant value, rather than a statistical conundrum of its own—which is actually the case. The technology for self-driving vehicles is also still evolving, creating a similarly moving target.
Additionally, 95 percent confidence is much lower than level typically required by industry and policy markers, which is upwards of 99.9 percent. All things considered, the findings of the report indicate that it is prohibitively impractical to attempt to demonstrate the reliability of autonomous vehicles through test driving alone.
These results suggest that the need for new approaches to testing, such as simulations, mathematical modelling and human behavioral testing—each of which have their own benefits and challenges. They also indicate the need for new policies from all parties affected by the implementation of these vehicles as well as formal definitions of what constitutes a “safe” self-driving car.
Still, as RAND justice policy distributor James M. Anderson writes, “promise outweighs peril.” The National Highway Traffic Safety Administration reports that 90 percent of crashes are caused by human error, including drunk, distracted, aggressive and impaired driving. These are problems that self-driving cars will never have. Autonomous vehicles have the potential to significantly improve road safety.
We just need to prove it.
The full RAND report is available here.