IIHS: Self-driving systems aren't that, and aren't ready for prime time

Self-driving systems aren't ready for prime time, says a new report by IIHS.

The Institute tested five cars: two Teslas, a BMW, a Mercedes-Benz, and a Volvo, all with systems that earned top marks in its tests of advanced driver safety aids.

In new tests on its track and on public roads, all of the systems made mistakes that could be fatal without driver intervention.

READ THIS: Tesla will roll out new Autopilot later this month

Many of those mistakes included drifting over lane lines even when active lane control (or Autopilot, in Tesla's terminology) was activated.

"Self-driving" seems to mean different things to drivers and to safety experts. SAE International and the NHTSA have laid out guidelines for five different levels of self driving cars. The systems that IIHS tested are considered Level 2 of the five levels.

Joe Young, an IIHS spokesman who works at the facility's test center and has driven these cars on nearby roads says, "I had a hard time with these systems. I did not feel at all comfortable with how these systems were driving going over crests and around curves, especially with oncoming traffic on a right curve," he told Green Car Reports in an interview. 

Tesla Model 3 dashboard in Autopilot testing with IIHS [CREDIT: IIHS]

In addition to automatic emergency braking and adaptive cruise-control tests on its test track, the IIHS tested each cars' active lane control systems six times going around three curves and over the tops of three hills—the types of situations where the systems have shown the most difficulty tracking lane markings—on public roads.

It tested two Teslas—a 2018 Model 3 running Autopilot version 8.1 software, and a 2016 Model S running Autopilot version 7.1 software—a 2017 BMW 5-Series, a 2017 Mercedes-Benz E-Class, and a 2018 Volvo S90, each with its automaker's Level 2 self-driving feature. 

The tests were part of an ongoing effort at the IIHS  to develop ratings for the systems that would allow car buyers to see which systems are better than others.

"We're not ready to say yet which company has the safest implementation of Level 2 driver assistance" says IIHS chief research officer David Zuby. "None of these vehicles is capable of driving safely on its own."

It is possible to draw a few anecdotal conclusions from the tests, though.

Active lane control not always active

The Tesla Model 3 with Autopilot version 8.1 stood out as the only car that consistently kept the car within its lane through all 18 curves, though it touched the center line once when going over a hill. The older Tesla Model S, with Autopilot version 7.1, crossed or touched a line 13 times going over hills and once on a curve. The Teslas were the only cars that never disengaged the self-driving systems in the six laps.

IIHS test results of lane-keeping assist systems [CREDIT: IIHS]

The BMW 5-Series was by far the most likely to disengage the system. When it couldn't track lane markings well enough going over a hill, it simply shut down the system and forced the driver to intervene. This happened almost half the time. Notably, in the other half the time, the BMW was at least as likely to touch or cross a lane marking as not. 

The other cars, including the Model S, negotiated the hills and curves successfully between half and two-thirds of the time.

All the cars showed a tendency at low speeds to follow a leading car out of the lane if the lead car changed lanes or turned off, the report notes.

Automatic braking

On its track, the IIHS tested automatic emergency braking by itself and under adaptive cruise control by driving the cars toward a fixed obstacle at 31 mph, and also by following another car in front that braked smoothly, braked abruptly, and moved out of the way just before a stationary obstacle, to see whether the following car could then "see" and brake in time for the obstacle.

Tesla Autopilot

The other way in which both Teslas distinguished themselves was by being the only cars in the test to hit the stationary obstacle when active cruise control (Autopilot) was disengaged. All the other cars' automatic emergency braking systems stopped in time.

With Autopilot engaged and adaptive cruise control active, both Teslas slowed more gradually and farther ahead and avoided the obstacle.

The Model 3 notably was also the only car that IIHS said braked unnecessarily and overly cautiously in traffic, though it said the behavior was not dangerous because it did so gradually. The Model 3 slowed down unexpectedly 12 times in 180 miles, seven of which were for tree shadows, the IIHS report says.

Tesla plans to rolling out a new version of Autopilot, version 9, later this month.

The other notable standout in braking performance was the Volvo S90, which braked abruptly in almost all the automated braking tests, braking late at 1.1 or 1.2 gs, even under adaptive cruise control. The other cars all braked earlier and at a much more comfortable rate.

Safety effects

Even with the mistakes that Autopilot 7.1 made in the testing, the IIHS noted in a separate report that the forward-collision warning system is lowering third-party property damage and injury claims, and that Autopilot has lowered collision claims in the Model S.

Automatic emergency braking systems are not designed to see far enough ahead or react quickly enough to prevent or make much difference in the types of head-on crashes that are often fatal, says IIHS spokesman Russ Rader.

"Automatic emergency braking systems are currently designed in a way that it will mainly affect front-into-rear crashes, and those are not usually fatal, but they do produce a lot of injuries," he told Green Car Reports.

READ THIS: Let's be clear: Tesla's Autopilot is not a 'self-driving car'

The owners' manuals for each of the five cars the IIHS tested also say that their adaptive cruise control systems may not brake for stationary objects once they come into sensor range, the report notes. The report cautions, though, that many owners may not read such warnings buried in owners' manuals.

In the tests, the BMW, Mercedes, and both Teslas did brake for the stationary object at 31 mph, though it's not clear whether they would have done so at higher speeds.

The IIHS has estimated that if all cars were equipped with active lane control that worked perfectly, the technology could save 8,000 lives per year.