November 7, 2014
Since the General Motors ignition switch debacle blew wide open last spring, the National Highway Traffic Safety Administration has defended its years-long failure to recall the deadly vehicles by arguing that several other vehicle models had more consumer injury-crash complaints related to airbag non-deployment (ABND) than either the 2005-2006 Cobalt or the 2003-2005 Ion. But a new analysis has shown NHTSA is hanging its hat on an unscientific analysis of data that doesn’t support its claim. The study found no statistically significant difference between the other vehicles’ crash complaints and those of the Cobalt or Ion. Combining the Cobalt and Ion complaints, which NHTSA didn’t do, shows the injury complaint rate was actually 54 percent higher for the GM vehicles than their peers. The agency could have performed a statistical analysis itself but chose instead to ignore it in favor of a quick look at the chart.
Trying to explain why it did nothing in the nine years after it first learned of a 2005 fatal accident in which a Cobalt’s front airbag inexplicably did not deploy, NHTSA has repeatedly pointed the finger at GM, saying the automaker didn’t provide enough information to detect a trend. It has based much of this assertion on two analyses of crash-injury complaint rates, in 2007 and 2010, respectively. The 2007 analysis found that 10 non-GM vehicles had more ABND injury complaints than either the Cobalt or Ion. The 2006 Cobalt had 2.03 injury crashes per 100,000 “exposure years,” slightly less than half of the peer 2005 Toyota Echo’s 3.90 crashes per 100,000 exposure years. The 2005 Cobalt had 1.99 injury crashes, and the 2003-2005 Ion had between 1.61 and 1.68 crashes.
In his written testimony before the House Committee on Energy and Commerce in April, NHTSA Acting Administrator David Friedman said that the analysis shows that there was no reason for GM to be on its radar: “The data available at the time of this evaluation did not indicate a safety defect or defect trend that would warrant the agency opening a formal investigation. In particular, the available data did not indicate that the Cobalt or Ion were overrepresented compared to other peer vehicles with respect to injury-crash incident rates.” Friedman further testified that the other vehicles were “significantly higher” in the complaint index.
But NHTSA did not present its underlying data, and the statistical research firm Quality Control Systems Corp. (QCS) got curious: What would the data show if the vehicles were compared to determine the statistical significance of their rankings? After obtaining the data through a Freedom of Information Act request, QCS analyzed it using classical statistical tests to determine if the differences NHTSA saw were, in fact, statistically significant.
QCS found that all of the comparisons fell within the range that showed there was no statistically significant difference between the Cobalts and individual peer models with higher complaint rates. According to its analysis, “the results in Table 2 do not sustain Mr. Friedman’s testimony regarding injury crash airbag non-deployment complaint rates that, ‘…there were several vehicles that were significantly higher,’ insofar as that conclusion does not rest on evidence from an appropriate test of statistical significance.”
Because the sample sizes were relatively small, QCS went a step further and combined the Cobalt and Ion complaints to compare those with all other peer vehicles combined. This result was very statistically significant: The average complaint rate for the combined GM vehicles was 54 percent higher than for the other vehicles.
“NHTSA was impressed by evidence no statistician would think was good, and the one piece of evidence that could have opened the eyes of a statistician was ignored,” said Randy Whitfield, a principle of QCS. “It’s the style of analysis NHTSA has been using for years and years. If they see something that lets them off the hook, where they don’t have to do hard work and they’re not going to have to fight with a manufacturer, if you can give them the least scrap of evidence, they jump on it.”
Even if the statistics weren’t scientifically unsound and overstated, a look at what was going on within NHTSA’s walls when it did the analysis shows that at least some people in the agency wanted an investigation into higher-than-average ABND incidents. The Energy and Commerce Committee’s staff report on the recall found that NHTSA undertook the analysis after the Early Warning Division confirmed 43 crashes—with 27 injuries and four deaths—linked to ABND. The agency had two investigation reports from crashes that mentioned that the engine was in the accessory mode. GM’s warranty claim rate for Cobalt airbags was significantly higher than that of its peer vehicles, and the Cobalt had the most airbag-related property damage claims. GM had also already issued three technical service bulletins for its airbag system.
That division referred the issue to the Defects Assessment Division (DAD), which opened an issue evaluation. DAD’s chief, Gregory Magno, eventually sent an email to the heads of several other NHTSA departments saying, “Notwithstanding GM’s indications that they see no specific problem pattern, DAD perceives a pattern of non-deployments in these vehicles that does not exist in their peers and that their circumstances are such that, in our engineering judgment, merited a deployment, and that such a deployment would have reduced injury levels or saved lives.” Magno was concerned enough to request that NHTSA open an investigation within two weeks.
The Office of Defects Investigation panel waited two months before deciding there wasn’t enough evidence of a defect, based in large part on the complaints analysis. They dismissed the earlier accidents because they were off-road, had multiple impacts, and involved unbelted occupants.
So it wasn’t just that NHTSA didn’t delve closely into what the rankings actually meant—it gave unscientifically analyzed consumer complaints rates more weight than the growing rate of accidents and deaths, the significant number of airbag warranty claims, the GM technical service bulletins, and reports connecting the dots between the ABND and the ignition switch.
Flash forward to the 2010 analysis, and it does appear the number of consumer complaints to NHTSA fell significantly. The QCS evaluation confirmed that complaint rates for the 2005 Cobalt fell by 90 percent, and for the 2006 Cobalt, they fell by 75 percent. In interviews, Friedman said it was because GM secretly changed the ignition switch design in later models. But QCS said that doesn’t explain a radical drop in complaints for older models, which weren’t repaired. It could be that people are much less likely to complain to NHTSA if their car is out of warranty or they are no longer making payments on it. Whitfield said they don’t know why the complaints dropped so much, but it’s a question that is no longer being asked despite no real answer.
“Friedman told Congress and wrote a letter to USA Today saying it was the secret ignition switch change. How could that be when nobody knew about it and it wasn’t recalled? Why should we think we’ve gotten to the bottom of this, when NHTSA’s explanation is unsupported by the evidence in the public record?”
And, again, the numbers don’t tell the whole story: According to the committee staff report, the Early Warning Division noticed that in the second quarter of 2009, GM’s data showed, as the division’s chief said in an email to DAD, “a lot of death and injury incident reported from the Chevy Cobalt and Chevy Trailblazer 360 where the primary component is air bag.” NHTSA may not have been racking up consumer complaints in its database, but it knew people were still dying. And what did it do? Nothing – until it was asked by attorney Lance Cooper to investigate GM’s initial ignition switch recall.
Solutions for NHTSA’s systemic problems need to include more than increased funding – which the agency certainly can use – but putting more money into an agency that hides behind incomplete, unscientific data to defend itself when scandals break is not a sound investment.
A copy of the QCS report can be found here.