Materials Science Facing Replicability Crisis Now As Well — What Are The Implications For Battery Research?

Sign up for daily news updates from CleanTechnica on email. Or follow us on Google News!

It’s been an open secret for quite some time now that much of what goes on in certain parts of the “scientific community” is at this point essentially fraud. It has been labelled the so-called “replicability crisis” and is facing biomedical and social sciences (amongst others). While the meta-science report throwing psychology under the bus was arguably riddled with its own scientific errors (ironically), well known problems that cross disciplines have been summarized well here.

To get to the point, though, a decent portion of the findings of recent decades in some fields can’t be independently replicated/verified — with the matter of how much this is due to simple incompetence versus varying degrees of fraud being an open question.

Some of those who work in the “hard sciences” and are reading this may at this point be scoffing, and saying that their work has nothing to do with that of the “soft sciences,” and that drawing parallels here is dishonest, but new research suggests otherwise.

A new study from the Georgia Institute of Technology comparing the results reported in many thousands of materials science papers — specifically, those concerned with the properties of metal organic framework (MOF) materials — has found that around 1 in 5 are “outliers.” Which means that they show results “far beyond the error bars normally used to evaluate study results.”

To take that further, the study found that out of all of the thousands of papers looked at, only 9 MOF compounds were identified “for which 4 or more independent studies allowed appropriate comparison of results.”

That certainly makes it sound as though there is likely quite a lot of fluff there.

“At a fundamental level, I think people in materials chemistry feel that things are reproducible and that they can count on the results of a single study,” stated David Sholl, a professor and John F Brock III School Chair in the Georgia Tech School of Chemical and Biomolecular Engineering. “But what we found is that if you pull out any experiment at random, there’s a 1 in 5 chance that the results are completely wrong — not just slightly off, but not even close.”

Sholl notes, though, that it’s not clear yet if this reality is broadly true across the whole of materials science research. But the findings are certainly enough to make one think.

Particularly with regard to materials science related to battery technology development, the number of “breakthroughs” that have occurred over the last decade and the lack of follow through on most of them could have many causes, but fraud and/or incompetence are two of the possibilities.

The press release provides more on the new study: “Sholl chose MOFs because they’re an area of interest to his lab — he develops models for the materials — and because the National Institute of Standards and Technology (NIST) and the Advanced Research Projects Agency-Energy (ARPA-E) had already assembled a database summarizing the properties of MOFs. Co-authors Jongwoo Park and Joshua Howe used meta-analysis techniques to compare the results of single-component adsorption isotherm testing — how much CO2 can be removed at room temperature.”

“That measurement is straightforward and there are commercial instruments available for doing the tests. ‘People in the community would consider this to be an almost foolproof experiment,’ said Sholl, who is also a Georgia Research Alliance Eminent Scholar in Energy Sustainability. The researchers considered the results definitive when they had 4 or more studies of a given MOF at comparable conditions.”

While the end result of such “errors” may be generally less dire than it is in, say, pharmaceuticals, it’s still worth making note of.

“The net result is non-optimal use of resources at the very least,” Sholl explained. “And any report using one experiment to conclude a material is 15% or 20% better than another material should be viewed with great skepticism, as we cannot be very precise on these measurements in most cases.”

Sholl continued: “We have presented this study a few times at conferences, and people can get pretty defensive about it. Everybody in the field knows everybody else, so it’s always easier to just not bring up this issue.”

Before ending this article I should probably note that, having taken a close look at it, climate science is one of the most solid bodies of research out there. While I would always take specific predictions or models with a hefty grain of salt, the basic science of greenhouse gases and their effects on climate is just about as solid as it gets … and there simply isn’t the financial incentive in the climate sciences to fudge the results the same way that there is in pharmaceuticals and materials science.

The new findings are detailed in a paper published on November 28th in the ACS journal Chemistry of Materials.

Have a tip for CleanTechnica? Want to advertise? Want to suggest a guest for our CleanTech Talk podcast? Contact us here.

Latest CleanTechnica.TV Videos

CleanTechnica uses affiliate links. See our policy here.

James Ayre

James Ayre's background is predominantly in geopolitics and history, but he has an obsessive interest in pretty much everything. After an early life spent in the Imperial Free City of Dortmund, James followed the river Ruhr to Cofbuokheim, where he attended the University of Astnide. And where he also briefly considered entering the coal mining business. He currently writes for a living, on a broad variety of subjects, ranging from science, to politics, to military history, to renewable energy.

James Ayre has 4830 posts and counting. See all posts by James Ayre