'Repeating experiments should be the norm'

In the fight against sloppy and fraudulent science, researchers should repeat each other’s experiments more often, according to a new advice from the Royal Netherlands Academy of Arts and Sciences (KNAW). Even if they don’t want to.

It came as a shock to the world of social psychology when, in 2011, the fraud of Tilburg professor Diederik Stapel was exposed: for years, he’d been fabricating all kinds of experiments. How was he able to do this without getting caught for so long?

In other academic disciplines, the results of academic studies are often questioned, too, if only because some scientists tend to exaggerate their results: if they don’t, their research isn’t remarkable enough.

What can we do to fight this kind of ‘science’? There’s only one thing we can do, says professor of public health care John Mackenbach. He’s the chairman of a committee started by the KNAW with the purpose of finding a solution to this problem. “Scientists should repeat each other’s experiments. That should be the norm.”

The committee is presenting its advice today. Non-reproducible research hinders scientific progression, the committee says. If scientific failures are overlooked, the results could be damaging – think, for instance, of incorrect treatment of patients.

Mackenbach’s advice is clear: replicate research more often, and make sure there are funds available for it. To make the possibility of replicating research easier, scientists should disclose more details about their methods. “Since scientific magazines are all published online these days, there is enough room available to do so.”

Repeating studies is, by definition, not innovative. Do you need a different type of scientist for this kind of research?
“No, we should all be doing this. This simply is high quality research. You need the same competencies, the same equipment, as you do for the initial research, so it needs to be done by the very best scientists. The only difference is that admittedly, it’s not exactly innovative.”

Should this be a form of scientific conscription?
“That’s not how we worded our advice. But perhaps all starting PhD candidates could, as a way of expanding their education, replicate an earlier study: they’ll not only learn a lot about the discipline, but they’ll also add to the reliability of their chosen field.”

Speaking of repetition: we’ve heard a similar call before. Even Nobel prize winners have called for repeating of research. Why is this message so difficult to get across?
“Scientists are interested in new things, and repeating is of course quite the opposite of new. It’s a different experience. There are also objective barriers. Magazines aren’t as interested in these results, so it doesn’t have the same status. Researchers will get more credit for innovative studies, which will make it easier for them to receive new grants, etcetera. We’ll need to try to remove those barriers.”

How?
“There are several large-scale international initiatives: quite a few disciplines are looking at where they are in terms of reproducibility of their research results. Magazines are changing. Some magazines are now promising to publish replication results of articles they published before – no matter the outcome. We need to start doing the same in the Netherlands. Research financing institution NWO has already started a program on replicate studies, but the budget is small, not nearly enough to really get a grip on this issue.”

For which disciplines is this the most relevant?
“Of course we understand that disciplines like psychology are at greater risk, because there’s a much bigger human factor. In disciplines like physics, observations are generally done by machines. You’ll sometimes hear physicists say: “this issue isn’t relevant in our field.”. But how do you know if you don’t research it? One of the most important recommendations in our report is that scientists should systematically study the reproducibility within entire research fields or sub-disciplines.”

Do scientists look at replicate studies as a show of distrust?
“I’m not ruling out that this might be a factor for some people, but it’s not something any of the scientists I’ve spoken with have mentioned. They’re simply interested.”

How much should this cost? One percent of the research budget, or rather ten percent?
“Somewhere in between those two, I think. But first, we’ll need to gather more information about the size of the problem.”

 

 

Advertisement