Master’s students looking for ‘failed’ research for new scientific journal

In this month’s edition of Nature, a Canadian researcher describes how much effort it had cost him to get the results of his study on the often-praised CRISPR method for genetic modification published. Most important reason: his study hadn’t yielded any positive results. The author states that scientists and journals have gotten so used to exclusively telling success stories, that they forget that scientific progress comes from failure. “We need more honesty in science.”

It’s these types of stories that gave a group of students of the UU Master’s programme in History & Philosophy of Science an idea. Apparently, there’s a gap between what’s being researched and what’s being published, they’d noticed. And that doesn’t help science. The four took the remarkable initiative for a scientific journal for articles that bring to light ‘mistakes’ and that – for that reason alone – would not be easily published elsewhere.

Journey of discovery
The title of the journal is Monty Python-esque: The Journal of Trial and Error. As the Ministry of Silly Walks had once exposed the absurdity of politics, here science seems to be challenged similarly. Founders Martijn van der Meer and Stefan Gaillard laugh when they’re – not for the first time – confronted with the comparison. Martijn: “Originally, we’d even considered naming it Journal of Failure and Journal of Failed Experiments.”

Still, the journal is an utterly serious affair. There’s currently a call for papers for the first edition, which is scheduled for release next summer. They’ve already met all the requirements for a credible scientific journal.

Stefan: “That was quite a journey of discovery in itself. You don’t want to know how much work that is. Your articles have to be able to be found in search engines, and have to receive unique characteristics according to a certain system – so-called digital object identifiers – to name one thing. And we also wanted to publish open access, for instance. Thankfully, we received a lot of help from employees of the University Library.”

Bad for science itself
President Martijn and treasurer Stefan say their enthusiasm comes forth from a kind of productive disillusionment. Like many other young researchers, they were confronted with the realisation that science is haunted by unrealistic expectations of frontier breakthroughs and the suffocating ‘publish or perish’ culture that regularly pushes researchers to test the limits just so they can yield positive research results.

Martijn: “Our Master’s programme partially focuses on studying that system. As a student, you can start to wonder whether science is really all that fun. But we feel like the issues are mostly bad for science and scientists themselves. With this journal, we hope to contribute – however small – to solving those issues. Also because we ourselves might one day want to continue our careers in research. And the fact remains that the process of trial and error is the foundation of good science.”

Dead salmon
The new journal wants to focus on articles of which the results don’t meet the expectations the researcher initially had, but that can indeed be useful for science as a whole, or for other researchers from that same discipline. Publishing these could, for instance, bring to light methodological errors, the two think. As an example, they point to the well-known experiment in which fMRI scans showed brain activity in dead salmon. That showed that not all researchers who worked with the scans understood that they had to correct their observations in order to get actual, valid results.

But the journal would also include articles that focus on conceptual issues. Martijn: “Sometimes, we hear emotional stories from PhD candidates. They can spend months working on a study without it leading to the expected results, and then, for instance, they’re at a social gathering after a scientific convention where they hear: ‘oh yeah, but that doesn’t work, we’ve known that for ages.’ But that had never been published anywhere. Everyone has to invent the wheel for themselves; wasted energy, that doesn’t exactly help science.”

Put in quotes
The two hope to receive short articles in which the authors are asked not only to explain their study according to the well-known IMRAD structure (Introduction, Methods, Results, and Discussion), but also to include a paragraph on the original purpose of the study, as well as a ‘take home’ message. The articles are peer-reviewed double-blind. On top of that, historians, philosophers, sociologists and methodologists of science and theoreticians will be asked to reflect on what exactly is going on, and which scientific ‘errors’ the article highlights.

“We’re not looking for sloppy science,” Stefan emphasises. “That’s why ‘errors’ really needs to be put in quotes.” Martijn: “It’s not like we’re looking to expose science or anything like that. We are aiming for a platform where scientists can simply say: this didn’t go right, for this and that reason.”

The reflection articles in the magazine, Stefan and Martijn say, force people to think critically about possible implicit assumptions in the research practice, and about what’s necessary and possible to help advance science. Martijn: “We’re in favour of an interdisciplinary, constructive view of science. We have to do this together.”

Coming clean
The idea isn’t entirely new, Martijn and Stefan acknowledge. Especially in the past few years, when there’s been an increased emphasis on Open Science, already established journals have also taken initiatives in clarifying how science works in practice – or doesn’t. But established scientists are unlikely to be eager to come clean about their errors in a new, unknown journals. Martijn grins: “We’re not quite that naïve.”

The duo mostly hopes that young researchers and Master’s students at the start of their careers will be brave enough to send in their work. Stefan: “We often see that students who get a 9 for their theses but whose research results are negative have less of a chance to be published than students who get an 8, but whose results are positive. In those cases, we offer an alternative platform.”

Additionally, there are plenty of students who’ve had to work with poor databases, or in an impossible experimental setting, and who wish to share their experiences, the duo knows. With some irony, Martijn says: “It’s remarkable that Bachelor’s students are often taught pure thinking and working, and that in the Master’s phase, that tends to disappear a little. Then, students join running studies of other scientists, which is when the compromises begin.”

The president thinks the journal has something new to offer, especially because of its emphasis on context and reflection on the articles. The team members also write blogs on the journal’s website about the idea behind the magazine.

A big, ugly beast
The first papers have been submitted by now. One of them is about a fieldwork study on the effect of specific types of drugs on the ability to drive. Stefan: “The study proved that the ability to drive wasn’t negatively influenced, and so, the paper didn’t get published anywhere. Failed science, is what they call it.”

Since launching the idea for the journal during a Christmas party of the Descartes Centre, the students have received many positive responses. Since then, financial support has been taken care of through the Ufonds, and a workplace has been found in the UtrechtInc Incubator in the Kruyt building. The journal has also joined the ‘Utrecht Open Science Community’, and it’s supported by the open access specialists of the Utrecht University Library.

Most scientists responded approvingly of the plan, which has since also been presented on international conferences at the universities of Cambridge, Strasbourg, and Lille. Some of the scientific employees is studying or working abroad, in places like Marseille and Montreal.

But the discussion about scientific ‘errors’ remains “a big, ugly beast”, Martijn notes. “Everyone thinks it’s fantastic what we’re doing, but when they realise what we’re actually asking of them, many back out. Many researchers are in an environment, or work under leadership that asks a certain attitude. It’s not easy to break out of that. Still, we’re seeing many, mostly young people who see the importance of a different attitude.”

The call for papers for the Journal of Trial and Error is open until December 1. Martijn would like to add that students can also submit a set-up of their articles or theses. “It doesn’t immediately have to be a perfect paper. We can help with that before sending it to the peer review.”