Science is not well served by people deciding that their methodology is the only legitimate one
As concern grows about questionable practices and outright misconduct in the life sciences, the pre-registration of study designs and hypotheses is being wrongly touted as the panacea.
The campaign’s latest push came in an last month written by Chris Chambers, research fellow at Cardiff University, and Marcus Munafo, professor of biological psychology at the University of Bristol, which was supported by more than 80 signatories.
Drawing on a paper that asked psychologists to self-report their own dubious behaviour, they argue that large numbers of life scientists cherry-pick data, hide null results, fail to employ adequate statistical power and reinvent the aims of studies after they have been completed to make it look as though unexpected findings were predicted.
They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced.
However, there are numerous problems with the idea. Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective. And the commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results.
With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists. Unsympathetic ones would also be handed the chance to veto studies at the outset. In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way.
Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses. Some, for instance, are observational, while many of the participant populations introduce significant sources of complexity and noise; as introductions to psychology often point out, humans are very dirty test tubes.
I am also very uncomfortable with the model’s implication that hypothesis testing is the only correct way of doing science. This may be true for the clinical trials from which pre-registration takes its inspiration, but we have known since Thomas Kuhn that scientists don’t just proceed, study by study, testing individual hypotheses.
In , Chambers has claimed that pre-registered studies would have “a substantially higher truth value than regular studies” because of measures such as a requirement for authors to declare whether they are discussing results they did not predict. But this sort of language is deeply misleading. Most scientific studies are “wrong” in the long term. Science is a process rather than a method of finding out all the things that are true: a process during which we must be allowed to run studies in which we get things wrong, change our minds and are led in directions we didn’t expect.
If we allow to pass uncontested the claim that the pre-registration model is a gold standard, we will permit the denigration of the vast majority of great research and allow a number of serious constraints to be placed on it.
Science is advanced by communities of researchers and is not well served by people deciding that their methodology is the only legitimate one. Instead, our responsibility is to do the best science we can; to be open-minded and interested in the findings of others; and to do our best to support the careers of those who come to work with us.
That way, there is a fighting chance that we will be remembered for the good stuff that we do. After all, even Newton sometimes employed dubious methodologies. His celebrated physical laws were supported by data, but history tends to overlook his equally enthusiastic pursuit of alchemy, which swam in a sea of null results.
