Studies that replicate are often cited, of course; the problem is that studies that fail to replicate are quoted more or less with the same frequency.
The most plausible explanation is that scientists don’t read the studies they cite. At least, studies in the field of social sciences.
The reproducibility It is the ability of a trial or experiment to be reproduced or replicated by others, in particular, by the scientific community. Reproducibility is one of the pillars of the scientific method, with falsifiability the other. Thus, if researchers indistinctly cite studies that manage to replicate and those that do not, we have a problem.
This is what the data collected by Replication Markets, a part of DARPA’s SCORE program, suggests to us, whose objective is to evaluate the reliability of social science research.
The studies were drawn from all disciplines of the social sciences (economics, psychology, sociology, management, etc.) and were published between 2009 and 2018.
If studies are cited so uncriticallyWe have to understand that researchers do not read these studies in depth because, well, perhaps many are self-deceived or cross-biased, but: what are the editors and reviewers doing?
Whatever the explanation, the fact is that the academic system does not assign quality citations, that is, to fundamentally true studies. This is bad not only because of the direct effect of basing further research on bogus results, but also because it distorts the incentives scientists face: If no one cited weak or bogus studies, we wouldn’t have as many of them. Rewarding impact without regard for the truth inevitably leads to disaster.
In fact, not even sticking to magazines with a high impact index frees us from this scourge. The reputation of the magazine is simply irrelevant. Because again, if scientists were more careful about what they quote, magazines in turn would be more careful with what they publish.
The problem, of course, is systemic. Authors are just one small cog in the vast machine of scientific production. In order for this material to be funded, generated, published and eventually rewarded, the complicity of funding agencies, journal editors, reviewers and recruitment committees is required.
No one really benefits from the current state of affairs, but isolated individuals cannot be asked to sacrifice their careers for the ‘common good’: the only viable solutions are top-down.
Share It seems many scientists do not read the studies they cite: both the studies that are replicated and those that are not cited the same