The Impact of Refereeing-Practices on Scientific Progress: Results from a Computer Simulation

The resistance of journals to publish articles with non-results (publication bias) is in so far a problem for scientific progress, as it encourages intentional biasing in order to avoid non-results. Such false "knowledge" is relatively immune against critique by replication studies, which show the true facts but are hard to publish due to the mentioned resistance against non-results. As a refutation of fake-results, their only advantage is their newness. However, in general the appreciation of newness is not encouraging replication studies, although they may show errors in earlier publications. So the question is whether the appreciation of newness really corrects the negative consequences of the avoidance of non-results.

In order to tackle this question the author presents a computer simulation, where a scientific community tests 1000 bivariate correlations, which are in reality partly zero and partly significant. Scientists may un/intentionally deviate from this reality but in any case they attempt to publish their investigations. Whether they succeed or not depends on the refereeing-practices of the editorial boards with regard to the resistance against non-results and the appreciation of newness. At the end of the simulation it is possible to compare the true with the last published correlations. This way it is possible to determine the share of the significant and of the zero correlations, which have correctly been identified in the publications of the scientific community. The advantage of computer simulations is the controlled variation of the following model-parameters, which influence these measures of scientific progress: a) the error rates in the submissions about the significant and the zero-correlations, b) the strength of the resistance against non-results, c) the strength of the appreciation of newness. Obviously it is also possible to determine the influence of these parameters on the number of published replication studies.