Peer review disadvantages

Traditional peer-review processes used by journal editors to aid in deciding which papers are worth publishing is not capable of filtering some of the more sophisticated techniques of covered marketing and conflicts of interest. The incorporation of ethicists in the peer review process would likely help to raise red flags and to properly consider the routine statement that the study was accepted by the “human review board” of some prestigious university. By rejecting suspicious ethical studies, editors may not be able to help make the world a fairer place, but they will help in building a healthier scientific community and sending a clear message, to both scientists and the industry, that it is unacceptable to exploit and potentially harm a few people for the sake of many ¹⁾

Because the double blind system is rarely used, due to its impracticality, there are often accusations that papers are judged on the reputation of the author instead of quality.

An excellent paper written by a new or maverick scientist can be rejected, whilst a poorer but uncontroversial paper by an established researcher can sail through the peer review process.

The whole process, especially for prestige journals, is very time consuming and expensive. Papers can be held up for many months, whilst approval is sought and, if a researcher has a series of experiments planned, progress may be hampered progress. Much of the decision-making power rests in the hands of the editors, who are the link between author and referee. In most cases, this is fine, and helps the process to proceed smoothly, but it can lead to misuse of this authority. Editors often stand accused of arbitrarily rejecting manuscripts before they reach the peers.

There is evidence that decisions are often judged by country: a US based journal is much more likely to reject non-US papers, whatever the quality.

Peer reviewers have not got an easy job spotting conflicts of interest, such as the source of a researcher’s funding, only that the research was followed to acceptable protocols.

Whilst the process can pick out any obvious omissions and errors, it is impossible for the reviewers to detect determined fraud without replicating the experiment. This limitation is not crippling, but journals could certainly do a better job of making this clear.

There are no grading systems about the quality of the peer review. Different journals have different standards, and there is no way to know the expertise and quality of the reviewers or editor. In the past, less reputable journals generally had a smaller readership, but the internet has made them just as likely to be accessed and used.

The core premise of evidence based medicine is that clinical decisions are informed by the peer reviewed literature. To extract meaningful conclusions from this literature, one must first understand the various forms of biases inherent within the process of peer review.

Hirshman et al., performed an exhaustive search that identified articles exploring the question of whether survival benefit was associated with maximal high grade glioma (HGG) resection and analysed this literature for patterns of publication. They found that the distribution of these 108 articles among the 26 journals to be non-random (p<0.01), with 75 of the 108 published articles (69%) appearing in 6 of the 26 journals (25%). Moreover, certain journals were likely to publish a large number of articles from the same medical academic genealogy (authors with shared training history and/or mentor). They term the tendency of certain types of articles to be published in select journals 'journal bias' and discuss the implication of this form of bias as it pertains to evidence-based medicine ²⁾.

Hirshman et al., propose that this concept can be applied to medical training and that this “medical academic genealogy” may influence the landscape of the peer-reviewed literature. They performed a comprehensive PubMed search to identify US authors who have contributed peer-reviewed articles on a neurosurgery topic that remains controversial: the value of maximal resection for high-grade gliomas (HGGs). Training information for each key author (defined as the first or last author of an article) was collected (eg, author's medical school, residency, and fellowship training). Authors were recursively linked to faculty mentors to form genealogies. Correlations between genealogy and publication result were examined. The search identified 108 articles with 160 unique key authors. Authors who were members of 2 genealogies (14% of key authors) contributed to 38% of all articles. If an article contained an authorship contribution from the first genealogy, its results were more likely to support maximal resection (log odds ratio = 2.74, p < 0.028) relative to articles without such contribution. In contrast, if an article contained an authorship contribution from the second genealogy, it was less likely to support maximal resection (log odds ratio = -1.74, p < 0.026).

They conclude that the literature on surgical resection for HGGs is influenced by medical academic genealogies, and that articles contributed by authors of select genealogies share common results. These findings have important implications for the interpretation of scientific literature, design of medical training, and health care policy ³⁾.