I came across this post while getting up to speed with Critical Insight, a UK-based online blog-come-journal club. It had a link to a recent BMJ editorial by David Schriger and Douglas Altman — you may remember David Schriger from his fantastic talk on ‘Risk in Emergency Medicine’ as featured on LITFL some time ago.
The editorial laments the lack of effective post-publication peer review of medical research. This is a critical failing given the imperfections of peer review — as amusing as it may be (to some).
The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong. A recent editorial in Nature was right to conclude that an over-reliance on peer-reviewed publication “has disadvantages that should be countered by adequate provision of time and resources for independent assessment and, in the midst of controversies, publicly funded agencies providing comprehensive, reliable and prompt complementary informationRichard Horton, Editor of The Lancet, MJA 2000; 172: 148-149
Schriger and Altman list the following potential reasons for a lack of post-publication peer review:
Paper perfect (yeah right…)
- No criticism needed
(rare occurrence since few papers are perfect and all papers need to be placed in context of prior work)
Paper imperfect yet no feedback submitted
- Paper trivial, unworthy of comment
- No one reads the paper
- Readers fail to detect flaws
- Readers assume that prepublication peer review has eliminated important errors
- Readers uncomfortable or scared to publicise flaws
- Readers dissuaded by logistics (eg, due date, word limit)
- Reader notices flaw after deadline passed
- Reader believes letter unlikely to be published
- No reward for writing a letter
Unreasonable constraints (eg, no more than 3 letters about each paper, letter arrives too late)
- No published response from authors
- Authors’ egos or motivations
- No disincentive for non-response
- Author response inadequate
- Evasive responses are common and accepted by journals
Failure to resolve outstanding issues
- Second set of correspondence extremely rare
Of note, Schriger and Altman point out that:
“if logistics were the sole problem we would expect that web based rapid response features that remove many of the logistical difficulties of publishing a critique would be popular.”
That this is not happening suggests that other factors are at play — a cultural problem relating to open communication among the medical/ scientific communities? Or something to do with the vested interests of journals, authors and/or those who fund research? The authors also emphasise that we are presently being buried under “a mountain of poor quality unfocused literature has left its readership fatigued, numb, and passive”. They conclude that:
“The solution to the absence of effective post-publication reviews does not lie within its mechanisms; it requires a fundamental reworking of what research is performed, how it is presented, and how it is assimilated into current knowledge. We need fewer papers that are of higher quality and importance. We also need a change in culture to value public discussion if we are to re-engage the medical research community in the kind of post-publication review process that patients deserve.”
The answer may be self-evident, but is there a role here for scientifically and medically trained bloggers? Sometimes reality only really bites once research reaches the blogging community. The recent hoo-hah over claims by NASA scientists about the possible existence of an arsenic-based life-form is a prime example. Perhaps the time has come, as Richard Smith suggests, to move from a world of “filter then publish” to one of “publish then filter”. However, this still requires someone to sort through all the crap…
- Schriger DL, Altman DG. Inadequate post-publication review of medical research. BMJ. 2010 Aug 11;341:c3803.
Chris is an Intensivist and ECMO specialist at the Alfred ICU in Melbourne. He is also a Clinical Adjunct Associate Professor at Monash University. He is a co-founder of the Australia and New Zealand Clinician Educator Network (ANZCEN) and is the Lead for the ANZCEN Clinician Educator Incubator programme. He is on the Board of Directors for the Intensive Care Foundation and is a First Part Examiner for the College of Intensive Care Medicine. He is an internationally recognised Clinician Educator with a passion for helping clinicians learn and for improving the clinical performance of individuals and collectives.
After finishing his medical degree at the University of Auckland, he continued post-graduate training in New Zealand as well as Australia’s Northern Territory, Perth and Melbourne. He has completed fellowship training in both intensive care medicine and emergency medicine, as well as post-graduate training in biochemistry, clinical toxicology, clinical epidemiology, and health professional education.
He is actively involved in in using translational simulation to improve patient care and the design of processes and systems at Alfred Health. He coordinates the Alfred ICU’s education and simulation programmes and runs the unit’s education website, INTENSIVE. He created the ‘Critically Ill Airway’ course and teaches on numerous courses around the world. He is one of the founders of the FOAM movement (Free Open-Access Medical education) and is co-creator of litfl.com, the RAGE podcast, the Resuscitology course, and the SMACC conference.
His one great achievement is being the father of three amazing children.
On Twitter, he is @precordialthump.