Time to publish then filter?

I came across this post while getting up to speed with Critical Insight, a UK-based online blog-come-journal club. It had a link to a recent BMJ editorial by David Schriger and Douglas Altman — you may remember David Schriger from his fantastic talk on ‘Risk in Emergency Medicine’ as featured on LITFL some time ago.

The editorial laments the lack of effective post-publication peer review of medical research. This is a critical failing given the imperfections of peer review — as amusing as it may be (to some).

The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong. A recent editorial in Nature was right to conclude that an over-reliance on peer-reviewed publication “has disadvantages that should be countered by adequate provision of time and resources for independent assessment and, in the midst of controversies, publicly funded agencies providing comprehensive, reliable and prompt complementary information

Richard Horton, Editor of The Lancet, MJA 2000; 172: 148-149

Schriger and Altman list the following potential reasons for a lack of post-publication peer review:

Paper perfect (yeah right…)

• No criticism needed
  (rare occurrence since few papers are perfect and all papers need to be placed in context of prior work)

Paper imperfect yet no feedback submitted

• Paper trivial, unworthy of comment
• No one reads the paper
• Readers fail to detect flaws
• Readers assume that prepublication peer review has eliminated important errors
• Readers uncomfortable or scared to publicise flaws
• Readers dissuaded by logistics (eg, due date, word limit)
• Reader notices flaw after deadline passed
• Reader believes letter unlikely to be published
• No reward for writing a letter

Unreasonable constraints (eg, no more than 3 letters about each paper, letter arrives too late)

• No published response from authors
• Authors’ egos or motivations
• No disincentive for non-response
• Author response inadequate
• Evasive responses are common and accepted by journals

Failure to resolve outstanding issues

• Second set of correspondence extremely rare

Of note, Schriger and Altman point out that:

“if logistics were the sole problem we would expect that web based rapid response features that remove many of the logistical difficulties of publishing a critique would be popular.”

That this is not happening suggests that other factors are at play — a cultural problem relating to open communication among the medical/ scientific communities? Or something to do with the vested interests of journals, authors and/or those who fund research? The authors also emphasise that we are presently being buried under “a mountain of poor quality unfocused literature has left its readership fatigued, numb, and passive”. They conclude that:

“The solution to the absence of effective post-publication reviews does not lie within its mechanisms; it requires a fundamental reworking of what research is performed, how it is presented, and how it is assimilated into current knowledge. We need fewer papers that are of higher quality and importance. We also need a change in culture to value public discussion if we are to re-engage the medical research community in the kind of post-publication review process that patients deserve.”

The answer may be self-evident, but is there a role here for scientifically and medically trained bloggers? Sometimes reality only really bites once research reaches the blogging community. The recent hoo-hah over claims by NASA scientists about the possible existence of an arsenic-based life-form is a prime example. Perhaps the time has come, as Richard Smith suggests, to move from a world of “filter then publish” to one of “publish then filter”. However, this still requires someone to sort through all the crap…

Let us all be thankful for blogs like The Poison Review and Resus.ME and the wonderful world of podcasts where some small steps are being made to make sense of this terrible mess.

Key reference

• Schriger DL, Altman DG. Inadequate post-publication review of medical research. BMJ. 2010 Aug 11;341:c3803.