So more thoughts on peer review? And after yet another discussion on the subject at the recent SpotOn conference for online scicomm, I have some notes.
The problems with peer review are clear (or, as said at the conference, “the peer review system is broken, or at least severely dysfunctional”):
- peer reviewers are overloaded with too much material to read through and not enough time to read it
- completing peer reviews does not carry any reward, so there is no motivation to do a good job
- impact factor abuse: competition to place articles in positions of high impact, not positions to reach the audience
- it is too highly praised: the peer review system is held up as the fulcrum of scientific equity, this makes it hard to accept that it is faulty or that big changes might be made; it is too well imbedded
- anonymity is doubtful – researchers often know other members of their field so well, there is no true anonymity in whose article you are peer reviewing, or who really reviewed yours
- reviewers do not repeat experiments, so the contents are not fully reviewed: some stuff is always taken on trust
- the speed of the publishing process and how this impacts upon pressure to peer review and pace of dissemination
I’m sure you can think of more, but there is a bare bones version.
So what to do? The problem with this is, there are lots of ideas, and whilst pooling ideas together and piloting new things sounds like a winner, in practice it doesn’t come out quite so cuddly. What we have now is a wealth of new ideas competing rather than collaborating to be add-ons to the existing system, each plugging their own sale. The hardened capitalists may say that competition is good, it will increase the quality of the product. So far, however, it sucks. Because of competiting priorities and the need for big changes, there is overwheleming choice, none of the systems are completely stable and none can be widely implemented. And remember, these systems are all add-ons; they are peer reviews of peer review; they slow down and add steps to the system
For example, here are just a few of the bizarre things that have come out of peer-review ideologies under the pressure to radically reform it. Just think about the implications…
- a second wave of peer review – DOUBLE PEER REVIEW
- being able to choose your own reviewers, especially for controversial subjects
- peer reviewers obliged to repeat experiments to check they are replicable (with no extra funding)
So much for the greater good.
And yet there are some ideas which are much more ambiguous. What do you think about, for example:
- doing away with impact factors (!)
- being allotted a publication by your reviewers (!)
- rewarding peer review similarly to publications (…)
It comes down to a compromise between making people happy and pushing for quality – something that doesn’t seem to be weighed out by the revision systems: you either make peer reviewers work harder for less returns and have a rigorous review system, or you start handing out candy and open up the reviews for abuse and neglect.