George Siemens sent me a link to a post by Cameron Neylon that attempts to pound yet another nail in the coffin of peer review. As an editor of a peer reviewed Journal (IRRODL) I was naturally both curious and a bit defensive about the charges.
Neylon argues that for peer review “ Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgment as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window.
Let me respond by describing some IRRODL practices. We don’t make referee’s reports public, mostly because of the work that would be involved in cleaning them up, getting permissions and my questioning the value of these reports. Do the readers really care if reviewer A doesn’t like the wording of the abstract, or Reviewer B’s thinks the author has missed a major reference, or reviewer C says the statistical analysis of the data is weak? We return all these comments to the reviewer and ask them (using tracked changes and a point form response) how they responded (or why they choose not to respond) to the concerns and suggestions of the reviewers. Now of course this applies only if we want to see the article again for further review or publication. As Neylon suggests if we decline the offer to publish, the process may start anew with a second journal, but the work of the reviewers is not “simply thrown out the window”. In some cases the author gives up (often appropriately so, as the reviewers found fatal flaws or ones that demand more work than the author is willing to give to the paper). The author also gains the lesson learned and experience (hard though it may feel) of how NOT to write a scholarly article. But in other cases, a smart author will take into account the comments of the reviewers and improve the paper- thus increasing its chances of successful publication elsewhere. This demonstrates the ongoing value of the reviewers’ work, even if the article is, or is not, published elsewhere.Read More