Image © CC BY NASA Goddard Space Flight Centre http://bit.ly/1wPSD0m
The idea of public post-publication peer review of journal articles would have been considered heresy just a couple of years ago, but in recent years there has been a growth in post-publication reviewing options. The following blog post looks at a few of them and discusses the idea of leaving post-publication reviews and whether its all as bad as some in the academy community seem to think it is.
The journal publishing model has been quite consistent for many years now, authors submit their paper to a peer-reviewed journal. Reviewers read and analyse it and give feedback, and depending on how good or bad that feedback is the paper is eventually accepted or dismissed for a variety of reasons, which may lead the authors to attempt a resubmission to another journal. For the papers that do get published, that often ends the cycle of conversation between peers unless the research is discussed at an event, such as a conference or appears in the media, where comments can be easily left - which are sometimes scathing, unjustified or plain unhelpful. And there is the rub, when does post-publication peer review become post-publication comment and how different are they?
Unlike many other things that appear on the Web, such as music, film, art and review, where comments and critique is normal and helps others make better informed judgements on what to consume; academic articles only appear above the surface for review and critique when it is used as part of someone’s research or teaching. Often there is no critique, good or bad of the research post-publication, just an important line, conclusion or otherwise to help build the hypothesis of new research; agreeing with or against previous works. To say there was no post-publication review system previously would be untrue as the likes of BMJ editors and others have accepted e-letters and Rapid Responses about research published in their journals, and blogging and social media have more recently offered platforms for researchers to discuss other’s works.
From my experience many researchers feel uncomfortable on speaking about other people’s research, which is understandable. Nobody wants to hear ill of their own hard work, as this is what they are potentially opening themselves up to. Take for example the reviews on such as YouTube or the Daily Mail website, often comments can become personal, malicious and quite damning. The issue is that everyone has an opinion and that can be on everything from Syria to fracking, and the Web has facilitated that opinion culture to the point where ‘trolling’ is now an acknowledged and serious problem. Yet academic publishing is different, certainly academics are more than culpable for their barbed comments, but making unjustified ones online will help no one, especially in the advancement of knowledge via discourse.
The journal publishing model has been criticised for being out of touch with modern publishing, and rightly so; in that a piece of research which can take years to complete can then take nearly as long to get published. So by that time things may have moved on in that topic of research, new methods, technologies and ideas may have surfaced. Post-publication reviews can help highlight this, and also may make researchers aware of potential future collaborators or similar research being undertaken.
For post-publication review to really be productive it has to be open, unlike sites such as YouTube which has allowed aliases and therefore trolls to flourish. Obviously not every piece of research commands a post-publication review and given the figures which range from about 12% to 90% of papers not being cited, it is pretty likely that not all papers will get reviews or at least have the mechanisms to be reviewed. We also have to remember that while some areas of research are less reliant on the journal publishing model, this does not mean post-publication review is not for them, in disciplines such as the humanities it may have just as much use.
Academic debate using the many Web 2.0 and social media tools freely available has only been embraced by a small percentage of academics. Interesting papers are more likely just shared using such as Twitter, Google+ and LinkedIn than discussed, but considering that it is far easier and less time-consuming just to share content on the web than review it, it is understandable. Reviewing takes time and unlike reviewing a film, which is foremost a subjective piece of writing and focuses on whether you enjoyed it or not and whether it was well made, peer-review requires more considered thought. Research is measured on whether it was well designed and conducted, not how well it was written (although that does come into the formal review process - but more about whether it is understandable, not just using long words to impress). That said I will cover JOVE below which helps aid that second part of the review process.
The debate on whether is the best way forward for post-publication review will continue and like other topics such as measurement of research, there appears to be no ‘silver bullet’. Instead there is a collection of sites and tools operating in silos, all offering to solve a problem, that being the lack of post publication discussion and assessment. Below are a list of some of the main tools and sites offering some kind of comment, discussion or review system- it is not exhaustive or comprehensive, but it will give you some idea as to what they are and do.
PLOS ONE
PLOS ONE’s refers to its mission statement as; “Accelerating the publication of peer-reviewed research”. First and foremost PLOS ONE is an open access collection of journals that unlike many traditional journals has sped up up the publication process and ensures authors retain copyright. Not a post-publication review site outright, it does allow users to comment on the published research, very much how newspapers allow visitors to comment their news articles. Commenting on research is in essence less formal than post-publication reviewing, the reader comments and has the remit to post something as indepth as they wish. They may want to write just a few lines about a part of the research, the methodology, results or conclusion or a longer more in-depth review about the whole paper. When commenting on papers in PLOS ONE you must be a registered user and identify any competing interests. The rules are quite simple and say that anyone commenting on someone else’s research must not post content as stated below:
Remarks that could be interpreted as allegations of misconduct
Unsupported assertions or statements
Inflammatory or insulting language
Anyone breaking these rules will be removed and their account disabled- obviously it does not stop them creating new accounts, but that will always be a problem for many interactive websites.
http://www.plosone.org/
PubMed Commons
PubMed is a huge publicly accessible search engine that accesses the Medline database of references and abstracts on life sciences and biomedical topics. It recently launched PubMed Commons to enable authors to share opinions and information about scientific publications in PubMed.
To be eligible to use PubMed Commons you have to be an author of a publication in the database, therefore preventing anyone from going in and leaving comments. The emails of eligible authors have been collected from the NIH and the Wellcome Trust and authors emails within PubMed and PubMed Central. You can also ask a colleague to invite you into the system.
© PubMed
The guidelines for PubMed Commons are more stringent than that of PLOS ONE and other such sites. Commenters must use their real name and again disclose any conflicts of interest. By contributing to Commons they grant other users a worldwide, royalty-free, non-exclusive, perpetual license under the Creative Commons Attribution 3.0 United States License. Again the usual rules of not posting inflammatory, offensive and spam comments apply. Full guidelines can be viewed here:
Open Review
Open Review is a new feature within the popular academic social network and research sharing platform ResearchGate. Open Review allows users to publish an open and transparent review of any paper they have read, worked with, or cited. ResearchGate say it is: “Designed to approach the evaluation of research in a different way, Open Review encourages scientists and researchers to focus on one key question: Is this research reproducible?” Users can discuss articles they click on, with a slant more towards asking questions about the publication, than commenting or reviewing it.
F1000 Research
© F1000
F1000 - standing for Faculty of 1000 - is made up of F1000 Prime, which is a personalised recommendation system for biomedical research articles from F1000. F1000 Research is an open science journal with post-publication peer reviewed research with underlying datasets. Finally there is F1000 Posters which is an open repository for conference posters and slide presentations.
F1000 Research has a system of open peer review which publishes referee responses and allows for replies by the authors and reader comments, so a bit of everything. In addition they offer incentives for reviewers which include a 50% discount on article processing charges for the 12 months following the submission of their referee report. They also offer a 6 month free personal subscription of their sister service F1000 Prime. Users of F1000 can track the conversation and even discuss the article at the bottom of the page, so the entire process, paper, review and discussion are taking place on one page. Even referee’s reports can be cited in F1000 Research and published under a CC BY license. A DOI (digital object identifier) is assigned to every referee report, so it can be cited independently from the article.
PubPeer
PubPeer refers to itself as the online journal club that allows users to search for papers via DOIs, PMIDs, arXiv IDs, keywords and authors amongst other options. PubPeer’s goal is to create an online community that uses the publication of scientific results as an opening for discussion among scientists. Researchers can comment on almost any scientific article published with a DOI or preprint in the arXiv. You can also browse the list of journals with comments, although presently it is rare to find a journal with more than a couple of comments. First and last authors of published articles are invited to post comments- I’m presuming the authors in between also get a say. Unlike some of the other tools mentioned, PubPeer allows for anonymous commenting, which could open the door for more trolling and abusive behaviour as reviewers feel an extra level of protection from what they say. -one researcher has filed a lawsuit over anonymous comments which they claim caused them to lose their job after accusations of misconduct in their research.
Publons
The primary aim of Publons is to help researchers get credit for peer review. Whilst writing for peer-reviewed journals have often been seen by many of handing their hard work over to someone else to benefit from financially, that being the publishers, at least there is the benefits of the author’s increased kudos, profile and knowledge-building and potential to gain promotion within their organisation. Peer reviewing can also have similar rewards with regards to the researcher’s CV and promotion prospects and that they get to see emerging research but the anonymous nature of much of it means less opportunity for profile building, yet it is no less part of the system that is the research publishing cycle. Publons sets to works with reviewers, publishers, universities, and funding agencies to turn peer review into a measurable research output. They collect peer review information from reviewers and from publishers, and use the data to create reviewer profiles with publisher-verified peer review contributions that researchers can add to their CV. Publons state that; “reviewers control how each review is displayed on their profile (blind, open, or published), and can add both pre-publication reviews they do for journals and post-publication reviews of any article.”
https://publons.com/
The Winnower
The Winnower is possibly the least academic looking post-publication platform of them all, but that should not put readers off; in fact it should have the opposite effect. An attractive site that sets its stall out on the homepage with the statement that; “The Winnower is founded on the principle that all ideas should be openly discussed, debated and archived. As with so many new academic tools and platforms it began life thanks to a PhD student, namely Joshua Nicholson from Virginia Tech. It provides an interesting new angle looking at research from both ends of the spectrum, that which has made a big impact and research that was retracted with its own ‘Grain’ and ‘Chaff’ page. The grain features publications with more than 1000 citations or a Altmetric score above 250. Whilst the chaff looks at papers that were pulled from publication and give authors an opportunity to talk about their research rather than just a ‘name and shame’ list. The Winnower is obviously still in its early stages due to the handful of reviews and publications, but not every post-publication review site can begin from the point of PubMed. It is one certainly worth keeping an eye on.
https://thewinnower.com/
JOVE
The Journal of Visualized Experiments has now been around for some time, almost a decade but it is only in the last couple of years it has really broken through and is now subscribed to by many university libraries. JOVE is a PubMed-indexed video journal with a mission to increase the productivity of scientific research. Although not at the forefront of JOVE’s priorities, they do allow for comments on the published research videos.
Peer J
Peer J is an open access peer-reviewed scientific journal that focused on publishing research in the biological and medical sciences. It received substantial backing of USD 950,000 from O'Reilly Media - which founder Tim O’Rielly is famous for popularising the term Web 2.0. It is part of the same publishing company that was co-founded by publisher Peter Binfield (formerly at PLOS ONE) and CEO Jason Hoyt (formerly at Mendeley), who obviously have a lot of experience in scholarly communications.
© CC BY Peer J
Peer J has a points system for authors and commentators as an incentive to publish and comment on research. Anyone who has ever argued that citations, H Indexes and such as Twitter and followers are just multi-levelled multiplayer games will get this. The points system are below:
Be an academic editor on a PeerJ article = 100 pts
Be an author on a published PeerJ article = 100 pts
Make your manuscript reviews public on a PeerJ article = 35 pts
Submit an "open review" as a reviewer on a PeerJ article = 35 pts
Be an author on a PeerJ PrePrint = 35 pts
Be an academic editor on a rejected PeerJ article without reviews = 35 pts
Have an answer on a question accepted = 15 pts
Have feedback deemed "very helpful" by an author of a PeerJ PrePrint = 15 pts
Receive an up vote for an answer = 10 pts
Receive an up vote for a question = 5 pts
Receive an up vote for feedback on a PeerJ PrePrint = 5 pts
Receive an up vote for reply to question or comment = 1 pt
Have first feedback approved in moderation on a PeerJ PrePrint = 1 pt
There are tables of the top authors and reviewers which can be filtered by topic area, publication date and those who have asked the most questions and given the most answers. The questions and answers aspect is a different angle to the commenting process as it does potentially open up further dialogue between authors and commentators. At present though there does not seem to be much activity in this area.
Peer J state: “Our annotation system goes beyond just answering questions or finding answers. Everyone from authors, editors, reviewers, and visitors to PeerJ are contributing in some way. Often, these are "hidden" contributions to the body of science that can go unrecognized. The points that we are starting to show on profile pages are just a light way to surface this participation.”
As for this points ranking system, it will appeal to some researchers, those with a competitive edge, but on the flipside will feel uncomfortable to others who do not want to see their work captured in numbers, and that applies to any kind of metric not just Peer J. Netherless, it is an interesting take on the publishing model and one that will continue to create interest and debate.
PaperCritic
One of the first proper research commenting tools, PaperCritic appears to have ceased business but is still worthy of a brief mention. Created using the Mendeley API, PaperCritic connected with a user’s Mendeley account and allowed them to comment on research hosted within Mendeley’s huge database of references. Their blog, Facebook and Twitter accounts all fell silent in 2012 leading me to believe that this was no longer running. The chances are that Mendeley will at some point create their own commenting and reviewing system, so still well worth the mention.
There seems to be some difference between the notion of reviewing, discussing and commenting, something Kent Anderson in The Scholarly Kitchen wrote about earlier this year. With Anderson summerising that; “Today’s commentators seem to have many axes to grind. Far too often, commentary forums degrade into polemical attacks with win or lose dynamics at their heart. The pursuit of knowledge and science isn’t the goal. Capitulation of one combatant to another is.” Anderson questioned the validity of comments being championed by publications and websites and that they could never be considered in the same light as peer-review.
There is a need for both as comments can be insightful and highlight or spot useful content for the original authors or other readers without going into indepth reviews. On the flipside they can be malicious, unfounded and just clog up the whole knowledge process if left un-moderated and anonymous. Peer-review may not be perfect, but as the social web becomes more useful as a platform for discussion and knowledge sharing, it makes sense that other options are explored, even if they are supplementary. This is growing case for Altmetrics, first seen as an alternative to the traditional measurement of citations and now argued as more of an alternative indicator, rather than measurement. The real problem is that like with many other technologies and platforms for communication we run the risk of not being able to see the wood for the trees. Post publication review platforms need to be explicit in their aims and explain that clearly to readers and reviewers. Like social media, it is unlikely that we will see every researcher using these unless they became standardised and part of the research cycle. It is an option, as with academic discussion lists, where the most insightful and on occasion barbed communications take place. Post-review commenting is happening right now and someone out there may have already commented on your research- whether you respond remains your choice.
Recommended reading:
