Do Reviewers Read References? And If So, Does It Impact Their Scores?
In March 2017, we wrote about federal funders’ policies on interim research products, including preprints. We encouraged applicants and awardees include citations to preprints in their grant applications and progress reports. Some of your feedback pointed to the potential impact of this new policy on the peer review process.
Some issues will take a while to explore as preprints become more prevalent. But some we can dig into immediately. For example, how do references cited in an application impact review? To start to address this question, we considered another one as well: do peer reviewers look at references – either those cited by applicants or others – while evaluating an application? We had heard anecdotes, ranging from “Yes, I always do,” to “No, I don’t need to,’ but we didn’t have data one way or the other. And if reviewers do check references, how does it impact their understanding and scoring of an application?
So, together with colleagues from the NIH Center for Scientific Review (CSR), we reached out to 1,000 randomly selected CSR reviewers who handled applications for the January 1, 2018 Council Round. There were an equal number of chartered (i.e. permanent) and temporary reviewers solicited to participate (n=500 each) over a three week period from November 16 to December 8, 2017.
Our survey focused on the last grant where they served as primary reviewer. Specifically, we asked if they looked up any references that were either included in t...
Source: NIH Extramural Nexus - Category: Research Authors: Mike Lauer Tags: blog Open Mike applications citations Peer review Source Type: funding