New Scoring Guidance for NIH Reviewers
Scientific Review Officers (SROs) are sharing with their reviewers updated guidance on scoring research grant applications
, training/career award grant applications
and scoring in general
. They are doing this to help reviewers better communicate their assessment of the top tier applications by spreading their scores and increasing their focus on the overall impact of the proposed research based on the criteria.
The Value of Spreading Scores
The vitality of NIH peer review depends on the dynamic use of our scoring system. When reviewers take advantage of the full range of scores, they can better communicate critical judgments that allow NIH to make better informed funding decisions.
Study section scores began to compress around the perceived funding range not long after NIH started using the new 1-9 scoring system in 2009. When we looked at preliminary scores from the last review round, we saw reviewers gave 17 percent of their applications an overall impact score of 2 despite the fact that this score represents 5 percent of the scoring range if scores were spread in a normal curve.
NIH Institutes and Centers (ICs) have a difficult time gauging reviewer assessments when so many applications receive exceptional scores, particularly when study section members all assign a “2” to an application.
Keeping Reviews Fair
Spreading scores means researchers who submit applications that are discernibly better are more likely to be funded. It is what all applicants hope for when they craft their applications.
Some CSR study sections started following the new scoring guidance during the last round. Those who earlier produced compressed scores succeeded in spreading their scores. To ensure that applicants were treated fairly, CSR recalculated the percentile base for these study sections and we also recalculated the “CSR ALL” percentile base, which is frequently used for normalizing scores assigned by special emphasis panels. NIH Institutes and Centers usually look at percentile scores when making funding decisions. Ordinarily, we calculate the percentile for a study section by using as a base the scores assigned by a study section in the current and previous two review rounds.