Keeping NIH Peer Review Robust in Difficult Times
One of the key themes at the May 6, 2013, meeting of CSR’s Advisory Council was how to keep NIH peer review robust in difficult times.
CSR Director Dr. Richard Nakamura summed up his comments to the council: “Due to the sequester and rising numbers of applications, NIH has had to significantly cut funding, and its success rates are an historic low of 16 percent. Applicants, reviewers and NIH staff are unsettled because
everyone knows researchers facing the devastating closure of labs with promising lines of research.”
“There is so much wonderful science being proposed now,” he continued. “The number and quality of research proposals going unfunded have never been higher in the 67-year history of NIH grants. During many of these years, much higher success rates allowed U.S. science, health and economy to flourish." He noted that since 2000, success rates for NIH grant applications have fallen nearly 50 percent.
“This situation has made it difficult for CSR and our reviewers,” said Dr. Nakamura, “because for NIH Institutes and Centers to make informed funding decisions, our reviews need to finely discern the relative merits of applications in the top 10-15 percent -- the applications reviewers have judged to have tremendous merit.”
What We Can Do With Peer Review
“I believe we can do better in gathering the knowledge and best judgments of our reviewers” said Dr. Nakamura. “To this end, I have been working with my colleagues  at NIH as well as with members of our advisory council.” He and council members discussed three ways of doing this:
- Provide New Guidance on Scoring: As paylines and budgets contracted, reviewer scores have compressed around the perceived funding range, leaving NIH Institutes and Centers in the difficult situation of having to discriminate between applications that got the same scores. NIH Office of Extramural Research (OER) has given reviewers updated guidance on scoring applications so they can provide reviews that will better help the NIH Institutes and Centers make funding decisions.
- Explore Ranking of Grant Applications: “It might be better if reviewers also ranked applications,” said Dr. Nakamura, “because it is hard to have perfect pitch and judge applications by an abstract ideal.” He said CSR will host a public meeting to discuss the pros and cons of ranking grant applications, possibly as a supplement to traditional scoring.
- Develop a Science of Peer Review: CSR is working to be more scientific in evaluating the quality of peer review and assessing CSR’s practices and programs. Dr. George Chacko Director of CSR’s Office of Planning, Analysis and Evaluation told council members how CSR is developing infrastructure, datasets and standard protocols to perform these assessments and collaborate with academic researchers. He discussed five areas for evaluating peer review outcomes:
- Design of study sections
- Referral of applications to study sections
- Recruitment of reviewers to study sections
- Management of study sections
- Selection of applications with the greatest potential for impact
CSR Welcomes New Council Members
Learn More About CSR’s Advisory Council online.
Dr. Nakamura is working closely with Drs. Sally Rockey and James Onken at the NIH Office of Extramural Research; Dr. James Anderson and George Santangelo at the NIH Division of Program Coordination, Planning, and Strategic Initiatives; and Dr. George Chacko, CSR Office of Planning, Evaluation and Analysis.