CSR Initiatives to Address Bias in Peer Review
CSR is committed to addressing bias in peer review. Learn about our commitment and relevant data.
Words from Dr. Noni Byrnes, Director
Words from Dr. Bruce Reed, Deputy Director
Bias Awareness and Mitigation Training for Reviewers, Chairs, and SROs
16,646
As of Dec 2022, 16,646 reviewers have completed the training.
91%
91% of reviewers thought that the training substantially improved their ability to identify bias in peer review.
93%
93% of reviewers stated the training made them substantially more comfortable intervening against bias.
CSR developed training specifically targeted toward mitigating the most common biases in the peer review process. The training includes personal testimonials, interactive exercises, and a narrated mock study section demonstrating techniques to intervene – all based on real-life examples. The training was developed with the assistance of a diverse CSR Advisory Council Working Group. Training has been provided to all CSR reviewers since August 2021.
Return to top
Reporting Avenues for Bias, Unfair Reviews, Uncivil Conduct on Panels
CSR launched a widely-publicized reporting avenue for issues related to respectful interactions, bias or anything else that could affect the fairness of the review process. The reporting channel is open to all – investigators, reviewers, and program staff.
Investigate
Every allegation is carefully investigated by CSR senior management (Dr. Fosu and the Scientific Division Director)
Resolve
If we agree the review was biased/flawed, CSR will re-review application in the same council round. If we don’t agree, the official NIH appeals process remains available to all investigators.
Closure and Culture Change
Re-review? CSR Scientific Division Director discusses the issue with the reviewer, if appropriate. Actions might also include not inviting the reviewer to serve in the future.
Gabriel Fosu, Ph.D.
CSR Associate Director for Diversity and Workforce Development
g.fosu_assocdir@csr.nih.gov
Return to top
Broadening the Reviewer Pool to Diversify Peer Review Groups
Scientific Background
Demography
Geography
Career Stage
Peer Review Experience
There is a critical need for NIH to hear diverse perspectives to fulfill peer review’s mission of identifying the best, most novel science. The most effective, highest-quality review committees are broadly diverse in multiple dimensions including 1) scientific background and perspective; 2) demography; 3) geography; 4) career stage; 5) peer review experience. The value of diversity is evident in expectations of scientific review officers in recruiting for panels and in our investment of resources and public programs to more easily identify reviewers who are not already known to NIH.
Study section membership
The selection process for members of standing study sections is thorough and involves multiple levels of oversight and approval. Guidelines for staff in developing a slate of nominees explicitly addresses the value of diversity on panels and the advance planning process includes an analysis of current diversity on the panel, gaps, and plans to address those gaps
Currently, CSR is increasing focus on diversity of special emphasis panels (SEP) and has developed tools to allow SROs and supervisors to track diversity on SEPs.
Early Career Reviewer Program
CSR’s Early Career Reviewer (ECR) program offers early career scientists the opportunity to gain first-hand experience in peer review. In 2020, the program was expanded by requiring all standing study sections to include two ECRs at each meeting. This benefits ECRs by providing experience they can use in crafting their own grant applications and benefits CSR in that ECRs are more diverse. In 2021, 16.8% of ECRs were underrepresented minorities, compared to 10.3% for all CSR reviewers.
Development of tools to assist SROs in finding new, qualified reviewers
CSR has developed an internal database that allows scientific review officers (SRO) to more easily identify active scientists who might not already be known to NIH. The tool includes scientists funded through other government agencies and some non-profit organizations, those recommended by NIH staff, and those recommended by scientific societies.
CSR also regularly facilitates sharing among SROs of best practices in broader recruitment strategies.
Impact of CSR’s Efforts
Return to top
Exploring Changes to Review to Make it More Fair and Effective
Leading efforts to simplify review criteria
CSR Advisory Council working groups (Clinical Trials, Non-Clinical Trials) developed recommendations to simplify review criteria in a way that focuses reviewers on the importance and feasibility of the research proposed and in which reputation of the investigator and institution, in the global sense, do not have a place. The reorganization of the current five review criteria into three factors allow for the possibility of a multi-stage, partially-blinded review process in the future. These recommendations were considered and modified by NIH leadership; see details of the proposed review framework. NIH gathered additional input from the scientific community on the proposed changes through a Request for Information, which closed March 10, 2023. An overview of the input received from more than 800 individuals, scientific societies, and academic institutions has been shared on Review Matters and a content analysis posted.
Improving NRSA Fellowship Review
A CSR Advisory Council Working Group on Fellowship Review developed recommendations to modify the review criteria and change the required application materials for NRSA fellowship applications. The changes are expected to make the process more fair and more effective in identifying the next generation of promising scientists.
Review criteria will be modified to better focus reviewers on three key assessments: (1) potential of the applicant; (2) strength of the science; (3) quality of the training plan. In order to provide equal opportunity, accomplishments of the applicant will be evaluated in the context of their opportunities. Sponsor and institution will be evaluated with respect to the quality of the proposed science and quality of the training plan, minimizing effects of reputation. Changes in required application materials fit with the modified review criteria. Importantly, coursework grades will no longer be considered.
Details of the recommended changes can be found here. These recommendations were considered and approved by NIH leadership. An RFI was issued to gather additional input from the public and closed in June 2023. Responses to the RFI were supportive but underscored the need for ample communications and thoughtful guidance and training for investigators, reviewers, and NIH staff. An NIH Guide Notice will be issued in late 2023 with more information.
Collaboration with the Common Fund High Risk, High Reward (tR01) program
- CSR collaborated with the Common Fund to test a multi-stage review process in which information about the investigator and institution is provided after assessment of the abstract, aims, and research plan. Study section reviews took place in April 2021 and the process is being evaluated by an external contractor. Initial results are encouraging with a statistically significant increase in the demographic diversity of the applicant pool.
- CSR developed and supports the TRA Anonymization Check, an online tool that allows applicants to verify that their specific aims and research strategy sections do not contain identifying information.
CSR investigated the effects of anonymization on review outcomes
CSR conducted a large-scale study of the effect of redacting identifying information from grant applications on the preliminary assessment of scientific and technical merit (Nakamura et al, 2021).
Results
What does this mean
