Bias awareness and mitigation training for reviewers, chairs, and SROs

 
 
 
6,312
6,312 participants completed the training – for a training completion rate of 61 percent.
 
91%
91% of reviewers thought that the training substantially improved their ability to identify bias in peer review.
 
93%
93% of reviewers stated the training made them substantially more comfortable intervening against bias.
 
 
                                        Data for the Jan 2022 Advisory Council cycle.
 

CSR developed training specifically targeted toward mitigating the most common biases in the peer review process. The training includes personal testimonials, interactive exercises, and a narrated mock study section demonstrating techniques to intervene – all based on real-life examples. The training was developed with the assistance of a diverse CSR Advisory Council Working Group. Training has been provided to all CSR reviewers since August 2021.

 


Return to top

  

 

Reporting avenues for bias, unfair reviews, uncivil conduct on panels

 
 

CSR launched a widely-publicized reporting avenue for issues related to respectful interactions, bias or anything else that could affect the fairness of the review process. The reporting channel is open to all – investigators, reviewers, and program staff.

 

Investigate
Every allegation is carefully investigated by CSR senior management (Dr. Fosu and the Scientific Division Director)

 

Resolve
If we agree the review was biased/flawed, CSR will re-review application in the same council round. If we don’t agree, the official NIH appeals process remains available to all investigators.

 

Closure and Culture Change
Re-review? CSR Scientific Division Director discusses the issue with the reviewer, if appropriate. Actions might also include not inviting the reviewer to serve in the future.

Gabriel Fosu, Ph.D.
CSR Associate Director for Diversity and Workforce Development



Return to top
 

  

Broadening the reviewer pool to diversify peer review groups

 
 


Scientific Background


Demography


Geography


Career Stage


Peer Review Experience

 
 

There is a critical need for NIH to hear diverse perspectives to fulfill peer review’s mission of identifying the best, most novel science. The most effective, highest-quality review committees are broadly diverse in multiple dimensions including 1) scientific background and perspective; 2) demography; 3) geography; 4) career stage; 5) peer review experience. The value of diversity is evident in expectations of scientific review officers in recruiting for panels and in our investment of resources and public programs to more easily identify reviewers who are not already known to NIH.

 

Study section membership

The selection process for members of standing study sections is thorough and involves multiple levels of oversight and approval. Guidelines for staff in developing a slate of nominees explicitly addresses the value of diversity on panels and the advance planning process includes an analysis of current diversity on the panel, gaps, and plans to address those gaps

Currently, CSR is increasing focus on diversity of special emphasis panels (SEP) and has developed tools to allow SROs and supervisors to track diversity on SEPs.

Early Career Reviewer Program

CSR’s Early Career Reviewer (ECR) program offers early career scientists the opportunity to gain first-hand experience in peer review. In 2020, the program was expanded by requiring all standing study sections to include two ECRs at each meeting. This benefits ECRs by providing experience they can use in crafting their own grant applications and benefits CSR in that ECRs are more diverse. In 2021, 16.8% of ECRs were underrepresented minorities, compared to 10.3% for all CSR reviewers.

 

Development of tools to assist SROs in finding new, qualified reviewers

 

CSR has developed an internal database that allows scientific review officers (SRO) to more easily identify active scientists who might not already be known to NIH. The tool includes scientists funded through other government agencies and some non-profit organizations, those recommended by NIH staff, and those recommended by scientific societies.

CSR also regularly facilitates sharing among SROs of best practices in broader recruitment strategies.


Impact of CSR’s Efforts

 


Return to top

 

Exploring blinded review processes

 
 
Exploring blinded review processes photo
 
 

Collaboration with the Common Fund High Risk, High Reward (tR01) program

  • CSR collaborated with the Common Fund to test a multi-stage review process in which information about the investigator and institution is provided after assessment of the abstract, aims, and research plan. Study section reviews took place in April 2021 and the process is being evaluated by an external contractor. Initial results are encouraging with a statistically significant increase in the demographic diversity of the applicant pool.
  • CSR developed and supports the TRA Anonymization Check, an online tool that allows applicants to verify that their specific aims and research strategy sections do not contain identifying information.

Leading efforts to simplify review criteria

CSR Advisory Council working groups (Clinical Trials, Non-Clinical Trials) developed recommendations to simplify review criteria in a way that focuses reviewers on the importance and feasibility of the research proposed and in which reputation of the investigator and institution, in the global sense, do not have a place. The reorganization of the current five review criteria into three factors allow for the possibility of a multi-stage, partially-blinded review process in the future. These recommendations are under consideration by NIH leadership.

 

CSR investigated the effects of anonymization on review outcomes

CSR conducted a large-scale study of the effect of redacting identifying information from grant applications on the preliminary assessment of scientific and technical merit (Nakamura et al, 2021).

 
400 R01s from Black/African American (B/AA) principal investigators (PI), 400 from matched white PIs, 400 from randomly-selected white PIs
 
Full and redacted versions underwent simulated peer review
 
Data collection and analysis were performed by an external contractor using a preregistered plan

 

Results

 
Redaction did not affect scores of B/AA PIs but worsened scores of white PIs (significant, but small effect size)
 
21% of the time, reviewers identified the PI despite redaction. Removing these instances did not change the findings.
 

What does this mean

 
Isolating the effect of race is challenging due to secondary, linked variables (e.g., institutional prestige, investigator pedigree) that are tied to racial disparities. Redaction may have reduced these “halo effects”.
 
Findings support review approaches that diminish the role of PI identity in review.


Return to top