CSR Data & Evaluations

National Institutes of Health
NIH All-of-Us Program Reviews, plus Other Transaction Authority Reviews

National Institutes of Health
Brain Research through Advancing Innovative Neurotechnologies (BRAIN)

National Institutes of Health
NIGMS Maximizing Investigators’ Research Awards (MIRA)

National Institutes of Health
Native American Research Centers for Health (NARCH)

Global Alliance for Chronic Disease
Global Alliance for Chronic Disease

National Institutes of Health
All Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI)/Common Fund review

National Institutes of Health
Many IC PARS and multi-IC RFAs

Food & Drug Administration
FDA/Tobacco

National Institutes of Health
Cancer Moonshot

National Institutes of Health
Alzheimer's Disease (AD)

CSR reviewed 161 special initiatives in FY23

National Institutes of Health
Office of Research on Women’s Health (ORWH) at NIH - Specialized Centers of Research Excellence (SCORE) on Sex Differences

National Institutes of Health
All Fogarty International Center Reviews

National Institutes of Health
All Office of the Director (OD/Office of Research Infrastructure Programs (ORIP) Reviews

NIH, NSF & DOE
NIH-NSF and NIH-DOE
All USA-China Reviews

Although CSR had prior experience with a range of formats for review meetings, the COVID-19 pandemic forced a move to entirely remote review meetings. To assess the impact of the change in meeting format on peer review, CSR conducted two surveys, one in 2020 and another in 2021, each involving approximately 8,000 reviewers, to evaluate the experience of the reviewers, determine if they have a particular preference for meeting format, and assess any potential impact on review outcomes and reviewer recruitment. We also issued a blog on the subject and collected and analyzed comments.

We found that in-person meetings were preferred by about 40% of the reviewers, a little

less than a third preferred virtual meetings, and about a third had no preference. A larger percentage of men, versus women, preferred in-person meetings, as well as those more established in their careers versus those in earlier career stages—see Boxes 1 and 2 below.

Reviewers rated the quality of in-person and virtual meetings similarly, and we found no significant change in score distributions between in-person and virtual meetings.

However, engagement appeared to suffer in virtual meetings. Nearly half of all reviewers (46%) reported having a shorter attention span during a virtual meeting and quantitative analyses also suggest that virtual meetings tend

to be longer. See Box 3 below for additional advantages and disadvantages of in-person versus virtual meetings.

In Fall 2022, CSR began again to hold some in-person meetings.

For standing panels and recurring special emphasis panels, we are holding one of three meetings per year in-person. Additionally, since Summer 2023, we’ve held 10 or more meetings each round in a hybrid format, where some reviewers attend in person and others virtually. The remainder of peer review meetings are held virtually. Multiple analyses of the effect of meeting format on meeting experience, characteristics of ad hoc reviewers recruited, and scoring behavior have been completed:

• Comparison of face-to-face, virtual, and hybrid meetings – Summer 2024 review meetings
• Initial analysis of hybrid meeting format – Summer 2023-Winter 2024 review meetings
• Comparison of face-to-face and virtual meetings - Summer 2023 review meetings
• Comparison of face-to-face and virtual meetings – Fall 2022 

The Early Career Reviewer (ECR) Program was established in late 2011. The program enriches the pool of NIH reviewers and gives scientists peer review experience that they can apply in crafting competitive applications.

• 8,032 individuals have received ECR training and have served on study sections.
• 49.7% of ECRs are women.
• 1061 former ECRs have served as members of standing study sections or are currently members. The program began in late 2011, focuses on new faculty, and time to tenure is typically 7 years. Thus this small percentage being appointed members of panels indicates progress and we expect this to grow with time.

  Data as of 9/3/2023

Analysis to be performed by CSR:

1. Analysis of success rates for R01s submitted by ECRs compared to early career researchers, matched for relevant demographic and career characteristics
2. Demographic analysis: geographic distribution and institution type for both the ECRs in our database and the ECRs recruited to review
3. Survey ECRs to identify areas of program improvement

Although CSR had prior experience with a range of formats for review meetings, the COVID-19 pandemic forced a move to entirely remote review meetings. CSR held > 600 review meetings using the Zoom platform from March to August 2020. All review meetings will continue to be held remotely at least through March 2021.

Moving to entirely virtual review meetings has given us the opportunity to collect a large amount of data to assess the effect of the Zoom review format on review process.

In the summer of 2020, we surveyed:

• 3,288 reviewers
• 128 scientific review officers
• 73 staff supporting review meetings

Performed quantitative analyses of meeting duration and roster composition (n = 143 meetings, matched to their in-person equivalent).

Reviewers and SROs prefer in-person meetings but the majority found overall quality of review in Zoom meetings to be the same as or better than that in in-person meetings. Nearly half of all reviewers (46%) reported having a shorter attention span during a Zoom meeting versus in an in-person meeting and quantitative analyses also suggest that Zoom meetings tend to be longer.

Reviewer engagement will be a focal point as we continue to examine the use of Zoom for review meetings.

There was no substantial change in female or minority representation on rosters for meetings held by Zoom versus those held in person in previous review cycles. Career stage and proportion of new reviewers was also unaffected by the move to review by Zoom.

CSR is continuing evaluations of Zoom review meetings.

Detailed findings are available here:
2021
2020

Initiated in 2019 restructured and new study sections in place October 2020:

• Healthcare Delivery and Patient Outcomes
  9 study sections evaluated, resulting in 11 new or modified study sections
  View the report
• Gastrointestinal, Renal, Endocrine Systems
  11 study sections evaluated, resulting in 10 new or modified study sections
  View the report
• Cardiac, Vascular and Hematologic Sciences
  10 study sections evaluated, resulting in 8 new or modified study sections
  View the report
• Functional/Cognitive Neuroscience
  12 study sections evaluated, resulting in 11 new or modified study sections
  View the report

Initiated in 2020 and in progress:

• Molecular and Cellular Basic Sciences
  16 study sections being evaluated
• Epidemiology and Population Sciences
  10 study sections being evaluated
• Oncology
  11 study sections being evaluated

Critically important for all of us

We must maintain the public trust in the NIH’s stewardship of taxpayer dollars to support U.S. biomedical science research.

Confidentiality is critical for candor in discussion and evaluation, and thus impacts the very basis of the peer review process.

Ensuring integrity requires the support of the entire research community – investigators, reviewers, study section chairs, NIH staff, institutional officials.

NIH is taking this issue very seriously. There do not appear to be widespread problems, but increased reporting and action is a cultural change.

What is the NIH Doing? Reporting & Action

We follow up on every allegation.

Cases are referred to Office of Management Assessment (OMA) – independent of CSR - an investigative unit conducts fact-finding, investigations, and issues a report of findings.

Actions have included

• withdrawal of application
• removal of reviewers from peer review committees
• notification of the investigator/reviewer’s institution, which has led to personnel actions
• pursuing government-wide suspension and disbarment
• referral to other agencies for criminal violations

Making Reporting Easier

To make reporting easier, all scientific review officers include reporting information in their signature line in email messages:

Integrity matters. Say something! For concerns or questions about possible violations of peer review integrity, please contact your Scientific Review Officer, or the CSR Review Integrity Officer at csrrio@mail.nih.gov, or the NIH Review Policy Officer at reviewpolicyofficer@mail.nih.gov. See the NIH Guide Notice on integrity in review.

Scientific Review Officers regularly discuss integrity and reporting avenues. This is done through pre-meeting training of all reviewers and in opening statements at review meetings.