CSR Overview


Scope of Review Operations at CSR

applications icon
77%
CSR reviews 77% of NIH proposals.
 
study sections icon
>200
In over 200 chartered or recurring Study Sections
 
budget icon
0.4%
CSR does this with less than 0.4% of the total NIH budget
 
reviewers icon
18,500
CSR engages 18,500 distinct reviewers
 
meetings icon
1,600
CSR conducts 1,600 annual review meetings
 
review officers icon
242
CSR employs 242 scientific review officers
 

CSR’s Mission: To see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so that NIH can fund the most promising research.

CSR Also Participates in the Review of These Initiatives and Inter-agency Collaborations


National Institutes of Health
NIH All-of-Us Program Reviews, plus Other Transaction Authority Reviews

National Institutes of Health
Brain Research through Advancing Innovative Neurotechnologies (BRAIN)

National Institutes of Health
NIGMS Maximizing Investigators’ Research Awards (MIRA)

National Institutes of Health
Native American Research Centers for Health (NARCH)

National Institutes of Health
Helping to End Addiction Long-termSM (HEAL) Initiative

Global Alliance for Chronic Disease

National Institutes of Health
Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI)/Common Fund

National Institutes of Health
Many institute PARS and multi-institute RFAs

Food & Drug Administration
Tobacco Center of Regulatory Science for Research Relevant to the Family Smoking Prevention and Tobacco Control Act

National Institutes of Health
Cancer Moonshot

National Institutes of Health
Alzheimer's Disease (AD)

National Institutes of Health
Office of Research on Women’s Health (ORWH) at NIH - Specialized Centers of Research Excellence (SCORE) on Sex Differences

National Institutes of Health
Fogarty International Center

National Institutes of Health
Office of the Director (OD)/Office of Research Infrastructure Programs (ORIP)

NIH, NSF & DOE
NIH-NSF and NIH-DOE
All USA-China Reviews

Ongoing Evaluations


Early Career Reviewer (ECR) Program


The Early Career Reviewer (ECR) Program was established in late 2011. The program exposes junior scientists to the peer review experience to make them more competitive as applicants and to enrich the pool of NIH reviewers.

  • 3,111 individuals have received ECR training and have served on study sections.
  • 19% of ECRs who have served on study sections are from under-represented racial/ethnic groups.
  • 47% of ECRs are women.
  • 143 former ECRs are now members of standing study sections. The program began only 8 years ago, focuses on new faculty, and time to tenure is typically 7 years. Thus this small percentage being appointed members of panels indicates progress and we expect this to grow with time.
early career reviewers icon

Analysis to be performed by CSR:

  1. Analysis of success rates for R01s submitted by ECRs compared to early career researchers, matched for relevant demographic and career characteristics
  2. Demographic analysis: gender, race, ethnicity, geographic distribution,and institution type for both the ECRs in our database and the ECRs recruited to review
  3. Survey ECRs to identify areas of program improvement.

Anonymization Study Progress to Date


CSR is working with an outside contractor to examine various potential sources of bias.

Potential biases being examined include race, gender, career stage, and institution assessment.

1,200 previously reviewed applications, both full and redacted, are being used for the assessment.

The experimental approach, results, and data analysis will be peer-reviewed and shared sometime in 2019.

Acknowledgment: Lee Mann & Richard Nakamura

Redaction of Application Information

 
  100%

Reviewer Recruitment

 
  100%

Assignments

 
  100%

Critiques Received

 
  91%

Data Gathering and Analysis

 
  Estimated Spring/Summer 2019

 


meeting group photo

 

Evaluating Panel Quality in Review (ENQUIRE)


New Framework for CSR Study Section Evaluations


Framework for Study Section Evaluations:

  1. Review of 12-20 scientific clusters, in a similar scientific area
  2. Assemble blue-ribbon working group of scientifically broad, senior scientists, with interests that span multiple study sections
  3. Provide information on current scientific guidelines for all study sections, titles, abstracts, specific aims, workload trends
  4. Provide enough time and guidance for meaningful evaluation and recommendation by the working group
  5. Ask 1 question designed to focus the discussion on science, not process

How well does the scientific scope of the study sections align with the current state of the science?


Recommend modifications if:

  • A study section is currently over- or under-subscribed
  • The scope of the science reviewed by a study section is not adequately representative of the scope of the field and emerging fields

Possible Actions for Restructuring Study Sections:

  • Change in scientific guidelines
  • Merge study sections
  • Create new study sections
  • Eliminate study sections
  • Move an area of science from one study section to another/others
  • Add emerging areas of science

Reviews, Restructuring, and Alignment


Examples of Better Scientific Alignment, Accommodation for Emerging Fields

Redistribute basic biomaterials/biomimetics/fluidics in neural systems from Biomaterials and Biointerfaces (BMBI) to Bioengineering of Neuroscience, Vision and Low Vision Technologies (BNVT) – CSR Advisory Council approved and CSR implemented in 2016.

Move emerging area of microphysiological systems from BMBI to form new study section called Cellular & Molecular Technologies (CMT) – CSR Advisory Council approved in 2016 and CSR implemented in 2017.

Expand scope of Biomedical Computing and Health Informatics (BCHI) to include extraction of data from electronic health records, clinical decision support systems, telemedicine, large image databases, image informatics to guide decision-making – CSR Advisory Council approved and CSR implemented in 2016.

Areas Evaluated to Date

Bioengineering (Pilot, 2015)

Basic Cancer Biology (2016)

Imaging Technologies (2017)

HIV/AIDS (2017)

Vision Sciences (2018)


Maintaining Scientific Alignment of Study Sections


Background
December 2017 – June 2018: Trans-NIH Working Group on Peer Review developed a robust study section evaluation process.

The Working Group, co-led by CSR and representatives from NIH institutes, considered data/metrics, multiple inputs and measures to design a process that could be practically implemented by CSR.

Outcome
Largely builds on CSR’s existing successful model of study section evaluations - external panel to evaluate science by examining workload trends, guidelines, random sample abstracts/specific aims, adds publication/bibliometric data.

Adds a process evaluation component - internal NIH panel to look at process issues, e.g., distribution across institutes, scoring patterns, reviewer and program officer surveys, discussions, rosters, New Investigator application/award rates, etc.

Evaluation Process

  • Combines strength of expert opinion and objective metrics
  • Stakeholder engagement – CSR Advisory Council, and NIH institutes
  • Continuous and systematic approach - Evaluate ~20% of the study sections each year (i.e. a study section is evaluated every 5 years).
  • Addresses both NIH Strategic Plan element to "Optimize approaches to inform funding decisions" and CSR’s mandate to continuously examine the function of study sections

Evaluation Progress

  • Clusters of study sections in two scientific areas were evaluated to pilot the process
  • External Scientific Evaluation Group Meeting:
    December 2018
  • Internal Process Evaluation Group Meeting:
    January 2019
  • Report to the CSR Advisory Council:
    March 2019
  • Iterative Approach – Continuous refinement/modification of process based on experience and feedback

Evaluation Timeline

Completed Cluster Formation

Completed Prioritization Of Clusters

Months 1-4 External Scientific Evaluation Panel

Month 5 Internal Process Evaluation Panel

Months 6-8 EAWG and CSR Advisory Council

Months 9-12 Implementation by CSR


Integrity of the Peer Review Process

peer review group photo

Critically important for all of us

We must maintain the public trust in the NIH’s stewardship of taxpayer dollars to support U.S. biomedical science research.

Confidentiality is critical for candor in discussion and evaluation, and thus impacts the very basis of the peer review process.

Ensuring integrity requires the support of the entire research community – investigators, reviewers, study section chairs, NIH staff, institutional officials.

NIH is taking this issue very seriously. There do not appear to be widespread problems, but increased reporting and action is a cultural change.

What is the NIH Doing? Reporting & Action

We follow up on every allegation.

Cases are referred to Office of Management Assessment (OMA) – independent of CSR - an investigative unit conducts fact-finding, investigations, and issues a report of findings.

Actions have included

  • withdrawal of application
  • removal of reviewers from peer review committees
  • notification of the investigator/reviewer’s institution, which has led to personnel actions
  • pursuing government-wide suspension and disbarment
  • referral to other agencies for criminal violations

Making Reporting Easier

To make reporting easier, all scientific review officers include reporting information in their signature line in email messages:

Integrity matters. Say something! For concerns or questions about possible violations of peer review integrity, please contact your Scientific Review Officer, or the CSR Review Integrity Officer at csrrio@mail.nih.gov, or the NIH Review Policy Officer at reviewpolicyofficer@mail.nih.gov. See the NIH Guide Notice on integrity in review.

Scientific Review Officers regularly discuss integrity and reporting avenues. This is done through pre-meeting training of all reviewers and in opening statements at review meetings.