CSR Data & Evaluations
CSR Overview
Scope of Review Operations at CSR
76%
CSR reviews 76% of NIH proposals.
>250
In over 250 chartered or recurring study sections.
0.4%
CSR does this with less than 0.4% of the total NIH budget.
~19,000
CSR engages approximately 19,000 distinct reviewers.
~1,200
CSR conducts approximately 1,200 review meetings each year.
>260
CSR employs >260 scientific review officers.
2023 Data
CSR Also Participates in the Review of These Initiatives and Inter-agency Collaborations
National Institutes of Health
NIH All-of-Us Program Reviews, plus Other Transaction Authority Reviews
National Institutes of Health
Brain Research through Advancing Innovative Neurotechnologies (BRAIN)
National Institutes of Health
NIGMS Maximizing Investigators’ Research Awards (MIRA)
National Institutes of Health
Native American Research Centers for Health (NARCH)
Global Alliance for Chronic Disease
Global Alliance for Chronic Disease
National Institutes of Health
All Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI)/Common Fund review
National Institutes of Health
Many IC PARS and multi-IC RFAs
Food & Drug Administration
FDA/Tobacco
National Institutes of Health
Cancer Moonshot
National Institutes of Health
Alzheimer's Disease (AD)
CSR reviewed 161 special initiatives in FY23
National Institutes of Health
Office of Research on Women’s Health (ORWH) at NIH - Specialized Centers of Research Excellence (SCORE) on Sex Differences
National Institutes of Health
All Fogarty International Center Reviews
National Institutes of Health
All Office of the Director (OD/Office of Research Infrastructure Programs (ORIP) Reviews
NIH, NSF & DOE
NIH-NSF and NIH-DOE
All USA-China Reviews
Return to top
Demographics of CSR Reviewers
Gender by Reviewer Type
| Gender | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | Distinct CSR-Reviewed Applicants | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| Female | 35.5% | 42.7% | 34.5% | 41.1% | 39.3% | 45.3% | 36.1% | 44.6% | 35.4% | 42.3% | 49.0% | 50.6% | 33.8% | 35.9% |
| Male | 63.3% | 54.9% | 64.0% | 55.9% | 60.2% | 54.4% | 63.1% | 53.3% | 64.0% | 57.7% | 48.5% | 46.1% | 59.3% | 55.5% |
| Unknown | 1.1% | 2.3% | 1.4% | 2.9% | 0.3% | 0.8% | 2.1% | 2.5% | 3.4% | 6.8% | 8.5% | |||
| Withheld | 0.1% | 0.1% | 0.1% | 0.1% | 0.1% | |||||||||
Ethnicity by Reviewer Type
| Ethnicity | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | Distinct CSR-Reviewed Applicants | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| Hispanic/Latino | 5.0% | 8.3% | 4.4% | 7.9% | 7.4% | 12.0% | 5.3% | 7.7% | 11.9% | 9.0% | 9.5% | 5.1% | 5.8% | |
| Non-Hispanic | 87.3% | 84.5% | 87.5% | 84.1% | 86.6% | 83.5% | 87.6% | 85.6% | 89.9% | 85.1% | 83.4% | 83.7% | 81.7% | 81.6% |
| Unknown | 2.8% | 2.1% | 3.2% | 3.0% | 2.3% | 0.5% | 1.7% | 1.3% | 1.6% | 6.9% | 6.3% | |||
| Withheld | 5.0% | 5.1% | 5.0% | 5.1% | 3.7% | 4.0% | 5.5% | 5.4% | 7.6% | 5.2% | 6.4% | 6.3% | ||
Cells are blank where numbers are less than 11 individuals.
Race by Reviewer Type
| Race | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | Distinct CSR-Reviewed Applicants | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| American Indian or Alaskan | 0.3% | 0.3% | 0.3% | 0.3% | 0.3% | 0.4% | 0.2% | 0.3% | 0.2% | 0.2% | ||||
| Asian | 20.8% | 25.1% | 20.4% | 24.2% | 21.1% | 23.5% | 23.5% | 28.3% | 13.8% | 12.9% | 24.8% | 31.8% | 21.9% | 24.1% |
| Black or African American | 2.7% | 4.1% | 2.2% | 3.9% | 4.5% | 5.4% | 2.9% | 3.9% | 7.1% | 6.5% | 2.7% | 3.3% | ||
| More than one race | 1.2% | 2.1% | 1.1% | 2.0% | 1.1% | 2.3% | 1.3% | 2.4% | 2.7% | 2.7% | 1.6% | 1.8% | ||
| Native Hawaiian or Pacific Islander | 0.1% | 0.1% | 0.2% | 0.1% | 0.1% | |||||||||
| White | 69.5% | 61.6% | 70.2% | 62.1% | 69.0% | 64.0% | 66.4% | 58.5% | 81.5% | 77.6% | 56.1% | 50.9% | 61.6% | 58.1% |
| Unknown | 1.0% | 1.2% | 1.3% | 1.8% | 0.7% | 0.5% | 0.7% | 1.6% | 5.8% | 6.2% | ||||
| Withheld | 4.5% | 5.5% | 4.4% | 5.6% | 3.1% | 4.1% | 5.2% | 5.9% | 8.8% | 6.4% | 6.1% | 6.3% | ||
Cells are blank where numbers are less than 11 individuals.
Under-Represented Minority Status by Reviewer Type
| URM | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | Distinct CSR-Reviewed Applicants | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| Yes | 8.5% | 13.4% | 7.4% | 12.9% | 12.8% | 18.5% | 9.0% | 12.5% | 7.9% | 16.4% | 18.2% | 16.7% | 8.5% | 9.8% |
| No | 82.8% | 79.5% | 83.5% | 79.1% | 80.7% | 77.4% | 82.9% | 80.8% | 87.3% | 80.6% | 73.3% | 76.3% | 78.3% | 77.3% |
| Unknown | 8.7% | 7.1% | 9.1% | 8.0% | 6.5% | 4.0% | 8.2% | 6.7% | 8.5% | 7.0% | 13.2% | 12.9% | ||
URM is defined as in NOT-OD-20-031; Blacks or African Americans, Hispanics or Latinos, American Indians or Alaska Natives, Native Hawaiians and other Pacific Islanders, or those reporting more than one race/ethnicity which includes any one of these.
Academic Title by Reviewer Type
| Academic Title | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| Professor | 50.0% | 39.4% | 48.4% | 40.0% | 76.6% | 60.4% | 40.0% | 26.1% | 87.8% | 83.6% | 2.3% | |
| Associate Professor | 30.3% | 33.0% | 31.5% | 32.7% | 17.8% | 33.1% | 36.8% | 35.4% | 10.9% | 20.0% | 2.5% | |
| Assistant Professor | 10.2% | 19.6% | 8.7% | 16.3% | 2.8% | 17.2% | 34.2% | 70.8% | 92.5% | |||
| Other | 9.6% | 8.0% | 11.4% | 11.1% | 5.4% | 3.7% | 6.0% | 4.3% | 6.9% | 5.5% | 6.9% | 4.9% |
Disability Stats by Reviewer Type
| Disability Stats | All Reviewers | SEP Reviewers | Chartered Members | Ad Hoc Reviewers | Chairs Chartered Meetings | *ECRs | Distinct CSR-Reviewed Applicants | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | 2018 | 2023 | |
| Yes | 1.5% | 1.4% | 1.4% | 1.6% | 1.2% | 1.3% | 1.7% | 1.1% | 2.1% | 1.2% | 1.5% | 1.7% | ||
| No | 93.3% | 94.5% | 93.2% | 93.4% | 94.1% | 95.7% | 93.4% | 95.6% | 96.3% | 98.0% | 93.8% | 96.1% | 88.6% | 88.8% |
| NULL/Missing | 1.3% | 1.0% | 1.5% | 1.6% | 0.8% | 0.8% | 0.2% | 5.7% | 5.7% | |||||
| Withheld | 4.0% | 3.1% | 3.8% | 3.4% | 3.9% | 2.7% | 4.2% | 3.1% | 4.1% | 2.6% | 4.1% | 3.8% | ||
*ECR = Early Career Reviewers; the program was substantially changed during the 2020 council year with revised criteria and increased recruitment to panels.
SEP = Special emphasis panel
Ad hoc reviewers = Data are provided for ad hoc reviewers serving on a standing study section.
CSR Contact PI Applicant = Demographic data for contact Principal Investigators for applications reviewed by CSR.
Cells are blank where numbers are less than 11 individuals.
2017 and 2022 refer to advisory council years.
Return to top
Demographics of CSR Scientific Staff
Gender of CSR scientific staff
Race and Ethnicity of CSR scientific staff
Data are for the second quarter of FY2022. These are the demographics of CSR staff in the Civilian Labor Force, Occupational Code 601. The vast majority of CSR staff in this category are scientific review officers.
Completed Evaluations
All 175 plus Study Section descriptions updated.
Added Scientific Shared Interest and Overlap Statements to all 175+ study sections.
100% redesign of CSR.NIH.GOV with improved search and Drupal content management system.
CSR Anonymization Study
CSR worked with an outside contractor to examine various potential sources of bias.
Potential biases being examined include race, gender, career stage, and institution assessment.
1,200 previously reviewed applications, both full and redacted, were used for the assessment.
The experimental approach, results, and data analysis will be peer-reviewed and shared in detail in early 2021.
We found:
- Redaction does not affect the scores for grants submitted by African-American investigators.
- Redaction slightly worsens the scores for grants submitted by White investigators. The effect size is small but statistically significant.
- Redaction was not very effective at preventing identification of the investigator; > 20% of reviewers correctly identified the investigator.
CSR Application Review Evaluations
Commensuration Bias Study
Do reviewers use criteria scores to derive overall impact scores differently for Black versus White applicants?
Outcome: Small but statistically significant commensuration bias, but these effects are very small and do not account for the much larger disparities in overall impact scores between applications from White and Black applicants. Science Advances 03 June 2020
Intra-IRG Ranking Study
Will reviewer rankings of the top 20% of applications across study sections within an IRG (CSR management cluster) reveal differences in the quality of science among the study sections?
Outcome: No large differences in quality evident among study sections. Rankings were also related to the reviewers’ familiarity with the science of each application and adjusting for reviewer familiarity further reduced the size of the differences in rankings between SRGs.
Half Point Pilot
Will allowing reviewers to use 0.5 increments
for scoring (e.g. 1.5, 2.0, 2.5) improve score compression?
Outcome: Half-point option did not significantly improve score compression for most study sections or significantly alter rank order of applications under evaluation. CSR will not implement.
Return to top
Ongoing Evaluations
How Does Meeting Format Affect Review?
Although CSR had prior experience with a range of formats for review meetings, the COVID-19 pandemic forced a move to entirely remote review meetings. To assess the impact of the change in meeting format on peer review, CSR conducted two surveys, one in 2020 and another in 2021, each involving approximately 8,000 reviewers, to evaluate the experience of the reviewers, determine if they have a particular preference for meeting format, and assess any potential impact on review outcomes and reviewer recruitment. We also issued a blog on the subject and collected and analyzed comments.
We found that in-person meetings were preferred by about 40% of the reviewers, a little
less than a third preferred virtual meetings, and about a third had no preference. A larger percentage of men, versus women, preferred in-person meetings, as well as those more established in their careers versus those in earlier career stages—see Boxes 1 and 2 below.
Reviewers rated the quality of in-person and virtual meetings similarly, and we found no significant change in score distributions between in-person and virtual meetings.
However, engagement appeared to suffer in virtual meetings. Nearly half of all reviewers (46%) reported having a shorter attention span during a virtual meeting and quantitative analyses also suggest that virtual meetings tend
to be longer. See Box 3 below for additional advantages and disadvantages of in-person versus virtual meetings.
In Fall 2022, CSR began again to hold some in-person meetings.
For standing panels and recurring special emphasis panels, we are holding one of three meetings per year in-person. Additionally, since Summer 2023, we’ve held 10 or more meetings each round in a hybrid format, where some reviewers attend in person and others virtually. The remainder of peer review meetings are held virtually. Multiple analyses of the effect of meeting format on meeting experience, characteristics of ad hoc reviewers recruited, and scoring behavior have been completed:
- Comparison of face-to-face, virtual, and hybrid meetings – Summer 2024 review meetings
- Initial analysis of hybrid meeting format – Summer 2023-Winter 2024 review meetings
- Comparison of face-to-face and virtual meetings - Summer 2023 review meetings
- Comparison of face-to-face and virtual meetings – Fall 2022
Both Formats (in-person, virtual) have pluses and minuses
Return to top
Early Career Reviewer (ECR) Program
The Early Career Reviewer (ECR) Program was established in late 2011. The program enriches the pool of NIH reviewers and gives scientists peer review experience that they can apply in crafting competitive applications.
- 8,032 individuals have received ECR training and have served on study sections.
- 17.9% of ECRs who have served on study sections are from under-represented racial/ethnic groups.
- 49.7% of ECRs are women.
- 1061 former ECRs have served as members of standing study sections or are currently members. The program began in late 2011, focuses on new faculty, and time to tenure is typically 7 years. Thus this small percentage being appointed members of panels indicates progress and we expect this to grow with time.
Data as of 9/3/2023
Analysis to be performed by CSR:
- Analysis of success rates for R01s submitted by ECRs compared to early career researchers, matched for relevant demographic and career characteristics
- Demographic analysis: gender, race, ethnicity, geographic distribution,and institution type for both the ECRs in our database and the ECRs recruited to review
- Survey ECRs to identify areas of program improvement
How Does Use of the Zoom Platform Affect Review Meetings?
Although CSR had prior experience with a range of formats for review meetings, the COVID-19 pandemic forced a move to entirely remote review meetings. CSR held > 600 review meetings using the Zoom platform from March to August 2020. All review meetings will continue to be held remotely at least through March 2021.
Moving to entirely virtual review meetings has given us the opportunity to collect a large amount of data to assess the effect of the Zoom review format on review process.
In the summer of 2020, we surveyed:
- 3,288 reviewers
- 128 scientific review officers
- 73 staff supporting review meetings
Performed quantitative analyses of meeting duration and roster composition (n = 143 meetings, matched to their in-person equivalent).
- In-person meeting
- In-person meeting + video conference
- Video assisted meeting
- Virtual meeting
- Telephone assisted meeting
Reviewers and SROs prefer in-person meetings but the majority found overall quality of review in Zoom meetings to be the same as or better than that in in-person meetings. Nearly half of all reviewers (46%) reported having a shorter attention span during a Zoom meeting versus in an in-person meeting and quantitative analyses also suggest that Zoom meetings tend to be longer.
Reviewer engagement will be a focal point as we continue to examine the use of Zoom for review meetings.
There was no substantial change in female or minority representation on rosters for meetings held by Zoom versus those held in person in previous review cycles. Career stage and proportion of new reviewers was also unaffected by the move to review by Zoom.
CSR is continuing evaluations of Zoom review meetings.
Return to top
Evaluating Panel Quality in Review (ENQUIRE)
CSR ENQUIRE
Science changes rapidly. Making sure that study sections change with the science is an ongoing challenge. CSR ENQUIRE integrates data and input from multiple stakeholders to determine whether changes in study section focus or scope are needed to facilitate the identification of high impact science, with special consideration of emerging science.
- Clusters of study sections are formed based on scientific topics (instead of CSR managerial units) for review via ENQUIRE
- Systematic, data-driven, continuous process – ~20% of CSR study sections are evaluated each year, every study section evaluated every 5 years
- Stakeholder input and involvement – external scientific community, extramural programs at NIH, CSR staff
- Iterative approach - continuous refinement of the process based on experience
- Critical to success – matching referral of applications and reviewer expertise to redefined scientific scope of study sections
Multiple Possible Actions Follow
- Change in scientific guidelines
- Merge study sections
- Create new study sections
- Redistribute areas across study sections
- Add emerging areas of science
- Eliminate study sections
Process Overview for Each Cluster of Study Sections
- Cluster Formation
- External Scientific Evaluation Panel
- Internal Process Evaluation Panel
- Approvals
- Implementation by CSR
Clusters Evaluated via ENQUIRE
Initiated in 2019 restructured and new study sections in place October 2020:
- Healthcare Delivery and Patient Outcomes
9 study sections evaluated, resulting in 11 new or modified study sections
View the report - Gastrointestinal, Renal, Endocrine Systems
11 study sections evaluated, resulting in 10 new or modified study sections
View the report - Cardiac, Vascular and Hematologic Sciences
10 study sections evaluated, resulting in 8 new or modified study sections
View the report - Functional/Cognitive Neuroscience
12 study sections evaluated, resulting in 11 new or modified study sections
View the report
Initiated in 2020 and in progress:
- Molecular and Cellular Basic Sciences
16 study sections being evaluated - Epidemiology and Population Sciences
10 study sections being evaluated - Oncology
11 study sections being evaluated
Return to top
Integrity of the Peer Review Process
Critically important for all of us
We must maintain the public trust in the NIH’s stewardship of taxpayer dollars to support U.S. biomedical science research.
Confidentiality is critical for candor in discussion and evaluation, and thus impacts the very basis of the peer review process.
Ensuring integrity requires the support of the entire research community – investigators, reviewers, study section chairs, NIH staff, institutional officials.
NIH is taking this issue very seriously. There do not appear to be widespread problems, but increased reporting and action is a cultural change.
What is the NIH Doing? Reporting & Action
We follow up on every allegation.
Cases are referred to Office of Management Assessment (OMA) – independent of CSR - an investigative unit conducts fact-finding, investigations, and issues a report of findings.
Actions have included
- withdrawal of application
- removal of reviewers from peer review committees
- notification of the investigator/reviewer’s institution, which has led to personnel actions
- pursuing government-wide suspension and disbarment
- referral to other agencies for criminal violations
Making Reporting Easier
To make reporting easier, all scientific review officers include reporting information in their signature line in email messages:
Integrity matters. Say something! For concerns or questions about possible violations of peer review integrity, please contact your Scientific Review Officer, or the CSR Review Integrity Officer at csrrio@mail.nih.gov, or the NIH Review Policy Officer at reviewpolicyofficer@mail.nih.gov. See the NIH Guide Notice on integrity in review.
Scientific Review Officers regularly discuss integrity and reporting avenues. This is done through pre-meeting training of all reviewers and in opening statements at review meetings.
