Review Process

I’ve read the definitions of Significance and Overall Impact but the two still seem rather similar. Can you provide some additional guidance?

Significance is a stand-alone assessment of the project’s goals in the context of the relevant field, and to a large extent assumes that the investigator(s), approach and environment are adequate to allow for successful completion of the aims of the project even if later discussion of each of these review criteria will identify problems. When reviewers assess the Overall Impact of an application they are expected to take into account the scored review criteria (e.g., significance, investigator(s), innovation, approach and environment) and the additional review criteria to judge the potential of the project to exert a sustained, powerful influence on the field. For more information, visit the Overall Impact versus Significance document.

When determining the Overall Impact score, should it equal the arithmetic mean of the scores for the scored review criteria?

Not necessarily. The Overall Impact score considers all scored review criteria as well as all applicable additional review criteria. In addition, an application does not need to be strong in all scored review criteria to be judged likely to have a major scientific impact. Therefore, it is possible for one or more review criteria to overshadow the other review criteria, thus driving the Overall Impact score up or down. Please remember that there is no formula to derive the overall impact score from the individual criterion scores. Reviewers are instructed to weigh the different criteria as appropriate for each application in deriving the Overall Impact score.

Is it possible for an application with numerous weaknesses in Approach to receive a very strong Overall Impact score?

Yes. No single review criterion (e.g., Approach) alone determines the Overall Impact score. A project may have numerous minor weaknesses that affect the score for Approach, yet still have a very strong Overall Impact score if the application is exceptionally strong in the other review criteria and the quality of the team and environment lend confidence that the project will have a major overall impact on the field. “Minor weaknesses” are defined as “addressable weaknesses that do not substantially lessen overall impact.”

Aren’t projects that address diseases of large public health importance (e.g., heart disease, cancer, autism or dementia) inherently significant? Should they automatically receive high marks for Significance?

Not necessarily. The Significance score reflects whether a project addresses an important problem or critical barrier to progress within the field. For example, while a project may generally address a devastating disease with high prevalence, the specific problem addressed in the project may be only tangentially related to the disease, the problem may not be very important for patients with the disease, the proposed work may duplicate already published reports, or the expected results may be unlikely to substantially change knowledge, concepts and/or practice in the field.

The definitions of Overall Impact and Significance refer to the project’s ability to exert a powerful influence or address an important problem within the research field(s) involved. I thought the goal of the NIH is to improve public health. What’s the difference between improving public health and addressing an important problem within a research field?

The mission of the NIH is to support research in pursuit of knowledge about the biology and behavior of living systems and to apply that knowledge to extend healthy life and reduce illness and disability. To accomplish this mission, the NIH supports biomedical and behavioral research representing a wide array of research fields as well as tool development, clinical trials and other projects in support of the biomedical research enterprise. In an effort to fairly evaluate scientific and technical merit through the peer review system of a broad range of applications (those that seek cures, not only for diabetes, heart disease, and autism, but also for the lesser recognized orphan diseases and those that ask basic biomedical questions), it is important that Significance and Overall Impact be evaluated within the context of the research field involved. NIH program staff and Institute leadership will evaluate each project’s relevance to their Institute mission in making funding decisions.

What should be considered in the Overall Impact score for fellowship (F) applications?

For fellowship applications (Fs),the overall impact score should reflect the reviewers’ assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific career in a health-related field, in consideration of the scored criteria (i.e., Fellowship Applicant, Sponsors/ Collaborators/Consultants, Research Training Plan, Training Potential and Institutional Environment & Commitment to Training) as well as all applicable additional review criteria.

What should be included in the Overall Impact paragraph?

The Overall Impact paragraph provides the reviewer with the opportunity of explaining how the Overall Impact score was derived (i.e., those factors that contributed to the score). If a project has a strong/weak overall impact score then the reviewer should highlight those scored criteria that contributed to the favorable/poor score. For example, if the potential significance of a study was so great as to overshadow a number of methodological weaknesses then this should be clearly stated. Likewise, if the design of the study is so flawed as to negate any potential significance and/or innovation of the study then this should be clearly stated. Importantly, the Overall Impact paragraph should provide a clear view into the reviewer’s thought process that led to his/her Overall Impact score. It is not intended to simply summarize and/or restate the strengths and weakness detailed in the critique.

How do you guard against a single reviewer having undue influence at the review?

Before a CSR review meeting, the Scientific Review Officer or SRO looks for instances where one reviewer’s scores or critiques are out of sync with the others. The SRO will notify the chair and assigned reviewers when this happens. If the differences are not resolved before the meeting, the chair will make sure the study section discusses them to see if they are well founded or not. Also, if a review discussion is one-sided, chairs are trained to ask questions and encourage other reviewers to join the discussion.

It is important to note that reviewers take their jobs seriously, and they routinely question each other when they feel a review is not balanced. In addition, all the reviewers in the room weigh what is said at the meeting and independently score the application. Finally, the SRO monitors the review meeting to ensure it is fair and unbiased. If it appears a reviewer is not meeting this standard, the SRO can pause the meeting to talk to the reviewer, conduct a re-review of the application at a later time, or take other appropriate actions.