- I've read the definitions of Significance and Overall Impact but the two still seem rather similar. Can you provide some additional guidance?
- When determining the Overall Impact score, should it equal the arithmetic mean of the scores for the scored review criteria?
- Is it possible for an application with numerous weaknesses in Approach to receive a very strong Overall Impact score?
- Aren’t projects that address diseases of large public health importance (e.g., heart disease, cancer, autism or dementia) inherently significant? Should they automatically receive high marks for Significance?
- The definitions of Overall Impact and Significance refer to the project’s ability to exert a powerful influence or address an important problem within the research field(s) involved. I thought the goal of the NIH is to improve public health. What’s the difference between improving public health and addressing an important problem within a research field?
- What should be considered in the Overall Impact score for fellowship (F) applications?
- What should be included in the Overall Impact paragraph?
- I addressed the concerns from the prior critiques and my score got worse. Why?
- What are the chances that you’ll end up with the same reviewers on a Special Emphasis Panel (SEP) or Scientific Review Group (SRG) if you resubmit?
- How do you guard against a single reviewer having undue influence at the review?
- I don’t like the review group you put my application into. What can I do?
- What are the biggest problems reviewers find in applications?
- When will I receive word on my application?
- When will my application be reviewed and who will be on the panel?
- Whom do I call about my score and summary statement?
Significance is a stand-alone assessment of the project’s goals in the context of the relevant field, and to a large extent assumes that the investigator(s), approach and environment are adequate to allow for successful completion of the aims of the project even if later discussion of each of these review criteria will identify problems. When reviewers assess the Overall Impact of an application they are expected to take into account the scored review criteria (e.g., significance, investigator(s), innovation, approach and environment) and the additional review criteria to judge the potential of the project to exert a sustained, powerful influence on the field. For more information, visit the Overall Impact versus Significance document.
Not necessarily. The Overall Impact score considers all scored review criteria as well as all applicable additional review criteria. In addition, an application does not need to be strong in all scored review criteria to be judged likely to have a major scientific impact. Therefore, it is possible for one or more review criteria to overshadow the other review criteria, thus driving the Overall Impact score up or down. Please remember that there is no formula to derive the overall impact score from the individual criterion scores. Reviewers are instructed to weigh the different criteria as appropriate for each application in deriving the Overall Impact score.
Yes. No single review criterion (e.g., Approach) alone determines the Overall Impact score. A project may have numerous minor weaknesses that affect the score for Approach, yet still have a very strong Overall Impact score if the application is exceptionally strong in the other review criteria and the quality of the team and environment lend confidence that the project will have a major overall impact on the field. “Minor weaknesses” are defined as “addressable weaknesses that do not substantially lessen overall impact.”
Aren’t projects that address diseases of large public health importance (e.g., heart disease, cancer, autism or dementia) inherently significant? Should they automatically receive high marks for Significance?
Not necessarily. The Significance score reflects whether a project addresses an important problem or critical barrier to progress within the field. For example, while a project may generally address a devastating disease with high prevalence, the specific problem addressed in the project may be only tangentially related to the disease, the problem may not be very important for patients with the disease, the proposed work may duplicate already published reports, or the expected results may be unlikely to substantially change knowledge, concepts and/or practice in the field.
The definitions of Overall Impact and Significance refer to the project’s ability to exert a powerful influence or address an important problem within the research field(s) involved. I thought the goal of the NIH is to improve public health. What’s the difference between improving public health and addressing an important problem within a research field?
The mission of the NIH is to support research in pursuit of knowledge about the biology and behavior of living systems and to apply that knowledge to extend healthy life and reduce illness and disability. To accomplish this mission, the NIH supports biomedical and behavioral research representing a wide array of research fields as well as tool development, clinical trials and other projects in support of the biomedical research enterprise. In an effort to fairly evaluate scientific and technical merit through the peer review system of a broad range of applications (those that seek cures, not only for diabetes, heart disease, and autism, but also for the lesser recognized orphan diseases and those that ask basic biomedical questions), it is important that Significance and Overall Impact be evaluated within the context of the research field involved. NIH program staff and Institute leadership will evaluate each project’s relevance to their Institute mission in making funding decisions.
For fellowship applications (Fs),the overall impact score should reflect the reviewers’ assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific career in a health-related field, in consideration of the scored criteria (i.e., Fellowship Applicant, Sponsors/ Collaborators/Consultants, Research Training Plan, Training Potential and Institutional Environment & Commitment to Training) as well as all applicable additional review criteria.
The Overall Impact paragraph provides the reviewer with the opportunity of explaining how the Overall Impact score was derived (i.e., those factors that contributed to the score). If a project has a strong/weak overall impact score then the reviewer should highlight those scored criteria that contributed to the favorable/poor score. For example, if the potential significance of a study was so great as to overshadow a number of methodological weaknesses then this should be clearly stated. Likewise, if the design of the study is so flawed as to negate any potential significance and/or innovation of the study then this should be clearly stated. Importantly, the Overall Impact paragraph should provide a clear view into the reviewer’s thought process that led to his/her Overall Impact score. It is not intended to simply summarize and/or restate the strengths and weakness detailed in the critique.
Each time your application is submitted, it competes within a new set of applications, it may be reviewed by new reviewers, or one or more of the original reviewers may see new concerns. So there is no guarantee your score will improve. The best course of action is to discuss your review concerns with your Program Officer before you resubmit and then be as responsive as possible to them. Including additional data, if available, could be helpful. In any event, if issues with significance and overall impact remain, addressing methodological concerns will not very likely improve the score.
There are no guarantees that a PI will receive the same reviewers when they resubmit an application for review. Scientific Review Officers are not supposed to retain records of the review assignments. It is more likely that on a SRG you could receive the same reviewers since it is a standing panel with regular members that return, but on a SEP, members are recruited as needed and it is unlikely that the same reviewers will be recruited.
Before a CSR review meeting, the Scientific Review Officer or SRO looks for instances where one reviewer’s scores or critiques are out of sync with the others. The SRO will notify the chair and assigned reviewers when this happens. If the differences are not resolved before the meeting, the chair will make sure the study section discusses them to see if they are well founded or not. Also, if a review discussion is one-sided, chairs are trained to ask questions and encourage other reviewers to join the discussion.
It is important to note that reviewers take their jobs seriously, and they routinely question each other when they feel a review is not balanced. In addition, all the reviewers in the room weigh what is said at the meeting and independently score the application. Finally, the SRO monitors the review meeting to ensure it is fair and unbiased. If it appears a reviewer is not meeting this standard, the SRO can pause the meeting to talk to the reviewer, conduct a re-review of the application at a later time, or take other appropriate actions.
Contact the Chief of the integrated review group for the assigned study section. He or she will be able to explain to you why the assigned study section is the best fit for your application. CSR listens to PIs and considers their preferences in making application assignments. When you submit your application, you may (but need not) use the Assignment Request form to suggest up to three study sections. Helpful resources for finding a CSR study section are the study section guidelines, and the Assignment Request Tool. You can find both on our home page: https://public.csr.nih.gov.
You also can use the Assignment Request form in your application to tell us the types of expertise needed to appropriately review your grant application. You should not request specific individuals. In addition, you can use this form to identify individuals who may be in conflict with your application. NIH will evaluate the situation according to our conflict of interest standards. See the NIH conflict-of-interest page: https://grants.nih.gov/grants/policy/coi. Please note that being competitors is NOT a compelling reason.
If you suggest one or more study sections and we can find a good fit among them, we will assign the application there. If our referral professionals judge the fit to be poor, they work to find an appropriate alternative. Our paramount concern is having appropriate expertise on the panel where the application is reviewed. If you don’t understand, ask. In the great majority of cases, PI concerns about application assignments are resolved through communication.
Here is a list of the most frequent problems reviewers in CSR cite when they critique grant applications:
- Lack of new or original ideas
- Absence of an acceptable scientific rationale
- Lack of experience in the essential methodology
- Questionable reasoning in experimental approach
- Uncritical approach
- Diffuse, superficial, or unfocused research plan
- Lack of sufficient experimental detail
- Lack of knowledge of published relevant work
- Unrealistically large amount of work proposed
- Uncertainty concerning future directions
For more insight, check out the reviewer tips for applicants in the CSR Insider Guide to NIH Peer Review at http://www.csr.nih.gov/applicantresources/insider
Notification that CSR has assigned your application to a scientific review group and Institute should appear in your eRA Commons account within 2 weeks of the submission deadline. If this notification does not appear in this timeframe, please contact CSR’s Division of Receipt and Referral at CSRDRR@mail.nih.gov or 301-435-0715
After the review, your scores should appear in your Commons account within 3 business days, and for reviews managed by CSR your summary statement within 30 days. It is important to note that funding decisions are not made until after the relevant Institute or Center council makes its recommendations.
You should receive an assignment notification in your eRA Commons account within 2 weeks of your application's submission due date. Most applications are assigned to CSR study sections for review, while about 30% of the applications are reviewed by review groups organized by an NIH Institute or Center.
CSR Review Groups: Once you know the scientific review group assigned to your application, you may look up the meeting date and the review group roster on the CSR Web site.
Other NIH Review Groups: Rosters and meeting dates for scientific review groups organized by other NIH Institutes and Centers are available via the NIH Office of Extramural Research Web site. If you do not find your review group listed, contact the scientific review officer assigned to your application.
Contact the assigned Program Officer at the NIH Institute(s) listed on the notification posted in your eRA Commons account.