What Happens to Your NIH Grant Application Transcript August 8, 2018 Presentation by Dr. James Mack, Scientific Review Officer NIH Center for Scientific Review What happens to your NIH grant application after you submit it? In this presentation, we will give you an over view of the NIH peer review process for evaluating an NIH research grant application, such as R01 and similar grant applications. Slide 2: The Gateway for NIH Grant Applications The literal gateway for submitting research grant applications to the NIH is the Center for Scientific Review, CSR, which receives all NIH grant applications, refers them to NIH institutes and centers, and reviews the majority for scientific merit. Slide 3: Our Mission Our core mission is to ensure that NIH grant applications receive fair, independent, expert, and timely reviews free from inappropriate influences so that NIH can fund the most promising research. Slide 4: 24 NIH Institutes and Centers Fund Grants CSR serves ALL of the 24 funding institutes and centers of the NIH that fund grants. Slide 5: What Happens When We Get Your Application? What happens when we get your application? Slide 6: We Check Your Application for Being . . . In CSR’s Division of Receipt and Referral, scientific administrative staff conduct initial checks for compliance of your application in regard to its timeliness of submission, formatting, and completeness. After this validation, your application is examined for its scientific objective and methodology in preparation for making two critical decisions. Slide 7: Your Application Is Assigned . . . A cadre of CSR Referral Officers - scientist administrators – first assigns your application to a review group – either at CSR, where we review about 75% of NIH grant applications or to one of the NIH Institutes or Centers, which have their own review groups. Slide 8: CSR Divisions and Integrated Review Groups (IRGs) If your application will be reviewed at CSR, it will be assigned to one of 24 Integrated Review Groups of study sections and then assigned to an appropriate study section with the needed expertise. CSR’s organization of study sections and panels are based on broad thematic areas that span biomedical and behavioral sciences. Details can be found online, but as you see here, there are five major divisions. If you used the Assignment Request Form in the application to suggest we review your application in a specific study section, we will work to accommodate your request if appropriate. Slide 9: Your Application May Be Assigned to a Special Emphasis Panel Sometimes your application will be assigned to a special emphasis panel rather than a standing study section. This may happen when, for example, assignment to the best study section would generate a conflict of interest, or when the application has very unique subject matter, or when a standing study section that would normally review it has an unusually heavy workload. Slide 10: Applications Also Are Assigned to . . . Before your application is sent to a review group, scientific referral officers at CSR assign it to one or more of the funding Institute or Centers for potential funding based on their overall mission and guidelines and their specific programmatic mandates and interests. Slide 11: Your Scientific Review Officer Takes Charge The review of your application now becomes the responsibility of the Scientific Review Officer or SRO. The SRO has doctoral level expertise related to the science reviewed in their study section. They select reviewers based on their knowledge of the field with the assistance of NIH information systems that catalog expertise of scientists. The SRO manages confidentiality and conflicts of interest, schedules and manages the study section meetings and prepares written summary statements on peer findings of scientific merit. Slide 12: Your SRO Seeks Reviewers Who Are Recognized Authorities in their Field In recruiting researchers who are recognized authorities to serve as reviewers, your SRO seeks those with doctoral degrees or equivalent, who have demonstrated scientific expertise and research support - - mature judgment and breadth of experience in research and research administration, and who work effectively in groups with a reputation for impartiality. In addition, having appropriate diversity on the panel is an important goal. Each CSR standing Study Section has typically about 12-25 regular members who are primarily from academia, but also from biomedical industry and sometimes government research laboratories. Temporary members may be recruited to fulfill specific review needs. Slide 13: The SRO Selects a Chair Among the Panel Who . . . The SRO also selects a chair from among the panel who has a particularly strong scientific reputation and NIH funding track record, is a leader in his or her field, broad in perspective, communicates particularly well, and has particularly strong reputation for impartiality. Slide 14: Your SRO Works to Ensure the Integrity of Your Review Your SRO works to ensure the integrity of your review in the following kinds of ways. Slide 15: Confidentiality First and foremost, confidentiality is a central watch word of the review process at the NIH. Review materials and meeting proceedings are confidential information. Reviewers and NIH staff are the only ones permitted to access it. At the end of each meeting, reviewers must destroy or return all review-related materials. They are not allowed to file or archive them for their personal use. Reviewers should not discuss review proceedings with ANYONE except the SRO. Reviewers should not even discuss their reviews among themselves outside of the confines of the actual study section meeting. Questions concerning review proceedings should be referred ONLY to the SRO. Applicants must never communicate directly with any member of a study section about an application. Slide 16: Reviewer Conflicts of Interest (COI) SROs also work hard to manage reviewer conflicts of interest. What constitutes a reviewer conflict of interest? Institution affiliations that are common with the PI or any of the PI’s staff represent conflicts of interest. A reviewer is in conflict with reviewing any application submitted by a family member or close friend or submitted by a collaborator or key personnel. A reviewer generally is also put in conflict with reviewing an application from an investigator with whom he has a publicly acknowledged contentious scientific disagreement, suggesting potential personal biases. We take pains in considering any of these factors to avoid not only real conflicts but even the appearance of a conflict of interest. Slide 17: The SRO Assigns at Least Three Reviewers to an Application So, in preparing for the review meeting, the SRO will assign at least three reviewers from the panel to each application about 6 – 8 weeks in advance of the meeting. They draft and submit preliminary evaluations – critiques and scores - to the SRO about a week before the meeting. These preliminary impressions serve to plan and frame the discussion that will take place at the actual review meeting. Slide 18: What Happens at Your Review Meeting? What happens at your review meeting? Slide 19: The SRO Convenes the Study Section Meeting The SRO convenes the study section meeting. Slide 20: Your Application May Be Reviewed Electronically While face-to-face meetings are still considered the gold standard for CSR review, our meetings also are facilitated by auxiliary electronic tools including telephone-assisted, video-assisted, or online virtual meetings. These enhance the reach of expertise that can be brought into the meeting to enrich the environment for peer review. Slide 21: Role of Study Section Chair Whatever the format, while the SRO is the official government presence, management of the meeting process is shared with the SRO-designated chair. The SRO provides the chair ground rules for conducting the meeting and advice as needed in the course of the discussions. The chair guides and summarizes study section discussions, ensuring that all study section member opinions are given careful consideration and that only scientifically relevant issues are broached. Chairs should seek a balance between reasonable conciseness and thoroughness. Slide 22: Discussions Focus on the Best Applications Discussions focus on the best applications on the meeting docket. Reviewers typically discuss the top half of applications based on an average of preliminary overall impact scores from assigned reviewers. The panel will discuss any application, however, that a member, even one not assigned but not in conflict, wishes to discuss. Slide 23: Review Criteria What features receive scores in NIH review? The objective of peer review is to arrive at a sense of each application’s overall impact, which essentially is an assessment of the likelihood for the project exerting a sustained powerful influence on the research field involved. NIH utilizes five basic scored criteria for evaluating an R01 application that help reviewers build an assessment of its overall impact: Significance, Investigators(s), Innovation, Approach, and Environment. Critique templates direct reviewers to comment on each of the 5 criteria – discussing relative stronger and weaker points of each; then draft a summary supporting the reviewer’s sense of overall impact and its most important basis; and to give scores for each criterion and, ultimately, for Overall Impact. Scores range from 1 to 9 with 1 being the best score. In reviewing an application, each assigned reviewer decides for them self the relative weights of criteria. This may vary from one reviewer to another, but it is very critical both in writing of critiques and in discussion of applications, that reviewers be very open and straight forward about those elements that they consider to be the most telling in framing their sense of overall impact. Slide 24: Scoring Overall Impact In scoring the overall impact of an application a score between one and three typically means reviewers consider it a high impact proposal. Scores of 4 to 6 indicate moderate impact and seven to 9 indicate low impact. Typically, when an application scores in the moderate or low range investigators have concern either about the significance of the proposed science or perceive substantial weaknesses in approach, team of investigators or innovation. Slide 25: Additional Issues Considered in Assessing Impact There are some additional criteria that also may be factored into the overall impact of your application. These importantly are: provisions for handling protections of human subjects and data drawn from them; the appropriate use of vertebrate animals; consideration of sex as a biological variable for both of these; and management of biohazards. Any of these features that are relevant to the science must be convincingly outlined and detailed in a successful grant application so as not to detract from judgement of overall impact. Slide 26: Review for Rigor and Transparency Reviewers are currently asked under the Significance criterion for research grant applications whether there is a strong scientific premise for the project. In evaluating the scientific premise of a proposed project, the reviewer considers the rigor of the prior research that serves as the key support for the proposed project. And under Approach, reviewers are asked if there are strategies to ensure a robust and unbiased approach. NIH is moving away from the term “scientific premise” for applications that come in for due dates of January 25, 2019, and beyond. In evaluating these applications, reviewers still will be asked whether the prior research that serves as the key support for the proposed project is rigorous. You should check the grants.nih.gov Web site for the latest policies on how applications are assessed for rigor and transparency, including consideration of Relevant Biological Variables, Such as Sex. You also should make sure you understand whether or not your research is considered and reviewed as a clinical trial. Slide 27: Application Discussion In launching an application discussion, once conflicts have been excused from the room, the chair identifies the assigned reviewers then announce their preliminary scores orally to all around the table. Reviewer one then introduces the application and presents a short precis of its essence and then their initial critique. Reviewers two and three in turn highlight new issues in areas that strongly impacted their judgment of the application. Once the three assigned reviewers have had their opening say about the application, all members present at the table are invited to join the discussion of the application’s stronger and weaker points most relevant to overall impact. Slide 28: Discussion Objectives It is essential that all major issues that seem to inform judgment by any of the panel members be openly revealed in discussion. Consensus per se is not the objective in itself; rather, reviewers are expected to discuss all perspectives leading to judgment to overall impact of the application. Slide 29: Final Scoring Once the chair and the SRO sense that the discussion has reached its limit in revealing all major points of view, the chair will call the discussion to an end, then provide a short summary of the critical points brought forward, and then ask for final Overall Impact scores to be declared by the assigned reviewers. This defines a range for the rest of the panel who then each provide a private final overall impact score. At CSR, a member of the panel may vote outside of the range but they must declare their intention to do so. The number itself remains private. Voting outside the range could be based on: a scientific difference of opinion, different weighting of the review criteria, or a perceived mismatch between the words/score provided by the assigned reviewers. Slide 30: What Reviewers Look for in Applications So speaking in common words, what do reviewers generally look for in an application? They're certainly looking for a significant problem [RB([1]to be attacked in regard to either health – e.g. curing Alzheimer’s Disease – or in fundamental scientific mechanism – e.g. pathways of a-beta mediation of neuronal toxicity - with high impact attending a successful outcome of the research proposed. They look for ideas that are exciting to pursue. They look for clarity of thought, in planning and in writing. They hope that they can readily grasp the essence of what the applicant proposes, so it's important in writing to not assume too much about the background that reviewers bring to the table regarding your very specific subject. They expect to see realistic aims and timelines in reaching those aims. And they often are critical of applications that seem to be overly ambitious in attainments to expect within a limited span of time. They expect brevity with things that really are rather common knowledge, and they love clear easy to read figures in a grammatically polished presentation. For further reviewer insights that might help you prepare your application, go to the Insider’s guide to peer review at the URL on the screen. Slide 31: Summary Statement Summary statements are released for all of the applications that are on a study section’s docket. The summary statement composition provides scores from each of the assigned reviewer for each criterion; each reviewer’s final critique (which may have changed from the preliminary) and their administrative notes, if any. If your application is discussed, you also will receive an overall impact score. And you may receive a rank percentile of that score. In addition, you will receive a summary of the actual discussion, called a resume, as drafted by the SRO. Finally, you will see the panel’s final budget recommendations, which you should note, play NO ROLE in merit evaluation. Slide 32: Your Career Stage Is Considered One major thing to know is that your career stage is given some consideration in the review process. If you are a new investigator or early stage investigator on an R01 application, NIH notes that fact. At review meetings, applications of new investigators are clustered together with the encouragement that things related to grantsmanship or lesser experience of the investigator with the processes of NIH be not graded as critically as they might would be for a more experienced investigator. You should ensure that NIH correctly has your status as a new investigator by periodically checking your personal profile in the eRA Commons database. Slide 33: What Happens Next . . . In a second stage of review, the assigned institute or center’s advisory council makes funding recommendations based on the outcome of the initial review and the institute or center’s program priorities. The Institute Director makes the final decision. [audio clipped from original audio associated with the former slide #6] Slide: 34 Who Can Answer Your Questions? This presentation may have generated many questions. If you are in the process of preparing an application or soon will be, you may wonder then whom you can turn to at various stages for getting answers to important questions that might arise. Before you even submit your application, it's very appropriate for you to consider talking to a program officer at one of the NIH institutes that you would consider appropriate to sponsor your research. Or you may want to talk with a scientific review officer who manages the study section that you expect to be a good fit for reviewing your research. After you have submitted the application but before review has been completed, your principal point of contact with NIH should be the Scientific Review Officer of the panel designated to handle your application. After that review has been completed and you've received your score, your principal point of contact shifts to your assigned program officer. Slide 35: Key NIH Review and Grants Web Sites General information on peer review and other points of interest about the grants process can be found at the website URLs on the screen Slide 36: Check Out of FAQ Web page Also, CSR has created a new Frequently Asked Questions web site and video that you might want to check out. www.csr.nih.gov/faq Thank you for spending this time with us. We hope you found this presentation helpful and we wish you well as you prepare your NIH grant application. [RB([1]You should distinguish between a significant health problem, e.g. curing Alzheimer's disease, and a significant scientific problem the field, e.g. understanding mechanisms of a-beta mediated neuronal toxicity