NIH Peer Review Process Revealed Final Transcript 6/18/2010 Running Time 14 minutes and 23 seconds AUDIO (Music up) NARRATOR: 70% of the major discoveries and Nobel Prizes in medicine and physiology are attributed to scientists in the United States. Most are funded by the National Institutes of Health. Most applications for funding are reviewed through the Center for Scientific Review at NIH. TONI SCARPA, Director of the NIH Center for Scientific Review, says, Our mission statement is really to do a proper review. Proper means timely, and it means fair, and it means without influence. So that NIH can fund the best applications and the best research. NARRATOR: Each year, CSR recruits over 20,000 independent experts from the scientific community to review more than 80,000 NIH applications. But for some, the application process may be intimidating. KATE BENT, Chief of CSR’s Health Care Delivery and Methodology Integrated Review Group, says, Lots of people are at different stages when they submit their first grant application. But the first time is still always your first time, and you’re nervous, you don’t really know what to expect from this process. NARRATOR: Understanding our peer review process is essential to writing a successful application. This video will walk you through two fictional but realistic applications to help you better navigate the peer review process. TONI SCARPA: You see how thorough it is, and how the decision is based on scholarly merit, and science, and innovation. And that is what makes peer review second to none in this country. Change music; music up. NARRATOR: Applications come into CSR’s receipt and referral area, where referral officers match each application to a study section with the expertise to review it, and to one or more of the NIH Institutes or Centers that could potentially fund it. Each application is assigned to a Scientific Review Officer, like Vonda Smith and John Bowers – both PhD’s and former researchers. JOHN BOWERS: …the Scientific Review Officer …the most important thing we do is to recruit reviewers. NARRATOR: These reviewers include preeminent researchers and scientists from around the country. They perform the actual scientific peer review on all funding applications. VONDA SMITH: This is my job to make sure your application is fairly reviewed. I will recruit the expertise that is needed to cover this application. JOHN BOWERS: If you’ve got the right people giving you their best advice, that’s kind of nine-tenths of it. NARRATOR: Once the peer review committee has been recruited, your application will be assigned at least three reviewers. Before the meeting, these reviewers will read your application in detail, with an eye toward NIH’s core criteria – assigning scores for each and providing preliminary overall impact scores of one through nine. Reviewers will also consider issues, such as the safety and appropriateness of research involving human subjects and vertebrate animals. The three reviewers will also write critiques covering the review criteria, so when the peer review panel convenes they’re ready to discuss each application in detail. JOHN BOWERS: “I’m John Bowers, I’m your Scientific Review Officer for the day…” NARRATOR: The peer review meeting begins with a statement by the SRO, stressing the confidentiality of the review and integrity of the review process – and discussing other important issues. KEITH YAMAMOTO is chairing the mock study section for the video. He is the executive vice dean of the school of medicine at the University of California in San Francisco. He starts of the mock study section meeting saying: “Okay, let me call us all to order, thanks for coming everyone….” NARRATOR : Grant applications at CSR are reviewed in order of their preliminary scores to help ensure that scores are well calibrated. Reviewers begin with the best scored application KEITH YAMAMOTO: “All right, so we’ll start with the first application, and just a reminder, this is from a new investigator so our special considerations for NIH new investigators apply here. This is from Dr. Fred Nicaros from Standiford University defining nickel accumulation in Cronstadt’s disease. The reviewers are David Grainger, Brian Brian Hoffman and Martin Philbert, and we’ll start by going around to the three of you and getting your preliminary impact scores.” : DAVID GRAINGER, professor of bioengineering and pharmaceutical chemistry at the University of Utah, says, “I was less than enthusiastic with a score of 5.” KEITH YAMAMOTO: ‘ 5? Okay, Great. Brian? BRIAN HOFFMAN, professor of chemistry at Northwestern University says, “I was very enthusiastic and came out with a score of two.” KEITH YAMAMOTO: Martin Philbert? MARTIN PHILBERT, professor of toxicology and senior associate director of research, University of Michigan School of Public Health says, “Two.” KEITH YAMAMOTO: “5, 2, 2. David, give us a quick summary of the substance of the proposal. And then your review comments.” DAVID GRAINER: Yeah, this proposal uses high density micro and nano electrode arrays from a talented analytical chemist to determine distribution of nickel… DAVID GRANGER reflects back on the meeting, On this particular application, I was assigned to be the primary reviewer. That reviewer’s job is generally to introduce the application to the group, to provide a brief synopsis of the high points and some low points to underscore the impact of the application and then to proceed through the criteria for review, which had been stipulated by the reviewing process. DAVID GRANGER, is shown back in the meeting: The weaknesses though in innovation are that the electrode array methods have been applied to tissues by other groups with success, and there are limitations that are recognized, and the methodology, and the experience with the other groups, and their limitations is not documented by the PI. In terms of the environment, the strength of the environment is that it is very well suited. The PI is well instrumented and the resources in the environment and its collaboration’s are in fact very appropriate. The weaknesses of the environment is that available equipment for preparing samples and running percale gradients particularly is not evident. Access to fluorescents microscopy is detailed, but not adequately described. I found that the over all impact strength was in defining the intercellular distribution of the accumulated nickel in conditions of nickel overload, and in the direct approach of the successful elucidation of these towards a new therapy. So overall a balanced enthusiasm with the new investigator. I found very nice for this particular PI but I found numerous technical concerns. KEITH YAMAMOTO: Thanks, David. So Brian if we could move to any of your views that are complementary or contrasting, BRIAN HOFFMAN: “I think I probably under-weighted some of the flaws - didn’t pick up on them or didn’t weight them properly–because I was so really enamored with the strengths, and I appreciate his reservations, but I come from a different direction, because I was so taken with the methodology…” DAVID GRANGER, reflects on the meeting. After my review, then the secondary reviewer, Dr. Hoffman, tempered some of the negative points that I had brought up and proposed an alternative view. And proposed some other strengths that I didn’t see in the initial read and the initial review. KEITH YAMAMOTO also reflects on the meeting: Dr. Hoffman raised the points about the broad significance of this work that I think really began to turn the minds of the committee. MARTIN PHILBERT back in the meeting says, “I agree with Brian’s assessment. This is a highly innovative proposal from a junior investigator. It uses novel technologies that I think are going to provide great answers. More over I think the innovation is heightened by fact that virtually nothing is known about Cronstadt’s disease.” KEITH YAMAMOTO: “Thank you. So let’s move into a general discussion, and here we’re really looking for your comments on aspects of the application that you think will really drive our consideration of the scores. So, Gail?” GAIL ECKARDT, professor of medicine and head of the division of medical oncology at the University of Colorado Cancer Center says, “Yeah, I had a question just because it is a new investigator, it’s a complex system, and a as yet relatively uncharacterized disease. What to you think about the new investigator’s skills in terms of trouble-shooting, or at least having access to collaborators that may help in that respect?” DAVID GRANGER: “You’re right. He is a new investigator. But none the less in this application he’s surrounded himself with general competence to, I think allay my problems with any confidence, in his ability to go to his collaborative team and to adjust his bandwidth within his expertise and go to collaborators where it’s appropriate.” BRIAN HOFFMAN: “What he hasn’t brought to the table in his electric chemistry, I think he’ll gain and gather from his collaborators, and so it makes for a good team and a good match.” KEITH YAMAMOTO: “Any other comments? Maybe I could go back to the first reviewer and see how you now consider your scores in view of the comments around the table and the nature of the discussion. David?” DAVID GRANGER: “So I think I’m a bit too harsh with my 5, and I would be willing to reduce my score and increase my enthusiasm based on what I’ve heard from my collaborators and friends here at the table…I guess I’ll go to a 3.” KEITH YAMAMOTO: “Okay, so thank you, and Brian?” BRIAN HOFFMAN: “I’m no less enthusiastic about the methodology, but the framework in which its embedded, the experimental framework that David’s pointed out, the weaknesses, I have to come up a little bit, and I’ll meet him at 3.” KEITH YAMAMOTO: “Thanks Brian, so two 3’s – and Martin?” MARTIN PHILBERT: “Yeah, I’m really sold on the innovation here, and on the promise, so I’m going to stay at 2, especially since our comments are going to be communicated to this investigator.” KEITH YAMAMOTO: “Okay so thank you… so strong enthusiasm for the new investigator, some reservations about the methodology and how well it will apply to the disease model, so 2, 3 and 3... terrific so let’s all record our final scores” NARRATOR: At the end of each review, the rest of the panel confidentially scores the application. KEITH YAMAMOTO reflects on the discussion saying, The discussion really worked and did what it was supposed to do, which was that the group evolved in its thinking and reached a place where I think they felt quite positive overall about the application, strongly positive about the investigator. And felt that he would find his way. Back in the meeting, Keith Yamamoto says, “Okay so next we go to a revised application from Dr. Paco Varnishy, Falling Water Medical Center, this is antigen presenting Tumor Fusion Hybrid Immunotherapy of Malignant Hepatoma. The reviewers are Gail Eckhardt, Bill Cantz, Gina Petroni . Head of discussion is Martin Philbert again, so if we can go around and get your preliminary impact scores. Gail? ECKHARDT: You know I really struggled with this application so I gave it a 5. KEITH YAMAMOTO: Okay, 5. Bill. CANCE: I had significant concerns. I gave it a 6. KEITH YAMAMOTO: A 5 and a 6. Gina? PETRONI: On initial review I gave it a 4. KEITH YAMAMOTO: 4, 5, and 6. And Martin? MARTIN PHILBERT: 7 KEITH YAMAMOTO: Let’s go back and get your evaluations. Gail? GAIL ECKHARDT: “Well just to give you an overview this application is really based on the hypothesis of using tumor associated antigen-presenting cells to really develop effective cancer vaccines. And so really this group is looking at a fusion of both the APC’s and tumor cells, and they generate these basically by using a new fusion technique that is ultrasonic.’” You know, this is a proposal from a well-known investigator who really I think technically wrote a spectacular proposal. ‘Well this is really where I struggled with this grant application, because I think it’s technically…’ But I think as you move into it the problem was this was an approach that’s been used a million times. WILLIAM CANCE, chair of surgical oncology and chief of surgery at the Roswell Cancer Center, says, “I basically agree with Gail I had significant concerns. The disease is extraordinarily significant. There’s no question Hepitoma has very little clinical…successful clinical treatments, and a very experienced investigator, but the approach is not novel, and at the end of the trial, I’m not sure we’re going to be left with significant information…” MARTIN PHILBERT: It just occurs to me that they could have done more work in conveying the idea that this could actually speed up the disease. KEITH YAMAMOTO: Ok, so why don’t we then go back and open this up for general discussion, and from what we’ve heard from our reviewers any comments from other members of the committee? DAVID GRANGER: Clinical monitoring is a problem when you have such a heterologous population of patients coming in with various stages of disease, some with metastatic complications, some with comorbidity, some with immune competence issues because of their disease or because of the drugs that they’re on, and I don’t sense that this is controlled for that. And what are going to get out of that when we subject a population of this heterogeneity to this type of treatment. It just doesn’t… it doesn’t smack very scientific to me. KEITH YAMAMOTO reflects on the meeting, saying, The scientists are very serious about the way they look at the process. There’s not an immediate pass given for example to someone who’s famous or well established in the field – everybody has to come in with an application that’s considered meritorious. Back in the meeting, KEITH YAMAMOTO says, Ok, good. So why don’t we go back then and revisit the scores having had this discussion. Uh, Gail. GAIL ECKARDT: Well I think I’m gonna move my five to a six. David’s points are well taken I would agree that basically we’re stuck with the fact that it’s you know, it’s a method and a clinical application that has been tried without a lot of success. I’m going to move to a six. KEITH YAMAMOTO: Ok, thank you. Bill? WILLIAM CANCE: I’m gonna stay at a six for the reasons that Gail just described. KEITH YAMAMOTO: Ok, thank you. Gina? GINA PETRONI, professor of biostatistics at the University of Virginia Health System says, Clearly there are concerns about the scientific rationale for these studies and the information that will be gained from that at the end of the study, so I will move to a six. MARTIN PHILBERT: I have sufficient remaining concern about the tumor progression issue, and some other toxicity issues with major heteroatom compatibilities complexes. I’m going to stay at 7. (Music begins to build) KEITH YAMAMOTO: Thank you. So we have three sixes and a seven. JOHN BOWERS: Ok, so that’s our last application. We are finished. JOHN BOWERS then reflects on the meeting, saying, NIH peer review is really active scientists reviewing active scientists. JOHN BOWERS, back at the meeting, says, Thank you very much. You’ve been a great panel. JOHN BOWERS reflects on the meeting, saying,: At the end of the day, it’s their advice that we value the most. NARRATOR: The review panel does not determine funding. They score each application on its scientific merits. After the review meeting, the scientific review officer writes a summary of the discussion and releases the summary statement with the reviewer critiques to you and the assigned NIH Institute or center –who will make the funding decision. Phone rings DAN WRIGHT, Program Director for Hematology Research at the National Institute of Diabetes and Digestive and Kidney Diseases, says, Once the review is over, the interaction between the applicant and the NIH is through the program officers. SEYMORE GARTE, Director of CSR’s Division of Physiology and Pathological Sciences, says, The program officer will also interact with the applicant after the review process to give advice and guidance as to whether or not an application will be funded. And if not, how that application might be revised the next time so that it could be funded. NARRATION: NIH is looking for ground-breaking projects supported by great science -- science that will help to make giant strides in medical research. NIH is here to help you. Thank you for your contributions to the health of America and the world. We look forward to seeing your application. For more information on CSR and the NIH peer review process, visit our web site at www.csr.nih.gov. This video is a production of the Center for Scientific Review and the NIH Division of Events Management, Multi Media Department and Rocket Media Group Logos are shown for the U.S. Department of Health and Human Services, the National Institutes of Health and the Center for Scientific Review. 8