New Methods to Detect Bias in Peer Review

Challenge #1: New Methods to Detect Bias in Peer Review

SUMMARY OF CHALLENGE: The National Institutes of Health (NIH) is the nation’s medical research agency whose mission is to enhance health and reduce the burdens of illness and disability. To achieve this mission, the NIH has created a peer review system to help identify the most promising biomedical research. The Center for Scientific Review (CSR) is the gateway for NIH grant applications. CSR strives to ensure that all applications receive fair and impartial reviews, free of inappropriate influences. Recent research (Ginther et. al., 2011; 2012) has shown that African American researchers are less likely than White researchers to receive grant funding. CSR is interested in knowing if reviewer bias is contributing to this problem. New strategies or methods are needed for detecting such potential bias. See Federal Register Notice (FRN Doc.2014-10196) for more details.

CHALLENGE: CSR is looking for the best ideas for detecting possible bias in peer review. Submissions could include approaches, strategies, methodologies and/or measures that would be sensitive to detecting bias among reviewers based on race/ethnicity, gender, institutional affiliation, area of science, and/or amount of research experience of applicants. This is an ideation challenge. It is not necessary to produce or submit completed measures.

AWARDS: Prizes will be rewarded in two categories: Best Empirically-Based Submission and Most Creative Submission. For each category, a first prize of $10,000 and the second prize of $5,000 will be awarded. Any given submission can receive only one award.


Submission Period: May 5, 2014 through 11:59 pm (EST) on June 30, 2014

Judging Period: July 16, 2014 through August 29, 2014

Winners announced: September 2, 2014


After CSR receives the submissions, they will be evaluated in a two-stage process: (1) Technical merit will be evaluated for each submission’s potential to detect bias in peer review (High, Medium, Low potential impact) by a panel of experts in fields relevant to peer review and reviewer bias, and (2) High Impact submissions will be evaluated and rank-ordered based on the judging criteria (see judging criteria below) by a panel of judges comprised of federal employees who will recommend the winning entries. The final awards will be approved by the Director of the Center for Scientific Review.

The judging criteria for the best empirically based and most creative submissions are as follows:

Best Empirically-Based Submission
  • Theoretically based and/or hypothesis driven
  • Proposes an experimental design
  • Well-grounded in peer reviewed empirical literature
  • Proposes measurement methods
  • Feasibility of implementation
  • Related to the Peer Review Process
Most Creative Submission
  • Proposes novel concepts or translates existing concepts in a novel way
  • Challenges existing paradigms
  • The proposed project has potential to be translated for use in an experimental design
  • Creative ways to apply ideas
  • Implementation is feasible
  • Relates to the Peer Review Process

Registration and submission process: There are 3 steps you must follow.
  1. Review the Rules below and go to FRN Doc.2014-10196 for more detailed information.
  2. Complete the Challenge coversheet with your name and contact information.
  3. Do not provide identifying information on your submission materials. The review of your submission will be anonymous. Submit your idea to CSRDiversityPeerRev@mail.nih.gov. You may not submit your idea on the challenge.gov website. If you do not wish to apply online, send your materials to Office of the Director, Attention: Denise McGarrell, Center for Scientific Review, 6701 Rockledge Drive, Suite 3030, Bethesda, Maryland 20892.
Proposal acceptance or submission period: Idea proposals will be accepted beginning May 5, 2014 through 11:59 pm (EST) on June 30, 2014. Incomplete submissions will not be considered. All entries must be written in English.

Eligibility: The Challenge is open to any Contestant, defined as an individual or group of U.S. citizens, U.S. Territory residents or permanent residents of the United States who are 18 years of age or older (signed permission of a parent or guardian required if under 18 years of age). A Contestant may also be a community-based organization, school (private/public, elementary-college/university), non-profit organization, faith-based institution, philanthropic organization, local government, or small business. Proposed ideas may be incorporated into a Request for Applications (RFA), Request for Proposals (RFP) or an implemented study, but an award of a prize does not guarantee the proposed idea will be implemented.

Number of entries: Only one idea may be included in each submission. However, there are no limits to the number of ideas that one proposer or group of proposers may submit. If multiple versions of the same proposal/idea are submitted, CSR reserves the right to read only one proposal and will not consider the other submissions.

Finalist Selection and Notification: The winners will be selected by a two-stage process for each category. The decisions of the panel on all matters relating to the Challenge are final. CSR reserves the right to refuse any proposals that do not adhere to the rules or that would result in a violation of Federal law, regulation, or the policies of the DHHS, NIH or CSR.

For questions about this challenge, please contact Monica Basco, Center for Scientific Review, phone: 301-300-3839 or email: CSRDiversityPeerRev@mail.nih.gov.


Ginther DK et al. (2011). Race, ethnicity, and NIH research awards. Science, 333 (1015-1019).

Ginther DK, Haak LL, Schaffer WT, & Kington R. (2012). Are race, ethnicity, and medical school affiliation associated with NIH R01 type 1 award probability for physician investigators? Academic Medicine, 87 (11), 1516-1524.ences: