OHTN Research Funding Review PROCESS

The mission of the OHTN is to improve the health and well-being of people living with and at risk of HIV in Ontario through a network that promotes research and evidence to drive change. To help achieve our mission, we fund research that: is relevant to one or more of the populations most affected by HIV in Ontario; has potential to have a measurable impact in the short to medium term (i.e., in the next 2 to 5 years); and is scientifically rigorous.

All proposals submitted to the OHTN for research grants or awards are evaluated for their relevance and potential impact, as well as their scientific and technical merit. Eligible proposals are assigned to scientific and community or research/knowledge user reviewers with appropriate expertise in the areas of the proposal.

Letter of Intent Phase

The Letter of Intent phase involves:

  • An administrative review completed by OHTN Research Funding Program staff upon submission to ensure that applications are complete and that they meet the requirements of the funding opportunity. Where possible, an attempt is made to address any administrative issues that would prevent eligibility.
  • A review of the relevance statement completed by OHTN Research Funding Program Staff, in consultation with the Ministry of Health and Long-Term Care AIDS Bureau, to ensure that the proposal is relevant to and aligned with the OHTN funding policy and its commitment to funding research with real-life impact, and the Ontario provincial HIV/AIDS Strategy and its commitment to evidence informed practice. A recommendation regarding the relevance is provided the review committee for consideration.
  • The review of the full LOI completed by an academic and community reviewer (members of the Scientific Review Committee or other external reviewers, as needed). The LOI review includes a preliminary assessment of the research approach to identify ways to clarify the intentions of the researcher and/or research team and guide applicants towards a successful application. The following components of the LOI are considered:
    • Relevance
    • Calibre and expertise of the applicant and/or research team
    • Meaningful stakeholder involvement
    • Integrated KTE plan
    • Potential for impact

Taking these components into consideration, the review committee identifies any suggestions for how the application could be improved and then holds a yes or no vote on whether the applicant should be invited to submit a full application. If the vote is split, further discussion will follow to establish consensus. Applicants should be notified of the results of their LOI within 8 weeks of the LOI submission deadline and will receive a summary of the discussions of the review committee, including a detailed summary of any suggestions for improving their proposal where applicable. For full submissions the Principal Applicant, title and amount requested must remain the same. Depending on the type of application, other components may also need to remain the same for a full submission – this is noted in the relevant funding guidelines.

Full Application Phase

All full submissions undergo another administrative review completed by OHTN Reserve Funding Program staff to ensure that applications are complete and that they meet the requirements of the funding opportunity. Where possible, an attempt is made to address any administrative issues that would prevent eligibility.

Eligible proposals are then assigned to scientific and community or research/knowledge user reviewers with expertise in the appropriate areas. Each application is independently reviewed by: (a) two scientific reviewers (primary / secondary) to evaluate the scientific and methodological aspects of the proposal (see scientific review criteria below), and (b) two community or research/knowledge user reviewers (primary / secondary) who will evaluate the relevance and impact to the community (see overall impact criteria below). Each reviewer rates five key elements (see below) and then provides an overall scientific or impact score.

At the scientific review meeting, the review begins with each of the four reviewers announcing their preliminary overall scores. The scientific review proceeds with the primary reviewer presenting the proposal and his/her overall scientific assessment (strengths and weaknesses) followed by the 2nd scientific review. The community or research/knowledge user reviewers presents their assessment of impact, with the primary reviewer presenting his/her overall assessment of the impact of the proposal (strengths and weaknesses) followed by the 2nd community or research/knowledge user review. After both the scientific and impact reviews have been presented, the Chair leads an open discussion with the larger committee about the relative merits of the team and the overall proposal. Following this discussion, the Chair re-polls the reviewers to see if they want to make any final adjustments to their scores. The scientific reviewers will come to consensus about the range of scores for the overall scientific review within which the other committee members will vote. The community or research/knowledge user reviewers undergo the same process.

Once the assigned reviewers have reached the range of scores for the overall science and impact, each member of the review committee votes within this range. In the event that a committee member feels that he/she must vote outside of the agreed-upon range, he/she makes this known to the Chair and committee. Each proposal receives final scientific and impact scores based on the average of all scores cast by the committee.

For funding decisions, the final two scientific and impact scores for each proposal is summed and averaged, and the scores for all proposals ranked from highest to lowest. Only proposals scoring less than or equal to 4.0 for their final score will be considered for funding.

Scientific Review Criteria

Scientific Reviewers will assess assigned proposals based on each of the five review criteria below to determine scientific and health / technical merit, and give a separate score for each (using the scoring chart below).

  1. Significance/Relevance: Does the proposal address an important and relevant problem or critical barrier to progress in the HIV research field in Ontario? Is the proposal relevant to the priority population(s)? How will the successful completion of the project/program of research improve treatments, services, or prevention interventions for those infected and at risk of HIV? Will the knowledge exchange approaches put research evidence into the hands of people who will use it? (Score out of 9)
  2. Investigator(s): Are the investigators and collaborators (if applicable) well suited to the proposal? If new investigators, do they have appropriate experience and training? If established investigators, do they have an ongoing record of accomplishments that have had a tangible impact? Have the applicants demonstrated and described a clear commitment to mentor, be mentored, work collaboratively and build capacity in the Ontario HIV sector? (Score out of 9)
  3. Innovation: Does the proposal challenge and seek to shift current research, clinical practice, or service paradigms by exploring novel theoretical concepts, approaches, methodologies, or interventions? Is a refinement, improvement, or new application of theoretical concepts, approaches, methodologies, or interventions warranted? (Score out of 9)
  4. Approach: Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the proposal? Does the proposal present potential problems, alternative strategies, and benchmarks for success presented? Does the proposal discuss and address any ethical risks and strategies to mitigate those risks? Has the investigator sought community input to develop the proposal, and will community feedback be an integral component of the research as it unfolds? Is capacity building an essential element of the proposal? Are the KTE methods and approaches proposed appropriate? (Score out of 9)
  5. Environment: Will the research environment in which the work will be conducted contribute to the probability of success? Are the organizational support, equipment and other physical resources available to the investigators adequate for the proposed activities? (Score out of 9)

Overall Impact Criteria

Community or Research/Knowledge User Reviewers will briefly describe and provide an overall impact score (Score out of 9) to reflect their assessment of:

  1. Relevance and Potential Impact to Priority Population: Rate the relevance of the proposed research proposal to address key issues in one or more of the populations most affected by HIV in Ontario (Score out of 9)
  2. What is the Potential Need for Change and Influence: What is the relative need for achieving the desired impact (i.e. will the proposed research have a substantial impact on a population that has unmet needs)? (Score out of 9)
  3. Timing for Potential Influence and Impact: What is the likelihood of the proposal having a short or medium term (i.e. in 2-5 years), sustained, tangible impact on the health and well-being of people living with and at risk of HIV in Ontario? (Score out of 9)
  4. How will the Potential Influence and Impact be Achieved: Are the proposed pathways to impact, the role of the proposed research in achieving impact and the proposed KTE methods clear and feasible? Are they likely to realize the expected impact and get research into the hands of those who need it most? (Score out of 9)
  5. How will the Potential Influence and Impact be Measured and Evaluated: Does the project team clearly describe how they will measure the potential for change and influence? How will they know it had the planned impact? How will they measure whether the information got into the hands of the targeted users and was used? (Score out of 9)

More on Impact

The OHTN is adopting the impact framework and five categories developed by the Canadian Academy of Health Sciences to measure returns on investment in health research.

Applications are expected to fulfill those categories that are appropriate to the research being proposed:

  1. Advancing Knowledge - includes new discoveries and breakthroughs from health research, and contributions to the scientific literature
  2. Research Capacity-Building - includes the development and enhancement of research skills in individuals and teams (personnel, activity funding, infrastructure)
  3. Informing Decision-Making - includes the impacts of research in the areas of science, public, clinical, and managerial decision making practice and policy
  4. Health-Impact - encompasses advances in prevention, diagnosis, treatment, and palliation when related to research (in groupings of health status, determinants of health, and health system performance)
  5. Broad Economic and Social Impacts - commercialization of discoveries, direct cost savings, and human capital gains

Note: Generating knowledge and contributing to the scientific literature is important but it is not enough of an impact on its own to qualify for funding. Proposals must also demonstrate impact in at least one of the other four categories. Ideally, the OHTN will fund research that, taken with other evidence, will lead to a new or improved intervention, program, service, treatment or policy.

Scoring Chart

Impact

Score

Descriptor

Additional Guidance on Strengths/Weaknesses

High

1

Exceptional

Exceptionally strong with essentially no weaknesses

2

Outstanding

Extremely strong with negligible weaknesses

3

Excellent

Very strong with only some minor weaknesses

Medium

4

Very Good

Strong but with numerous minor weaknesses

5

Good

Strong but with at least one moderate weakness

6

Satisfactory

Some strengths but also some moderate weaknesses

Low

7

Fair

Some strengths but with at least one major weakness

8

Marginal

A few strengths and a few major weaknesses

9

Poor

Very few strengths and numerous major weaknesses

Additional Information for Scoring Guidance Table

Non-numeric score options: NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed

Minor Weakness: An easily addressable weakness that does not substantially lessen impact

Moderate Weakness: A weakness that lessens impact

Major Weakness: A weakness that severely limits impact

Adapted from the U.S. National Institutes of Health Enhanced Review Criteria (NOT-OD-09-025)

Additional Review Criteria

All reviewers will pay special attention to the following additional items when assessing a proposal and its overall impact:

  1. Protections for Human Participants: For research that involves human participants, the committee will evaluate the proposal in accordance with the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans - Version 2. In particular, the committee should highlight any issues relating to: 1) vulnerability of participants, 2) potential risks to participants, 3) adequacy of protection against risks, 4) potential benefits to the participants and others, 5) importance of the knowledge to be gained, and 6) data and safety monitoring for clinical trials.
  2. Inclusion of participants of all genders, ages and ethno-racial backgrounds: The committee will evaluate the proposed research with respect to the appropriate inclusion of participants from groups that have historically been under-represented, such as racialized people, women, transgender people and children.
  3. Research involving the First Nations, Inuit, and Métis peoples of Canada: For proposals involving the First Nations, Inuit, and Métis peoples of Canada, the committee will evaluate the proposal in terms of its adherence to Tri-Council Policy Statement Version 2, Chapter 9, and to the Ownership, Control, Access and Possession (OCAP) Principles, which have been developed as an expression of self-determination in research. The key notions of those principles relate to the collective ownership of group information, control over research and information, management of access to their data, and physical possession of the data.
  4. Resubmissions: When reviewing a resubmitted proposal, the committee will evaluate the proposal as now presented, taking into consideration the responses to comments from the previous scientific review group and the resulting changes to the project.
  5. Renewals: When reviewing a proposal to renew funding, the committee will consider the progress made in the last funding period, and the short- and medium-term impacts achieved.
  6. Biohazards: Reviewers will assess whether materials or procedures proposed are potentially hazardous to research personnel and/or the environment, and if needed, determine whether adequate protection is proposed.

DID YOU
KNOW

More than 600 people registered to attend the 2012 OHTN Research Conference - Research with Real-Life Impact. Distinguished researchers from across Canada, the United States, Britain, New Zealand and Australia, delivered presentations.

FOLLOW
THE OHTN