New Frontiers in Research Fund Evaluation Summary
About the evaluation
The purpose of this evaluation was to provide New Frontiers in Research Fund (NFRF) leadership with evidence on the program’s progress to date, with focus on areas for improvement and optimization important for NFRF to meet its long-term objectives.
The evaluation’s scope included NFRF’s Exploration and Transformation streams. It focused on NFRF’s contributions to date to support interdisciplinary and high-risk/novel research, as well as related enablers and/or challenges in these areas.
About the funding
NFRF was created in 2018 with a mandate to support interdisciplinary and high-risk/novel research with anticipated high reward. NFRF was established as part of the federal government’s response to recommendations made in Canada’s Fundamental Science Review (Advisory Panel for the Review of Federal Support for Fundamental Science, 2017) to enhance federal support for research in Canada.
NFRF aims to fund interdisciplinary, high-risk, high-reward (HRHR), transformative research led by Canadian researchers, generating opportunities for Canadian researchers and building Canadian strength and leadership in transformative and interdisciplinary research.
Evaluation questions
The evaluation questions were:
- EQ1. To what extent is NFRF making progress toward advancing interdisciplinary research?
- EQ2. To what extent is NFRF making progress toward advancing high-risk/novel and anticipated high-reward research?
- EQ3. What is helping or hindering NFRF’s progress to fill the gaps it was created to address? What could be improved?
Key findings
- The evaluation findings clearly show that NFRF is closing an identified gap in funding in terms of support for interdisciplinary and HRHR research.
- NFRF’s design features, merit review criteria and processes are aligned with evidence-based recommendations in academic literature on assessment of interdisciplinary and HRHR research. Key informants describe NFRF as a home for interdisciplinary and high-risk research projects, due to the nature of its support and design.
- Both Exploration and Transformation streams have attracted and supported interdisciplinary research teams and projects that go beyond common/conventional research practice as reflected in the agencies’ core programs. This is most evident in terms of the disciplinary diversity of NFRF research teams, which clearly show higher levels of disciplinary diversity and novelty relative to research teams funded by core agency programs.
- These higher levels of disciplinary diversity and novelty are linked to NFRF’s role in enabling and incentivizing collaborations between researchers from different disciplinary backgrounds. There is also clear evidence that at least some of the collaborations fostered by NFRF are being sustained via subsequent joint applications for core funding.
- The topics of NFRF-funded projects also reflect higher diversity and novelty relative to those funded by core programs, in terms of the disciplinary perspectives reflected in proposals. This is important because of the objective to support interdisciplinary research, not only interdisciplinary teams. However, interdisciplinarity is more pronounced at the level of NFRF research teams than in funded research proposals by available measures. There is some variation in the extent to which cross-disciplinary perspectives appear to be integrated in the projects funded by NFRF.
- Some natural sciences and engineering disciplines show higher representation in NFRF-funded teams relative to their representation among core-funded teams. Social sciences and humanities show lower representation in NFRF. Differences observed in disciplinary representation appear to be due primarily to application rates.
- NFRF’s guidance and review criteria for Interdisciplinarity and High Risk were described by interview participants as being helpful. In addition, the quality of the multidisciplinary review panel discussions and of program staff support were identified as key strengths to maintain and build on. Challenges to interpreting and applying the criteria in practice were also identified, however. Some of these challenges are inherent to the complexity of interdisciplinary and high-risk research and the development of understanding of this type of research in the research community. The evaluation identified opportunities to further optimize NFRF’s processes in ways that can support applicants and reviewers to understand NFRF’s expectations and increase the program’s ability to achieve its intended long-term impact.
Recommendations
The report includes four recommendations for the program, summarized below. Additional details appear in the “Recommendations” section at the end of the report.
Recommendation 1. If supporting interdisciplinary and high-risk, high-reward research remain priorities for the government, the program warrants continuation. NFRF is meeting a need and addressing an identified gap in terms of support for these areas of research.
NFRF is well situated to address an identified gap and is considered highly relevant by members of the research community. NFRF is clearly contributing to closing that gap and is providing unique opportunities for Canadian researchers to pursue interdisciplinary and high-risk research with anticipated high reward.
Recommendation 2. Build on existing strengths in level of program support to reviewers and applicants.
NFRF is still unconventional among funding programs in terms of its emphasis on interdisciplinary and high-risk research. Maintaining high levels of program support is essential to facilitate understanding of NFRF’s expectations, as they are different from conventional funding programs. Normalizing these unique features of NFRF and allowing for learning in the research community are expected to take time and investment. Maintaining strong support for the deliberations of the multidisciplinary review panel is especially important.
Recommendation 3. Revisit the Interdisciplinarity, High-Risk and Feasibility merit criteria and scoring process to provide more clarity and consistency.
The selection of research projects that are most aligned with NFRF’s funding objectives will determine NFRF’s long-term outcomes. Continuing efforts to optimize the merit review criteria will support understanding of NFRF’s expectations and threshold levels for interdisciplinarity and risk (i.e., “how much is enough”). This will also ensure continued alignment of funded projects with NFRF’s mandate.
A primary point of focus is to improve the reliability of the merit review matrices, and consistency in how they are applied, as they are the primary instrument used by both applicants and reviewers. To this end, the program is encouraged to do the following:
- Refine the Interdisciplinarity criterion to provide more clarity and to support consistency in scoring. Options for consideration are documented in the report. They include reviewing the elements in the criterion to reduce the appearance of inconsistency and simplifying elements for scoring; and continuing to emphasize the distinction between interdisciplinary and multidisciplinary research, as a key point of distinction for NFRF versus other funding programs.
- Provide enhanced clarity for the High-Risk and Feasibility criteria. Perceived conflict between NFRF’s High-Risk and Feasibility criteria reflects interpretation challenges encountered by applicants and reviewers. This is partly because NFRF’s Feasibility criterion departs from the way feasibility is approached in merit review in conventional programs. Misapprehension of the Feasibility criterion may lead applicants to constrain the level of risk in their proposals. Review of the wording of the elements of the Feasibility criterion is recommended to clarify its intent, i.e., to ensure the project includes key ingredients to enable response to the unexpected, rather than expecting prediction and mitigation of all risks, given the uncertain nature of HRHR research projects.
- Revisit Exploration’s “Interdisciplinarity subcriterion: Fit to Program.” A pass/fail subcriterion titled “Interdisciplinarity: Fit to Program” was added to Exploration in 2019, aimed at reducing applications in which the proposed collaboration is not novel, i.e., the disciplines involved already have an established tradition of collaborating. The original wording specified fit to program in terms of a proposal’s interdisciplinarity and high-risk nature. A change in wording after 2019 to refer to interdisciplinary and/or high-risk should be revisited, as it could result in deviation from the original intent.
- Revisit the level of discretion offered to reviewers when using the criteria. Flexibility in the merit review of this type of research is needed to ensure the criteria and process are adaptive to diverse forms of interdisciplinary and HRHR research. However, too much flexibility can contribute to confusion. It may not be necessary to allow the current level of discretion while still allowing for flexibility. Options for consideration are documented in the report.
Recommendation 4. Continue to reinforce that the program is open to all research domains through program communications and documentation, and monitor participation and success rates by discipline.
Relative to their respective representation in core teams, the predominance of some natural sciences and engineering disciplines in NFRF teams and projects is clearly observed. In contrast, lower participation rates are observed for social sciences and humanities researchers. These differences seem to be mainly attributable to higher or lower application rates for researchers from specific fields of research (e.g., biomedical engineering), rather than differences in success rates. Multiple other factors are also expected to influence these trends (e.g., differences in characteristics of core programs in the respective domains).
It is important to recognize that NFRF does not have specific objectives for ensuring equal representation by domain, nor a specific expectation for representation of domains in NFRF to be proportional to their respective representation across core programs overall. Nonetheless, it is important to continue monitoring for the presence of any barriers specific to some research areas, as well as application and success rates—ensuring NFRF is open to all research domains.