Evaluation of the New Frontiers in Research Fund—Final Report

Evaluation Division of the Social Sciences and Humanities Research Council and the Natural Sciences and Engineering Research Council

On this page

• Terms and acronyms used in this report
• Executive Summary
  • Recommendations summary
• Background
  • About the evaluation
• Methodology
  • Overview of design
  • Lines of evidence
• EQ1. To what extent is NFRF making progress toward advancing interdisciplinary research?
  • IDR in NFRF’s design
  • Extent of interdisciplinarity in NFRF research teams and projects
  • Extent to which new relationships are fostered, strengthened, and sustained as a result of involvement in NFRF-funded projects.
  • Nature and depth of integration of disciplines in proposed projects and in implementation of funded projects.
  • NFRF’s selection processes for IDR
• EQ2. To what extent is NFRF making progress toward advancing high-risk/novel and anticipated high-reward research?
  • NFRF Exploration’s selection processes for HRHR
  • Extent to which NFRF’s elements and process optimally support selection of high-reward research
  • Perspectives on outcomes of mature or complemented projects
• EQ3. What is helping or hindering NFRF’s progress to fill the gaps it was created to address? What could be improved?
  • Equity, Diversity and Inclusion
  • Official languages
  • Opportunities for ECRs
  • Smaller academic institutions
  • Summary of findings: NFRF’s contributions to advancing interdisciplinary and HRHR research
• Recommendations
• References
• Appendix A – Details on methods
  • Overview of design
  • Social network analysis and administrative data analysis
  • Key informant interviews
  • Focus groups
  • Literature and document reviews
• Appendix B – Summary of NFRF proposal review process
  • Exploration
  • Transformation

Terms and Acronyms Used in This Report^{Footnote 1}

Applicant(s)	Applicants that are associated with successful and unsuccessful applications (unless specified)
CIHR	Canadian Institutes of Health Research
Core / core programs	Flagship agency-specific programs across CIHR, the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). See Appendix A for list of core programs.
CRDC	Canadian Research and Development Classification
CRCC	Canada Research Coordinating Committee
Discipline(s)	Research disciplines following the CRDC’s field of research (FoR) group-level classification. In the CRDC classification, there are 44 unique disciplines. For more information see Canadian Research and Development Classification 2020.
Domain(s)	Research domains following the CRDC’s field of research (FoR) division-level classification. The CRDC has six divisions: natural sciences (NS); engineering and technology; medical health and life sciences (MHL); agriculture; social sciences (SocSci); and humanities and the arts. Each domain contains multiple disciplines.
ECR	Early career researcher
EDI	Equity, diversity and inclusion
FoR	Field of research
FO	Funding opportunity
Grantee(s)	Applicants that are associated with funded applications (i.e., grant recipients)
HRHR	High-risk, high-reward
IDR	Interdisciplinary research
LOI	Letter of intent
MHL	Medical, health and life sciences
NFRF	New Frontiers in Research Fund
NOI	Notice of intent
NPI	Nominated principal investigator
NS	Natural sciences
NSE	Natural sciences and engineering
NSERC	Natural Sciences and Engineering Research Council
PI	Principal investigator
Projects	Proposed research projects (both successful and unsuccessful, unless specified)
SNA	Social network analysis
SocSci	Social sciences
SRSR	Standing Committee on Science and Research
SSHRC	Social Sciences and Humanities Research Council
TAIPR	Tri-Agency Interdisciplinary Peer Review Committee
Team(s)	Research teams associated with successful and unsuccessful applications (unless specified)
TIPS	Tri-agency Institutional Programs Secretariat

Executive Summary

The New Frontiers in Research Fund (NFRF) was created in 2018 with a mandate to support interdisciplinary and high-risk/novel research with anticipated high reward. NFRF was established as part of the federal government’s response to recommendations made in the Fundamental Science Review (Advisory Panel for the Review of Federal Support for Fundamental Science, 2017) for enhancing federal support for this type of research in Canada. One of the recommendations was to support researchers who work across disciplines and research domains covered by each of the three federal research funding agencies’ mandates (i.e., social sciences (SocSci) and humanities; natural sciences and engineering (NSE); and health sciences). The Panel also recommended that the tri-agencies create and enhance structures for supporting high-risk, high-reward (HRHR) research. Recent reports from the Standing Committee on Science and Research (SRSR) (2022, 2023) highlight the continued importance of federal support to interdisciplinary research (IDR) to address broad and complex issues facing Canadian society.

Considering NFRF’s stage of development, the purpose of this evaluation was to provide NFRF’s leadership with evidence on NFRF’s progress to date, with focus on areas for improvement and optimization that are important for NFRF to meet its long-term objectives. The scope of the evaluation included NFRF’s Exploration and Transformation streams. The evaluation focused on NFRF’s contributions to date to support interdisciplinary and high-risk/novel research, as well as related enablers and/or challenges in these areas.

The evaluation questions were:

• EQ1. To what extent is NFRF making progress toward advancing interdisciplinary research?
• EQ2. To what extent is NFRF making progress toward advancing high-risk/novel and anticipated high-reward research?
• EQ3. What is helping or hindering NFRF’s progress to fill the gaps it was created to address? What could be improved?

EQ1 was identified as the priority for the evaluation.

The evaluation findings clearly show that NFRF is closing an identified gap in funding in terms of support for interdisciplinary and HRHR research.

NFRF’s design features, merit review criteria and processes are aligned with evidence-based recommendations in academic literature on assessment of interdisciplinary and HRHR research. Key informants describe NFRF as a home for interdisciplinary and high-risk research projects due to the nature of its support and design.

Both Exploration and Transformation streams have attracted and supported IDR teams and projects that go beyond common/conventional research practice as reflected in the agencies’ core programs. This is most evident in terms of the disciplinary diversity of NFRF research teams, which clearly show higher levels of disciplinary diversity and novelty relative to research teams funded by core agency programs.

These higher levels of disciplinary diversity and novelty are linked to NFRF’s role in enabling and incentivizing collaborations between researchers from different disciplinary backgrounds. There is also clear evidence that at least some of the collaborations fostered by NFRF are being sustained via subsequent joint applications for core funding.

The topics of NFRF-funded projects also reflect higher diversity and novelty relative to those funded by core programs, in terms of the disciplinary perspectives reflected in proposals. This is important because of the objective to support IDR, not only interdisciplinary teams. However, interdisciplinarity is more pronounced at the level of NFRF research teams than in funded research proposals by available measures. There is some variation in the extent to which cross-disciplinary perspectives appear to be integrated in the projects funded by NFRF.

Some NSE disciplines show higher representation in NFRF-funded teams relative to their representation among core-funded teams. SocSci and humanities show lower representation in NFRF. Differences observed in disciplinary representation appear to be due primarily to application rates.

NFRF’s guidance and review criteria for Interdisciplinarity and High-Risk were described by interview participants to be helpful. In addition, the quality of the multidisciplinary review panel discussions and program staff support were identified as key strengths to maintain and build on. Challenges to interpreting and applying the criteria in practice were also identified, however. Some of these challenges are inherent to the complexity of interdisciplinary and high-risk research and the development of understanding of this type of research in the research community. The evaluation identified opportunities to further optimize NFRF’s processes in ways that can support applicants and reviewers to understand NFRF’s expectations and increase NFRF’s ability to achieve its intended long-term impact.

Recommendations summary

This report includes four recommendations for the program, summarized next. Additional details are found in the “Recommendations” section at the end of the report.

Recommendation 1. If supporting interdisciplinary and high-risk, high-reward research remain priorities for the government, the program warrants continuation. NFRF is meeting a need and addressing an identified gap in terms of support for these areas of research.

NFRF is well situated to address an identified gap and is considered highly relevant by members of the research community. NFRF is clearly contributing to closing that gap, and is providing unique opportunities for Canadian researchers to pursue interdisciplinary and high-risk research with anticipated high reward.

Recommendation 2. Build on existing strengths in level of program support to reviewers and applicants.

NFRF is still unconventional among funding programs in terms of its emphasis on interdisciplinary and high-risk research. Normalizing the unique features of NFRF and allowing for learning in the research community are expected to take time and investment.

Maintaining high levels of program support is essential to facilitate understanding of NFRF’s expectations, as they are different from those of conventional funding programs, and support by program staff may need to continue at a high level. Maintaining strong support for the deliberations of the multidisciplinary review panel is especially important.

Recommendation 3. Revisit the Interdisciplinarity, High-Risk and Feasibility merit criteria and scoring process to provide more clarity and consistency.

The selection of research projects that are most aligned with NFRF’s funding objectives will determine NFRF’s long-term outcomes. Continuing efforts to optimize the merit review criteria will support understanding of NFRF’s expectations and threshold levels for interdisciplinarity and risk (i.e., how much is enough). This will also ensure continued alignment of funded projects with NFRF’s mandate.

A primary point of focus is to improve the reliability of the merit review matrices, and consistency in how they are applied, as they are the primary instrument used by both applicants and reviewers. To this end, the program is encouraged to:

• Refine the Interdisciplinarity criterion to provide more clarity and to support consistency in scoring. Options for consideration are documented in the report. They include reviewing the elements in the criterion to reduce the appearance of inconsistency, and simplifying elements for scoring, and continuing to emphasize the distinction between interdisciplinary and multidisciplinary research, as a key point of distinction for NFRF from other funding programs.
• Provide enhanced clarity for the High-Risk and Feasibility criteria. Perceived conflict between NFRF’s High-Risk and Feasibility criteria reflects challenges with interpretation by applicants and reviewers. This is partly because NFRF’s Feasibility criterion departs from the way feasibility is approached in merit review in conventional programs. Misapprehension of the Feasibility criterion may lead applicants to constrain the level of risk in their proposals. Review of the wording of the elements of the Feasibility criterion is recommended to clarify its intent, i.e., to ensure the project includes key ingredients to enable response to the unexpected, rather than expecting prediction and mitigation of all risks, given the uncertain nature of HRHR research projects.
• Revisit Exploration’s “Interdisciplinarity subcriterion: Fit to Program.” A pass/fail subcriterion titled “Interdisciplinarity: Fit to Program” was added to Exploration in 2019, aimed at reducing applications in which the proposed collaboration is not novel, i.e., the disciplines involved already “have an established tradition of collaborating.” The original wording specified fit to program in terms of a proposal’s interdisciplinarity and high-risk nature. A change in wording after 2019 to refer to interdisciplinary and/or high-risk should be revisited, as it could result in deviation from the original intent.
• Revisit the level of discretion offered to reviewers when using the criteria. Flexibility in the merit review of this type of research is needed to ensure that the criteria and process are adaptive to diverse forms of interdisciplinary and HRHR research. However, too much flexibility can contribute to confusion. It may not be necessary to allow the current level of discretion, while still allowing for flexibility. Options for consideration are documented in the report.

Recommendation 4. Continue to reinforce that the program is open to all research domains through program communications and documentation, and monitor participation and success rates by discipline.

Relative to their respective representation in core teams, the predominance of some NSE disciplines in NFRF teams and projects is clearly observed. In contrast, lower participation rates are observed for SocSci and humanities. These differences seem to be mainly attributable to higher or lower application rates for researchers from specific fields of research (e.g., biomedical engineering), rather than differences in success rates. Multiple other factors are also expected to influence these trends (e.g., differences in characteristics of core programs in the respective domains).

It is important to recognize that NFRF does not have specific objectives for ensuring equal representation by domain, nor a specific expectation for representation of domains in NFRF to be proportional to their respective representation across core programs overall. Nonetheless, it is important to continue to ensure that NFRF is open to all research domains, by monitoring for the presence of any barriers specific to some research areas, as well as application and success rates.

Background

About the evaluation

This report presents findings and recommendations from the first evaluation of the New Frontiers in Research Fund (NFRF). This evaluation is a legislative and policy requirement as per section 42.1 of the Financial Administration Act, and subsection 4.3.15.4 of the Policy on Results, which requires that all federal grants and contribution programs be evaluated on a five-year cycle. The evaluation’s scope includes NFRF’s Exploration and Transformation streams and covers the period 2018-22.

NFRF was established in 2018.^{Footnote 2} Considering NFRF’s stage of development, the purpose of this evaluation is formative: to provide NFRF’s leadership with evidence on NFRF’s progress to date, with focus on areas for improvement and optimization that are important for NFRF to meet its long-term objectives.

This evaluation focused on NFRF’s progress on meeting two key objectives of the Exploration and Transformation streams, namely to support interdisciplinary research (IDR), and high-risk/novel, high-reward research in Canada. The priority for the evaluation was NFRF’s progress in advancing IDR, as it was the most prominent topic raised by stakeholders during the design phase of the evaluation.

The evaluation questions were:

• EQ1. To what extent is NFRF making progress toward advancing interdisciplinary research?
• EQ2. To what extent is NFRF making progress toward advancing high-risk/novel and anticipated high-reward research?
• EQ3. What is helping or hindering NFRF’s progress to fill the gaps it was created to address? What could be improved?

About the New Frontiers in Research Fund

NFRF aims to fund interdisciplinary, high-risk, high-reward (HRHR), transformative research led by Canadian researchers, generating opportunities for Canadian researchers and building Canadian strength and leadership in transformative and IDR.^{Footnote 3}

NFRF was established as part of the federal government’s response to recommendations made in the Fundamental Science Review (Advisory Panel for the Review of Federal Support for Fundamental Science, 2017) for enhancing federal support for research in Canada. One of the recommendations was to establish new joint programs to support researchers who work across disciplines and research domains covered by each of the three federal research funding agencies’ mandates (i.e., social sciences (SocSci) and humanities; natural sciences and engineering (NSE); and health sciences). The Panel also recommended that the tri-agencies create and enhance structures for supporting HRHR research. The Canada Research Coordinating Committee (CRCC) subsequently conducted a national consultation, which informed the development of NFRF (SSHRC, 2018a).

Recent reports from the Standing Committee on Science and Research (SRSR) (2022, 2023) have documented progress since the Fundamental Science Review, made in part through joint programs for research that cross agency mandates (such as NFRF). These reports highlight the continued importance of federal support to IDR to address broad and complex issues facing Canadian society.

The Tri-agency Institutional Programs Secretariat (TIPS) administers NFRF on behalf of Canada’s federal research funding agencies: the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC).^{Footnote 4} The CRCC provides strategic direction for NFRF.

New Frontiers in Research Fund’s Exploration and Transformation Streams

NFRF includes three separate streams (Exploration, Transformation, International), and the ability to launch special calls to respond to emerging research areas. The Exploration and Transformation streams were in scope for this evaluation.

• Exploration generates opportunities for Canadian researchers to conduct innovative high-risk, high-reward, and interdisciplinary research. Exploration seeks to inspire projects that bring disciplines together beyond traditional disciplinary or common interdisciplinary approaches by research teams with the capacity to explore something new that might fail, but that has the potential for significant impact. Exploration provides two years of funding^{Footnote 5} and a maximum value of $250,000 per project ($125,000/year^{Footnote 6}).
• Transformation is designed to support large-scale, Canadian-led interdisciplinary projects that address a major challenge with the potential to realize real and lasting change (high-reward). Transformation looks for deep integration across disciplines and beyond usual disciplinary perspectives.^{Footnote 7} Transformation grants provide six years of funding and a value of $12 million to $24 million per project ($2 million to $4 million/year).

The timeframe in scope includes five competitions for Exploration and two for Transformation. Exploration and Transformation are distinct streams with their own aims and purpose. A total of 690 Exploration grants and 13 Transformation grants were awarded in these competitions, funding a total of 2,713 researchers. At the time of this evaluation, most projects funded through the early Exploration competitions had completed their grant terms, although had experienced special challenges due to the COVID-19 pandemic. Projects funded through the other streams were in progress. This is reflected in the approach to the evaluation questions and results below.

Methodology

Overview of design

A mixed methods design was used for this evaluation. Qualitative and quantitative methods were implemented in stages, allowing results to inform subsequent stages in an iterative way (Fetters et al., 2013). Nearing the end of the study, results from all lines of evidence were again compared for convergence, allowing for reassessment and further development of some results.

Interdisciplinarity is a multidimensional concept, both theoretically and in its practical applications. In order to assess the extent to which NFRF has made progress in advancing IDR, multiple indicators were used to ensure adequate coverage (content validity). Measures of diversity, novelty, distance and integration were used, drawn from review of NFRF’s program material and the literature on IDR.

Quantitative measures were used to assess diversity of disciplines evident in NFRF projects and research teams, novelty of collaborations, and the integration of disciplinary perspectives in research projects. Quantitative methods included social network analysis (SNA) and conventional statistics using agency administrative data. Qualitative methods were used to complement the quantitative assessment of diversity, novelty and integration, and for data on disciplinary distance in collaborations. Qualitative methods included key informant interviews and focus groups, analysis of secondary data from Exploration end of grant reports, NFRF’s program literature and other reports (e.g., Exploration co-chairs’ reports [SSHRC, 2019, 2020b, 2021, 2022]), and reviews of academic literature.

Lines of evidence

Social network analysis and administrative data analysis. SNA as well as other descriptive and inferential quantitative analyses were conducted using the agencies’ administrative data on NFRF and the agencies’ flagship programs across CIHR, NSERC and SSHRC. The flagship programs are referred to in the rest of this report as the agencies’ core funding programs. Administrative data included data from funded and unfunded grant applications, for example: the grant proposals and demographic data for applicants (nominated principal investigators [NPIs], principal investigators [PIs], co-PIs and co-applicants).^{Footnote 8} The administrative data were assembled into a database (N =156,890 grant proposals and n=83,845 unique researchers). A machine learning process supported by human verification was used to assign standardized disciplinary codes to the records in the dataset (for more detail, see Appendix A). Indices of disciplinary diversity and novelty were applied to NFRF researchers, teams and projects, and then compared to those in flagship programs across CIHR, NSERC and SSHRC (core programs).

Trends within core funding programs, as an aggregate, represent a proxy for the broader research funding system.^{Footnote 9} In other words, core programs represent a “business as usual” comparator against which NFRF’s progress to advance IDR was assessed. As one of the primary objectives behind NFRF’s development was to fill a gap in the agencies’ pre-existing programs for IDR (see above), the purpose of comparison is to infer the extent to which interdisciplinarity of NFRF projects may be above and beyond what already occurs in the agencies’ flagship programs.

Core program data were also used in an analysis of disciplinary representation in NFRF teams and projects. The core program data provided a reference population of researchers by discipline who participate in federal research funding programs generally.

It is important to note that the research system in Canada is multifaceted, with many interrelated, co-dependent parts. As such, differences observed between NFRF and core programs may be driven by multiple factors, including some that are external to NFRF (e.g., features of core funding programs, such as the size of research teams). Examples of other factors that may be contributing to findings from comparisons with core funding are noted in relevant sections of the report. These examples are illustrative but not exhaustive. General limitations of naturally occurring comparisons are further documented in Appendix A.

Key informant interviews. Forty (40) key informant interviews were conducted with grantees (15), unsuccessful applicants (9), as well as multidisciplinary review panel members (16) who participated in Exploration and/or Transformation competitions between 2018 and 2022.^{Footnote 10} Key informants were purposefully selected based on their knowledge of and experience as participants in NFRF. Interviews allowed for in-depth data collection from multiple perspectives on aspects related to NFRF’s grant proposal and merit review processes; dynamics of collaborations within NFRF research teams; nature of interdisciplinarity, risk and novelty in NFRF research projects; and implications of NFRF design elements for their work. Note that when the term “participants” is used in this report, it refers collectively to grantees, applicants and review panel members who participated in key informant interviews.

Focus groups. Two focus groups were held with NFRF multidisciplinary review panel members to aid with interpretation of results from other lines of evidence, including the key informant interviews and SNA.

Literature and document reviews. Two literature reviews were conducted. The first focused on academic and grey literature related to IDR (n=34 articles), and the second focused on HRHR and transformative research (n=45 articles). Articles were selected using a systematic search and screening process in Covidence.^{Footnote 11} The literature reviews helped contextualize NFRF’s definitions of these constructs to inform their operationalization in the evaluation, and to contextualize results.

A review of NFRF’s program literature included documentation of NFRF’s purpose, intended outcomes, and application and review processes. This included program web pages and reviewer manuals for all competitions in scope, NFRF’s performance measurement strategy, Exploration co-chairs’ reports from 2018 to 2021 (SSHRC, 2019, 2020b, 2021, 2022), review panel member ratings and comments, and Exploration end of grant reports (n=295). Appendix A provides more details on methods used in the evaluation.

EQ1. To what extent is NFRF making progress toward advancing interdisciplinary research?

This question considered the extent to which NFRF has attracted and supported IDR projects that go beyond common and/or conventional research practice as seen in the agencies’ core programs. Strengths and weaknesses in the program’s design that may affect its progress toward long-term objectives were also assessed. EQ1 was identified as the priority for the evaluation.

Key points from the academic literature. A few key points from the academic literature are important to understand key strengths and areas for improvement in NFRF’s design.

There is broad consensus that IDR is indispensable to addressing complex societal challenges (Boon & Van Baalen, 2019; Klein & Falk-Krzesinski, 2017; Villeneuve et al., 2020). IDR bring opportunities to address limitations of individual disciplinary viewpoints and to create new understanding, but only with meaningful contribution from each participating discipline (Ramadier, 2004; Rodela & Alašević, 2017; Villeneuve et al., 2020). To successfully integrate their diverse ideas into the research, team members need to learn new terminology, concepts, and skills from other disciplines (Nowell et al., 2022; Villeneuve et al., 2020). Although challenging, this integration is considered to drive the innovation needed to address complex problems.

In contrast to conventional research, which has traditionally involved single or adjacent discipline collaboration,^{Footnote 13} IDR can carry a higher cost related to the time, effort and resources needed to ensure sufficient contribution throughout the project from each discipline involved (Brown et al., 2015). Reaching a coherent understanding across diverse perspectives on the team is key to high-reward, and considered a defining characteristic of IDR. Some consider the extent of integration to be what distinguishes IDR from multidisciplinary research.^{Footnote 14} Given the effort required to integrate perspectives, it is not surprising that interdisciplinary collaborations do not occur automatically or easily (Lyall & Fletcher, 2013). IDR also carries unique risks for a researcher, such as those related to securing funding or career advancement (Boon & Van Baalen, 2019; Petts et al., 2008), and this is a reason for targeted funding to incentivize and support IDR.

Specific definitions of IDR vary in the literature. Broadly, IDR is described as an approach that integrates concepts, theories and practices from two or more disciplinary fields to advance fundamental understanding, to solve a complex problem or develop technological innovations (Leigh & Brown, 2021; Newman, 2023; Rodela & Alašević, 2017; Villeneuve et al., 2020). Novel collaborations, when they allow for different perspectives to be jointly applied to a problem, can spur innovation. Generally, interdisciplinary approaches to research are defined as those that integrate perspectives (e.g., theories) and/or methods from diverse academic disciplines in novel ways (Bark et al., 2016; Brennan & Rondón-Sulbarán, 2019; Newman, 2023).

The literature describes many challenges intrinsic to IDR, and IDR is often characterized as a form of HRHR research. IDR can be just as challenging for funders and merit reviewers as for researchers, due to its multiple dimensions, the variations in interdisciplinary approaches across research fields, and its divergence from traditional research practices and norms (Newman, 2023; Nowell et al., 2022; Von Wehrden et al., 2019). IDR projects vary in terms of the structure and dynamics (e.g., research practices) of the research team. This means that IDR projects are themselves very diverse; they vary based on the different combinations of disciplines in different projects (Hoffmann et al., 2017; Von Wehrden et al., 2017), and in the purpose, goals and context of each project (Lengwiler, 2006; Lyall, 2019). This leads authors to advise against overly rigid metrics for IDR assessment, and to advocate for some flexibility within otherwise structured and transparent frameworks.

The next section outlines findings on design elements of NFRF related to IDR. Subsequent sections present findings on the extent of interdisciplinarity (diversity, novelty and integration) in NFRF projects and research teams. The last section focuses on elements of NFRF’s merit review process.

Interdisciplinary research in the New Frontiers in Research Fund’s design

NFRF references IDR consistently throughout its program documentation.^{Footnote 15} NFRF’s descriptions of IDR align with defining characteristics in the literature, including the importance of diversity and novelty of disciplinary perspectives on a research team, and the integration of these perspectives in the research itself. NFRF provides flexibility in the merit review of proposals within a structured and transparent framework, which is also consistent with recommendations from the literature. Reports from review panel co-chairs, published after the first four Exploration competitions, further describe expectations for IDR specific to Exploration.

Overall, the three elements of integration, diversity and novelty are clearly embedded in NFRF’s guidance and merit criteria, reflecting the program’s evidence-based approach to IDR. For example, the NFRF Exploration program literature specifies involvement of multiple disciplines that do not traditionally collaborate to bring different perspectives to the research.^{Footnote 16} The focus on diversity and novelty of perspectives is consistent with a central argument for funding IDR, as noted above (i.e., that this drives innovation to address complex problems). NFRF also stresses the importance of integration of disciplinary perspectives, approaches, tools and/or techniques in the research. Exploration describes a lack of integration, which is characteristic of multidisciplinary research, to be inconsistent with its expectation for IDR. Similarly, Transformation distinguishes IDR as “not an amalgamation of disciplinary-specific approaches.” Both Exploration and Transformation also include a criterion that the research proposal be designed from the outset using an interdisciplinary perspective. Both contrast IDR “by design” from a project in which other disciplinary perspectives are “added on.”

The literature documents several challenges that researchers face to implement IDR successfully, which leads authors to describe IDR projects as higher risk. NFRF recognizes this through the inclusion of IDR elements in the High-Risk criterion for both Exploration and Transformation. Specifically, this is reflected in the reference to novel/uncommon combinations of disciplines in Exploration, and the integration of expertise across disciplines in Transformation.

NFRF also intentionally builds flexibility into the evaluation of IDR proposals, while also providing a structured and transparent framework.^{Footnote 17} Reviewers are asked to assess the merit of NFRF proposals using structured review matrices as a guide. IDR is built into the matrices as:

1. a multi-element pass/fail criterion specific to IDR
2. an element of the criterion for high risk^{Footnote 18}
3. a criterion labeled: “Interdisciplinarity subcriterion: Fit to Program” (Exploration only)

In terms of flexibility, NFRF’s instructions for the use of its review matrices clearly state that they are intended as a reference. The reviewer is invited to consider all elements, a subset of the elements, and/or relevant elements not listed in the matrix, at their discretion. Reviewers are also offered discretion in how they arrive at an overall rating. While the Interdisciplinarity criterion is pass/fail, the High-Risk criterion provides guidance for four rating levels: Exceptional, Very Good, Fair and Poor; reviewers may also assign ratings between each of those defined four levels, which allows them a seven-point scale to assign scores.^{Footnote 19}

Applications are reviewed by members of a multidisciplinary panel.^{Footnote 20} A panel meeting is held at which top-rated proposals are discussed by the panel to arrive at a final assessment.^{Footnote 21} Applications are assigned to members of the multidisciplinary panel in a way that allows for a range in disciplinary perspectives and breadth of expertise in the review. This is also consistent with the literature, in which authors recommend review panels with a broad range of disciplinary and knowledge backgrounds because they align better with IDR assessments than panels of experts focused within their respective disciplines (Huutoniemi, 2012). NFRF’s Transformation stream also prioritizes a consolidation step through panel discussion prior to finalizing assessment and selection.^{Footnote 22}

NFRF as a home for IDR. In key informant interviews, almost all applicant and grantee participants (23/25), as well as several review panel members (6/17), described NFRF as a home for IDR.^{Footnote 23} Participants used words like “embrace,” “promote” and “nurture” to describe NFRF’s approach to IDR. One grantee described it this way: “[NFRF] is a centre of gravity that we can all collect into” [NFRF grantee]. Another stated: “I just breathe interdisciplinary work.… It’s like one of those fish in water things. This was good water for me, this grant …” [NFRF grantee]. NFRF was described by some participants (6) to have pushed them to take IDR to a higher level, for example through more diverse and distant disciplinary collaborations, and by being challenged to develop a proposal for reviewers from multiple disciplines.

There was consensus in the focus groups that NFRF held a unique position in the Canadian funding ecosystem, and about the relevance and need for NFRF. Similarly, many interview participants (17) provided concrete examples for why funding for interdisciplinary work can be difficult to obtain from other funding programs. In contrast, participants described NFRF as a “perfect fit” with their research, and/or that NFRF allowed them to pursue a project that would not have been possible through other funding mechanisms. For example, one grantee stated: “… not just me but everyone I’ve met and talked with about NFRF together, we feel that this is the one opportunity. This is the opportunity for going outside your immediate area of expertise” [NFRF grantee]. Multiple participants (11) identified elements of NFRF’s design that help it address a gap in existing funding.^{Footnote 24} Examples included the “up front” emphasis on IDR in NFRF, and that it is “hardwired in” to the program in concrete ways throughout, such as in the proposal requirements, the merit criteria and processes, and the flexibility of the funding (e.g., in terms of eligible expenditures, extensions). Transformation’s funding amount^{Footnote 25} and duration^{Footnote 26} were also highlighted as enablers for IDR, as were NFRF’s relatively streamlined application and reporting requirements, and NFRF’s overall openness to risk. All of the above were raised as enablers for IDR.

The combined support for IDR and risk, in particular, was described as a key differentiator between NFRF and other agency programs that support IDR. For example, a reviewer noted that the Tri-Agency Interdisciplinary Peer Review Committee (TAIPR)^{Footnote 27} has now “allowed some of the other interdisciplinary aspects to be brought in but they are not necessarily high-risk” [NFRF reviewer]. Eleven participants identified interdisciplinary collaborations as a primary source of risk and potential reward for NFRF projects, particularly when the collaboration is with “distant” disciplines and/or there is an aim to pursue deep integration of the different disciplinary perspectives in the project. One of these participants directly linked IDR and risk in this way: “The boldness required to really work across disciplines and between them ... the boldness that comes with embracing failure and the real risk inherent in the program—these are truly intertwined. Interdisciplinary work done at this kind of level is risky. With the strongest proposals, the risk bubbles out often. The risk of working across disciplines in this truly authentic way” [NFRF reviewer]. This reviewer contrasted this “high level” IDR with more conservative/less risky proposals written from the perspective of one discipline, “with the other [disciplines] as add-ons.” Another described IDR and risk as interlinked with respect to challenging established assumptions or consensus: “J’aimerais bien que les gens pensent à autre chose.... C’est ce qui ne fait pas de consensus qui devient de la recherche. C’est pourquoi le risque est important” [NFRF reviewer]. As noted above, risks related to interdisciplinary work are amplified with the inclusion of more distant disciplines, as recognized in Exploration’s merit criterion for High-Risk.

Areas for improvement. Most participants were enthusiastic about NFRF’s support for IDR; however, they also identified several areas where support for IDR could be enhanced. Some areas for improvement pertain to NFRF’s selection processes, and these are outlined further below in the section on NFRF’s merit review of IDR. Two other areas are:

1. More space in the Exploration application. NFRF Exploration’s Interdisciplinarity merit criterion includes an element that requires that the project be designed from an interdisciplinary perspective. Eight participants explained that it is very difficult to integrate multiple ways of thinking from different disciplines in the space available on the Exploration application. Members of their team “speak different languages” and have different ways of thinking, even a different perception of risk: “because of our differences in training and approaches.” Similarly, page limits have been identified in the literature as a risk when set too low, because novel approaches and ideas can require more explanation. The risk when page limits are too low is that it can confer a relative advantage to more conventional proposals (NASA, 2018).
2. Facilitating knowledge sharing about effective IDR. Researchers new to IDR are challenged to learn quickly in the early phase of their projects. Various types of support were suggested to increase the likelihood that projects would achieve their objectives. For example, events to help new cohorts of grantees learn strategies from earlier cohorts, such as the kick-off meetings organized for new cohorts of Transformation grantees. This can be also important for Exploration grantees, considering the shorter timeframe of these grants. In the words of one participant: “there’s not a lot of framing or support for how to do convergent research like this. This is a new field within the last 10-12 years. So, there’s a lot we’re figuring out on the fly … it would be helpful to have more sharing around this because all of it is new.… It’s definitely like flying the plane while you’re still building it at the same time” [NFRF grantee].

Extent of interdisciplinarity in New Frontiers in Research Fund research teams and projects

Quantitative indices were used to measure the extent to which research teams include diverse expertise (i.e., across different disciplines and domains) and the extent to which disciplinary collaborations are novel, relative to the core.^{Footnote 28} For details on measurement see Appendix A.

Diversity across disciplines and domains. Overall, NFRF has a notably^{Footnote 29} larger proportion of teams with cross-disciplinary and cross-domain connections than found in core teams.

It is important to note that cross-disciplinary and cross-domain research is already established in the agencies’ flagship programs. Almost two-thirds (65%) of CIHR-funded teams include cross-domain expertise, as do about half of SSHRC-funded teams (46%). A smaller proportion of NSERC grants (26%) cross domains, although this is partly related to the proportion of individual researcher grants in NSERC^{Footnote 30} and partly an artifact of the Canadian Research and Development Classification (CRDC) system.^{Footnote 31} The existence of IDR in the core was also highlighted by interview participants, although not enabled in the way that NFRF does, as noted above.

NFRF teams are more diverse relative to teams funded by any of the individual agencies, as shown in Figure 1. Specifically, 79% of Exploration-funded are cross-domain, and all Transformation-funded are cross-domain,^{Footnote 32} compared to 37% of core-funded teams overall. Also, a larger proportion of Exploration- and Transformation-funded teams include expertise from more than two research domains in a single team, relative to core.

Figure 1 long description

Figure 1 shows a stacked bar graph that depicts the proportion of teams funded by each of core and NFRF grants based on the number of domains present in the team. For NFRF Transformation teams, 8% had three domains, 31% had four domains, and 62% had more than five or six domains. For NFRF Exploration teams, 21% had one domain, 39% had two domains, 28% had three domains, 11% had four domains, and 1% had five or six domains. For CIHR teams, 35% had one domain, 36% had two domains, 21% had three domains, 7% had four domains, and 1% had five or six domains. For SSHRC teams, 54% had one domain, 34% had two domains, 9% had three domains, 2% had four domains, and 1% had five or six domains. For core (overall) teams, 63% had one domain, 27% had two domains, 8% had three domains, 2% had four domains, and less than 1% had five or six domains. For NSERC teams. 74% had one domain, 22% had two domains, and 3% had three domains.

Figures 2, 3 and 4 show disciplinary representation within teams in the core, Exploration and Transformation, respectively. Each of the six CRDC domains are graphed in blue. The horizontal axis (X-axis) reflects the percentage of members within a team that have expertise in each domain. The vertical width of the graphic for each domain reflects the percentage of teams that reflect that composition. For example, the left side of each graphic shows the proportion of teams with 1% representation or less^{Footnote 35} (proportion of teams that have minimal expertise from that domain). The right side of each graphic shows the proportion of teams composed entirely of researchers from that domain (100% of members have expertise from that domain). A red dot in each figure shows the median percentage of teams, while the blue diamond shows the mean percentage of teams. These figures illustrate a striking difference in disciplinary composition of NFRF Exploration teams (Figure 3) and NFRF Transformation teams (Figure 4) relative to the core teams (Figure 2).

Figure 2 long description

Figure 2 shows a violin plot that depicts the distribution of disciplinary expertise within core-funded teams, and the proportion of team members with expertise in each domain: natural sciences; engineering; medical, health and life sciences; agriculture; social sciences; and humanities. The vertical width of the graphic for each domain reflects the percentage of teams that reflect that composition. The left side of the graph shows the proportion of teams that have minimal expertise from that domain, and the right side of each graph shows the proportion of teams composed entirely of researchers from that domain. A red dot in each figure shows the median percentage of teams, and the blue diamond shows the mean percentage of teams.

For natural sciences, the median is 100% and the mean is 88%. For engineering, the median is 100% and the mean is 88%. For medical, health, and life sciences, the median is 56% and the mean is 63%. For agriculture, the median is 100% and the mean is 79%. For social sciences, the median is 100% and the mean is 81%. For humanities, the median is 100% and the mean is 72%.

As shown in Figure 2 (above), teams funded by core are typically made up of researchers with expertise in the same domain, with the exception of MHL teams.^{Footnote 37} This is especially the case for teams with expertise in NS and engineering, which are widest at the right side (the 100% point on the X-axis). This is also the case, to a lesser extent, for teams with expertise in SocSci and humanities.

As shown in Figure 3 (below), a relatively small proportion of Exploration teams are teams where all/most members share expertise in the same domain. For Exploration, the graphics for all domains are wider towards the middle and left of the X-axis, indicating that most teams involve expertise from multiple domains.

Figure 3 long description

Figure 3 shows a violin plot that depicts the distribution of disciplinary expertise within Exploration-funded teams, and the proportion of team members with expertise in each domain: natural sciences; engineering; medical, health and life sciences; agriculture; social sciences; and humanities. The vertical width of the graphic for each domain reflects the percentage of teams that reflect that composition. The left side of the graph shows the proportion of teams that have minimal expertise from that domain, and the right side of each graph shows the proportion of teams composed entirely of researchers from that domain. A red dot in each figure shows the median percentage of teams, and the blue diamond shows the mean percentage of teams.

For natural sciences, the median is 50% and the mean is 54%. For engineering, the median is 50% and the mean is 45%. For medical, health and life sciences, the median is 33%and the mean is 42%. For agriculture, the median is 33% and the mean is 39%. For social sciences, the median is 40% and the mean is 45%. For humanities, the median is 33% and the mean is 36%.

As shown in Figure 4 (below), all Transformation-funded teams^{Footnote 38} include expertise from multiple domains. However, it is also clear that some domains, such as NS and, to a lesser extent SocSci and MHL, have higher representation within teams than other domains, such as humanities and agriculture, and to a lesser extent, engineering. These three domains are widest at the left-side of the X-axis, indicating they are a minority on most teams (in which they are present).

Figure 4 long description

Figure 4 shows a violin plot that depicts the distribution of disciplinary expertise within Transformation-funded teams, and the proportion of team members with expertise in each domain: natural sciences; engineering; medical, health and life sciences; agriculture; social sciences; and humanities. The vertical width of the graphic for each domain reflects the percentage of teams that reflect that composition. The left side of the graph shows the proportion of teams that have minimal expertise from that domain, and the right side of each graph shows the proportion of teams composed entirely of researchers from that domain. A red dot in each figure shows the median percentage of teams, and the blue diamond shows the mean percentage of teams.

For natural sciences, the median is 41% and the mean is 41%. For engineering, the median is 11% and the mean is 15%. For medical, health and life sciences, the median is 22% and the mean is 23%. For agriculture, the median is 5% and the mean is 8%. For social sciences, the median is 13% and the mean is 23%. For humanities, the median is 5% and the mean is 8%

In addition to higher connectedness across domains, NFRF teams also exhibit higher connectedness across disciplines relative to core teams. Network visuals below (Figure 5) show connectedness across disciplines in core- and NFRF-funded teams. The nodes (circles) in each diagram represent disciplines. Each node is marked with the discipline’s CRDC group-level code, and colour-coded by research domain (see legend).

In Figure 5, the size of the node is proportional to the number of observations in the network of that discipline. Disciplines that appear more often in the network have larger nodes. The thickness of the lines connecting the nodes is proportional to the frequency of connections between any two disciplines, reflecting the strength of ties between them.

As shown in the figure, teams funded by Exploration and Transformation reflect a broader profile of cross-disciplinary connections than teams funded by core grants. Network density^{Footnote 39} is much higher in NFRF teams, at 63% in Transformation and 46% in Exploration, compared to 20% in the core.^{Footnote 40}

Figure 5 long description

Figure 5 shows three network visuals of the representation of disciplines and connections across disciplines in core- and NFRF-funded teams.

In each panel, nodes represent disciplinary groups and are colour-coded by disciplinary domain: natural sciences (red); engineering (pink); medical, health and life sciences (green); agriculture (purple); social sciences (light blue); and humanities (dark blue). Node size reflects the relative prominence of each disciplinary group across teams. Grey lines connecting nodes indicate interdisciplinary collaboration, where two or more disciplinary groups are represented within the same team.

The core panel displays a network of interconnected disciplinary groups. The Exploration panel shows a network with a greater number of connections among disciplinary groups. The Transformation panel shows a densely connected network in which all disciplinary groups appear as larger nodes, indicating broader representation across teams.

A legend accompanies the figure and identifies the six disciplinary categories and their associated colours, which are used consistently across all three panels.

Novelty of disciplinary collaborations in NFRF. There is clear evidence that NFRF is funding novel disciplinary collaborations. Most Exploration and Transformation teams involve connections between disciplines that are not commonly observed in the core.

Findings on the novelty of disciplinary collaborations within teams provide important context for those presented above on diversity. As noted above, cross-domain teams already exist in the core. However, they tend to feature a limited set of disciplines. For example, the majority of CIHR-funded cross-domain teams (75%) are made up of a few pairings: 1) basic medicine and life sciences (MHL) with biological sciences (NS); 2) health sciences (MHL) with biological sciences (NS); and 3) health sciences (MHL) with psychology and cognitive sciences (SocSci).^{Footnote 43} In contrast, disciplinary collaborations within Exploration-funded teams are typically novel. Average novelty scores^{Footnote 44} for disciplinary collaborations in Exploration-funded teams are three to five^{Footnote 45} times higher than those of core-funded teams. Average novelty scores are even higher for Transformation teams, at five to 17 times higher than those of core teams.

The findings above focus on team composition. Similarly, the topics of NFRF-funded projects also reflect higher novelty, in terms of disciplinary perspectives represented, relative to those funded in the core. This indicates that novel disciplinary perspectives present on the team are also being incorporated into project topics (an indicator of integration in the design of the research project, going beyond just the team composition). On average, novelty scores for funded Exploration topics were two to seven^{Footnote 46} times higher than topics funded by core, and novelty scores for funded Transformation topics were four to 12^{Footnote 47} times higher than topics funded by core. While topic novelty is higher on average in NFRF than in the core by this measure, the difference is less than the difference in novelty of team compositions. In other words, the distinctiveness of NFRF is more pronounced at the team level and less pronounced at the level of research topics.^{Footnote 48}

Quantitative analysis shows variation in novelty scores across Exploration teams and Exploration projects. This supports descriptions by key informants (grantees and reviewers) indicating that there is variation in the extent to which funded research teams and projects have incorporated novel and/or distant collaborations. This suggests that a range of collaboration types have met NFRF’s threshold for interdisciplinarity. In key informant interviews, different perspectives were raised about the importance of integrating very novel and/or distant disciplinary perspectives in a research project. Key informants raised considerations that may limit or disincentivize researchers from working with others from more distant disciplines. Similarly, focus group participants described collaborations between closely related fields as “safe” collaborations, and a pragmatic approach to time-limited funding. For example, some participants asserted that collaborations between adjacent disciplines can be more efficient because they share tools and norms of practice, and/or that integration of more distant perspectives is not necessary for some research questions. Other key informants, as well as focus group participants, asserted the importance of NFRF encouraging unconventional partnerships, in the view that working with researchers from more distant disciplines and the integration of these perspectives in the research is essential for innovation. In short, the data reflect observations in the academic literature, i.e., that while IDR has defining characteristics (distance, novelty, integration), IDR is relatively new and still diverse in practice. NFRF’s threshold for IDR is revisited below in the section on merit review.

Representation of disciplines in research teams

Disciplinary representation emerged as a strong theme in key informant interviews. Almost half of interview participants (46%), including eight review panel members, raised questions about disciplinary representation, even though interview participants were not asked about it.^{Footnote 49} Nearly all observed that NFRF projects were predominantly health- and/or engineering-focused, that SocSci and humanities appears to be underrepresented, or both. Similarly, Exploration co-chairs’ reports from 2018 to 2021^{Footnote 50} highlighted a low representation of SocSci and humanities among applications. Within NFRF, descriptive statistics show that SocSci and humanities researchers, and projects with SocSci and humanities-related topics, have a lower representation relative to NSE and MHL.

As such, follow-up quantitative analyses were conducted as part of the evaluation. The evaluation assessed disciplinary representation in NFRF relative to the core population, and by funding status within NFRF (i.e., all proposals versus funded proposals; all applicants versus grantees). Representation in NFRF could be expected to follow patterns in the core, other factors being equal, given that NFRF is not designed to prioritize or allocate funding by domain in any way. As noted earlier, the core represents an aggregated dataset of researchers who participate in federal research funding programs and was used as a baseline reference for the relative size of population by domain. Where representation deviates from the core, a variety of factors may be at play.

In this section, results are reported in some detail due to the interest expressed by key informants and others consulted for the evaluation. Overall, NFRF appears to be attractive to researchers from some disciplines in NSE, visible in somewhat higher researcher participation rates, in more frequent majority membership on teams, and in project topics (see below). A variety of drivers may be at play, including some that are latent in the research system unrelated to NFRF. Trends are important to assess because of NFRF objectives to support diverse and novel research collaborations, which is aided by broad-based participation. Lastly, differences in success rates by discipline are slight, indicating no need for action other than continuation of monitoring already taking place.

It is evident that researchers from NSE represent a larger proportion of NFRF applicants relative to their representation among core applicants.^{Footnote 51} NSE researchers are more prevalent in teams applying to and funded by NFRF, and NSE researchers are more likely to form a majority within an NFRF team relative to other domains. In contrast, both SocSci and humanities and MHL researchers are more likely (relative to NSE) to form a minority on a team when present. SocSci and humanities researchers have lower overall representation among those applying to and funded by NFRF relative to their representation in the core.

The data indicates that differences in representation of domains in NFRF relative to core are primarily due to application rates. Figures 6 and 7 below show the differences in representation of researchers by domain in Exploration and Transformation relative to core.

Figure 6 long description

Figure 6 shows a bar graph of the percentage difference in representation of domains among applicants for Exploration, relative to core applicants. For natural sciences, it is a positive 13% difference. For engineering, it is a positive 10% difference. For agriculture and medical, health and life sciences, it is a 0% difference. For humanities, it is a negative 6% difference. For social sciences, it is a negative 11% difference.

Figure 7 long description

Figure 7 shows a bar graph of the percentage difference in representation of domains among applicants for Transformation, relative to core applicants. For natural sciences, it is a positive 11% difference. For engineering, it is a positive 6% difference. For agriculture and medical, health and life sciences, it is a positive 1% difference. For humanities, it is a negative 6% difference. For social sciences, it is a negative 8% difference.

Of note, the higher representation of NSE researchers is largely due to high participation rates by researchers from some, but not all, NSE disciplines. Most notably, researchers with expertise in computer sciences, chemical sciences, and medical and biomedical engineering show high participation rates in NFRF. Other disciplines within NSE are not showing higher participation rates in NFRF (e.g., environmental engineering). Almost three-quarters of teams applying to Exploration include researchers with NS expertise and, of these teams, 62% were predominantly or entirely^{Footnote 52} made up of NS researchers.^{Footnote 53} Nearly half of teams applying to Exploration (45%) and nearly three-quarters of those applying to Transformation (73%) include researchers with engineering expertise. However, in contrast to NS, engineering researchers are more likely to be minority members of a team.^{Footnote 54}

Representation of MHL researchers among applicants to NFRF was almost identical to their representation among core applicants. About one-quarter of Exploration, Transformation and core applicants have MHL expertise. Yet, at the team level, researchers with MHL expertise are present in a relatively larger proportion of teams applying to Exploration (42%), and especially Transformation (76%), compared to core (32%). Researchers from MHL tend to be somewhat spread out across teams applying to NFRF (see figures 2, 3 and 4). Unlike NS researchers, MHL researchers typically represent half or less of the members on the team.

Although researchers with SocSci backgrounds have somewhat lower representation among NFRF applicants (35% in Exploration and 39% in Transformation) relative to core applicants (46%), close to half of Exploration applicant teams (45%) and almost all of Transformation teams (90%) include at least one SocSci researcher. Where present on an NFRF team, SocSci researchers are a majority in about half of Exploration teams (54%) but a minority on most Transformation teams (89%).

Humanities researchers make up a small proportion of NFRF applicants (9% in Exploration and Transformation). However, they also form a minority in the core (15% of core applicants). At the team level, researchers with humanities expertise are found on a similar proportion of Exploration applicant teams (17%) and core applicant teams (15%), but a much higher proportion of teams applying to Transformation (55%). Humanities researchers most often represent a minority on NFRF applicant teams generally, and Transformation applicant teams specifically.

While differences in success rates were found and are mostly in the same direction as application rates, they are very small. For example, Exploration team-level success rates by domain range from 17% for SocSci to 21% for each of NS, engineering and humanities, a difference of only three percentage points.

Representation of disciplines in research projects

The representation of disciplines in the topics (i.e., subject matter) of projects proposed to NFRF were compared to projects proposed for core programs. Overall, differences in representation in topics show a similar pattern to researcher and team participation, but are less pronounced at project level, as shown in figures 8 and 9 below.

Figure 8 long description

Figure 8 shows a bar graph of the percentage difference in representation of domains in proposal topics for Exploration, relative to core applicants. For engineering, it is a positive 12% difference. For natural sciences, it is a positive 4% difference. For agriculture, it is a positive 1% difference. For medical, health and life sciences, it is a 0% difference. For humanities, it is a negative 4% difference. For social sciences, it is a negative 7% difference.

Figure 9 long description

Figure 9 shows a bar graph of the percentage difference in representation of domains in proposal topics for Transformation, relative to core applicants. For engineering, it is a positive 8% difference. For medical, health and life sciences, it is a positive 6% difference. For agriculture, it is a positive 1% difference. For natural sciences, it is a 0% difference. For social sciences, it is a negative 1% difference. For humanities, it is a negative 5% difference.

Representation of NS-related projects proposed to Exploration (40%) and Transformation (36%) is very similar to their representation among projects proposed to core (36%).^{Footnote 55} NS-related projects had a slightly higher success rate in Exploration (they are more prevalent among funded projects, at 45%), but a slightly lower success rate in Transformation (23% of funded projects are related to NS). Notably, the representation of NS expertise among teams is much higher than observed in project topics, suggesting that many NS researchers are working on non-NS-related projects.

In contrast, engineering has a somewhat higher representation in projects proposed to each Exploration (26%) and Transformation (23%), relative to core (15%). Engineering-related topics also had a slightly higher success rate in Transformation, as they are represented in 31% of funded projects.^{Footnote 56}

Representation of MHL-related projects proposed to Exploration (39%) was identical to the core (39%). MHL-related projects have somewhat higher representation among projects proposed to Transformation (45%), a difference of plus six percentage points. No notable difference in success rate was found.^{Footnote 57}

Somewhat lower representation of SocSci-related projects is observed among Exploration proposals (20%) relative to their representation among core proposals (26%), a difference of minus six percentage points,^{Footnote 58} while representation of SocSci-related projects among Transformation proposals was identical (26%) to core.^{Footnote 59} Humanities-related projects are slightly less represented among Exploration proposals (5%) and Transformation proposals (4%) relative to core proposals (9%).

Summary: Representation of disciplines

Overall, researchers from NSE are more prevalent in teams applying to NFRF and teams funded by NFRF, and NS researchers are more likely to form a majority within a research team when present. In contrast, SocSci and humanities researchers have lower representation among NFRF applicants and grantees relative to their representation among core applicants. SocSci, humanities and MHL researchers were more likely to be a minority on the research teams they participate in.

It is important to note that teams (and projects) that are constrained to a single domain can still be interdisciplinary. Exploration has merit criteria intended to ensure that NFRF-funded projects go beyond what can be funded under by individual agencies in terms of interdisciplinarity.^{Footnote 60} However, Exploration does not require cross-domain collaboration.^{Footnote 61} Moreover, some disciplines are considered “boundary disciplines” (e.g., biological sciences; psychology and cognitive sciences). The work of researchers in these disciplines can relate to the mandates of more than one of the agencies, and this subset of researchers also often participates in core funding programs of more than one agency.

Interview participants suggested multiple reasons why researchers from some disciplines might be more likely to apply to NFRF. Overall, the data from the evaluation suggest that there may be multiple drivers for this trend. Some participants from SocSci and humanities described an impression that NFRF was not really intended for researchers in their disciplines, in part due to terminology and language used in program communications and in the application (e.g., descriptions of some merit review criteria). Some participants reported that researchers from some NSE and MHL disciplines are more likely to see the HRHR messaging from NFRF as relevant for their line of work.^{Footnote 62} Some participants pointed to a predominance of NS and health projects in previous NFRF competitions. Past competition results were interpreted as the type of projects that NFRF is looking for, potentially discouraging researchers from other disciplines to consider applying.

Some participants raised capacity as a potential barrier for humanities researchers, who may have less time to develop proposals and participate in large projects due to a heavier teaching load and fewer opportunities for teaching relief, relative to engineering or NS researchers. They also referenced differences in administrative support available for development of grant proposals.^{Footnote 63} Suggestions raised by participants to offset perceived barriers in the research system included offering support for teaching relief; more time between stages of the application process, because researchers with lower levels of support to prepare their proposals and/or higher teaching obligations may struggle more to prepare in a short period; and working with institutional administration to offset perceptions that may exist that NFRF is seeking research from some fields more than others, and to reinforce in program communications how diverse types of research can fit with NFRF’s funding objectives.

Extent to which new relationships are fostered, strengthened and sustained as a result of involvement in New Frontiers in Research Fund-funded projects

Evidence of new relationships on research teams is an important indicator of NFRF’s progress to advance both IDR and HRHR research, beyond what already takes place through conventional programs (the status quo). Both quantitative and qualitative data were examined to assess the extent of novel relationships in NFRF teams: administrative data on prior joint application/funding history^{Footnote 64} in the core, and descriptions of collaborations by key informants in interviews.

Exploration. The majority of applicants to NFRF had no prior history of collaboration with their NFRF teammates in terms of co-applications in the core.^{Footnote 65} Moreover, the majority of teams that applied for Exploration (58-77% depending on competition year) are made up entirely of researchers with no previous core co-applications.^{Footnote 66} The remaining Exploration teams (23-42%) were mixed teams with at least one pair of researchers who had a prior joint application in core, along with at least one pair of researchers who had no prior joint applications in core. Only a small proportion (2-10%) of teams that applied for Exploration consisted entirely of researchers with a previous joint core application.

When isolated to funded core collaborations (i.e., if unsuccessful joint core applications are excluded), an even larger proportion of funded Exploration teams (74-88%) involve researchers that had not previously been co-funded by core programs.^{Footnote 67} This suggests that some NFRF-funded researchers had attempted to collaborate in proposals to core programs, but had been unsuccessful in the core.

These results are in line with Exploration end of grant reports, in which 89% of grantees reported that, through NFRF, they had created new collaborations between disciplines with different perspectives, to a moderate or great extent.^{Footnote 68}

Transformation. Unlike Exploration, only 9-16% of teams that applied for Transformation were made up entirely of members with no prior co-applications in the core.^{Footnote 69} The vast majority of Transformation unsuccessful applicant teams (84%) and funded teams (92%) include a mix of existing and new relationships among team members, at least in terms of core co-applications.^{Footnote 70} This is consistent with descriptions by grantees in key informant interviews, that they had pre-existing relationships with some team members, but the team was expanded for their NFRF project.^{Footnote 71} The larger proportion of prior relationships within Transformation teams (relative to Exploration) may be related to the different nature (greater size and complexity) of Transformation projects as explained by key informants.

While the above results focus on prior co-applications in the core, grantees and applicants participating in interviews described a variety of prior working relationships with members of their NFRF team, including on projects funded by other funders.^{Footnote 72} About two-thirds (15/24, 62%) reported having worked with at least some of their NFRF team members before applying to NFRF.^{Footnote 73} This suggests that the administrative data underestimate the proportion of prior relationships generally, as the administrative data are limited to core funding. Nonetheless, the prevalence of co-applications new to the agencies in both Exploration and Transformation indicates that relationships may be limited in the core and that NFRF incentivizes new collaborations at the federal funding level. For example, as explained by a grantee: “[NFRF] makes us think outside the box and it makes us talk to people who are definitely not in our school or department and be like, oh, how can we bring those two together?” [NFRF grantee]. Among the two-thirds of grantees and applicants who report having worked previously with their NFRF teammates, nearly all reported that NFRF prompted them to expand an existing collaboration to include new members for their NFRF team.^{Footnote 74}

Several interview participants (n=6) raised the importance of NFRF’s focus on HRHR and on novel research relationships to motivate them to reach out to new people. Some described how NFRF prompted bigger questions or afforded them the freedom to expand their research collaborations in ways that they would have not pursued otherwise. Several participants (n=8) also noted that having at least some existing relationships within the research team is important for reducing risk and facilitating progress. A substantial number of interview participants (n=18/41) emphasized the effort, skills, time and experience needed to develop and sustain a strong and integrated interdisciplinary team. Sufficient funds, flexibility and grand duration were described as important to help IDR teams overcome barriers and increase the likelihood of project success. Some examples included funds and/or flexibility for convening and coordinating diverse teams, because, as explained by one grantee: “It’s totally different from what I’m used to in terms of language, research language, processes, and we’re from different universities also.…” [NFRF grantee]. In addition, researchers have typically had less experience writing for a multidisciplinary review panel or in how to prepare a funding proposal for reviewers outside their discipline, requiring substantial upfront commitment and effort. Participants also described other challenges, such as additional logistical steps (i.e., multiple research ethics boards or agreements between different institutions). Ensuring enough funds can be allocated to each team member was highlighted both in interviews and in the Exploration focus group. This is so that all team members have adequate support for the effort, time and engagement an IDR collaboration requires.

Sustainability of relationships

In interviews for this evaluation, grantees were asked whether relationships developed through NFRF were being sustained in new projects. Of those asked,^{Footnote 75} all confirmed they were planning to work together again, or were already working on other projects together.

NFRF funding does not seem to have an effect on the likelihood of subsequent joint core applications specifically. While between 20% and 42% of Exploration teams^{Footnote 76} had at least one pair of grantees jointly apply for core funding,^{Footnote 77} this is roughly equal to the proportion of teams that had applied unsuccessfully to Exploration, and who went on to submit at least one subsequent joint core application (22-42%).^{Footnote 78}

A considerable proportion of unsuccessful Exploration applicants jointly reapply for NFRF. About one-third (27-30%) of unsuccessful Exploration teams involve at least one pair of researchers that applied again to Exploration or Transformation. A similar reapplication rate is observed for Exploration grantees who had completed their grants. Of Exploration teams funded in the 2018 and 2019 competitions, 25% and 19%, respectively, include at least one pair of researchers that jointly applied for another Exploration or Transformation grant.^{Footnote 79} Likewise, 22% of applicants for Transformation 2020 and 30% of applicants for Transformation 2022 had previously applied for Exploration.

Relatedly, in Exploration end of grant reports (n=295), many grantees reported that unexpected results of their project had influenced their future research plans, expanding the scope of their research and/or opening up new possibilities for securing additional funding. Grantees reported a variety of sources of funding through which they intended to continue their research after their Exploration-funded project. Most common sources selected by grantees were FOs from one of the tri-agencies (76%), another Exploration grant (62%), and/or grants from another national or provincial funding organization^{Footnote 80} (58%). About half of grantees (49%) indicated they would seek a Transformation grant.

Nature and depth of integration of disciplines in proposed projects and in implementation of funded projects

As noted above, the integration of perspectives from different disciplines is a defining characteristic of IDR. Exploration and Transformation both include elements related to integration of perspectives in their IDR pass/fail merit review.^{Footnote 81} Both qualitative and quantitative methods provided insight into the level of integration of different disciplinary perspectives in NFRF proposals and projects.

Integration in project design. In interviews, NFRF applicants and grantees were asked to describe how their proposal was developed. Among those who provided background on who participated in the writing of their NFRF proposal (n=15),^{Footnote 82} about half (n=8) described at least some involvement by other team members. Involvement ranged in level of depth from one member drafting the proposal and then receiving feedback from other team members, to all team members co-writing the proposal. For example: “We started holding meetings together to talk about what the project would look like and how it could reach across disciplines.… Actually the process itself of the application and exploring something together was probably where the learning began already” [NFRF grantee]. The other half (n=7) reported writing the proposal on their own. For example: “I wrote the entire proposal. I conceived the idea. I did everything from A to Z” [NFRF grantee]. At least one grantee noted that teams applying from their institution sometimes employ consultants to support the proposal writing or added team members to the projects later on, i.e., waiting until they “saw that they were gaining traction” before involving others [NFRF grantee]. Another explained why it can be unrealistic to expect extensive engagement of other team members at proposal stage: “In some ways it’s a hard sell to get people to take the time to participate in the co-writing and co-development and so on because everyone knows it’s such a low yield” [NFRF grantee].

In a similar vein, 10 review panel members^{Footnote 83} described variation in the level of integration of disciplines reflected in NFRF proposals they had reviewed. This variation ranged from some proposals appearing to have disciplines included to meet eligibility requirements or the integration of disciplines seeming “forced,” to other proposals that demonstrated an integration of disciplinary perspectives at a fundamental level in the research. A few reviewers highlighted the importance of involving multiple disciplines at the conceptual stage of the project. In the words of one reviewer: “If the people from the different disciplines have actually been brought together … somebody’s … gone to a great deal of effort to find those voices and integrate them very early in the process not just as the add-on to make them eligible for the program. I think it’s what stage they are brought into the research idea [that is important]” [NFRF reviewer].

At least five key informants brought up balance among disciplines within a NFRF team as important, although this was not a question asked in interviews (see also the sections above on representation of disciplines in research teams). Participants had different perspectives about potential implications for NFRF’s objectives for IDR. Some reviewers cautioned that teams made up of a primary discipline, with other disciplines representing a minority on the team, can signal relatively peripheral involvement of the latter, i.e., that they may be “added-on” as a “helper.”^{Footnote 84} Multiple participants gave examples of situations where disciplines serve a “helper role” (e.g., researchers bringing expertise in machine learning, statistics or ethics to a project that is focused in health). For example: “quand on lit les demandes, on se rend compte que oui, des fois, c’est interdisciplinaire.… Mais il y a d’autres demandes où c’est une fausse interdisciplinarité dans le sens où l’autre discipline devient juste un outil, peut-être de deuxième niveau” [NFRF reviewer]. Another reviewer stated that team compositions with primary disciplines and helper disciplines increase the likelihood of success and can be effective: “… honestly, the most successful [proposals in competitions] are the ones where you’ve got a science and then a science adjacent discipline that is supporting it. By science adjacent I’m referring to social work, business or law. For example: a legal expert who understands the intellectual property development of this, etc.” [NFRF reviewer].

Review panel participants, when describing the assessment of disciplinary integration in proposals, identified a few signals of lower integration. For example, proposals in which other disciplines are effectively in service to a unidisciplinary project; disciplinary representation on the team is oriented to a single discipline; or the proposal’s description of risk does not reflect the challenges to integrate different (and/or more distant) disciplinary perspectives in research. Two general types of IDR were suggested: an “additive” form, in which other disciplines are involved as contributors to help a project go further, although the project is still centred in one disciplinary area; and a “generative” form, with greater synergy and mutual influence among the disciplines. The former may carry less risk related to challenges of working across disciplines, while the latter may carry more potential for innovation arising from integration of perspectives. Overall, reviewers described a range of possible types of proposals that could merit funding under NFRF’s criteria, and some differences in perspective in what meets NFRF’s objectives. Suggestions for supporting the IDR merit review in terms of integration are outlined below (see the “New Frontiers in Research Fund’s selection processes for interdisciplinary research” section).

Diversity across disciplines and domains. In terms of the research topics of funded projects, generally, Exploration-funded projects show higher levels of cross-domain connections relative to core-funded projects. As shown in Figure 10, about one-third of Exploration-funded projects reflect more than one domain (35%), relative to one-fifth of core (22%). Transformation-funded projects show higher levels of cross-domain connections, as 46% of Transformation-funded projects reflect multiple domains.^{Footnote 85}

Figure 10 long description

Figure 10 shows a bar graph of the distribution of core, Exploration, and Transformation projects, based on number of domains they relate to. For Transformation-funded projects, 54% had one domain and 46% had two domains. For Transformation-unfunded projects, 63% had one domain, 33% had two domains, and 2% had three domains. For Exploration-funded projects, 62% had one domain, 33% had two domains, and 2% had three domains. For Exploration-unfunded projects, 65% had one domain, 30% had two domains, and 2% had three domains. For core-funded projects, 78% had one domain, 21% had two domains, and 1% had three domains. For core-unfunded projects, 70% had one domain, 27% had two domains, and 1% had three domains.

Exploration-funded projects depict higher connectedness across disciplines relative to core-funded projects. This is shown by the higher number of edges (lines) connecting disciplines (nodes), as shown in Figure 11. This is also evident in density^{Footnote 86} value, which is five percentage points higher in Exploration (20%) relative to core (15%). This greater density in Exploration represents about 40 new pairs of disciplines connected via NFRF-funded projects.

In both core-funded and Exploration-funded projects, the most common pair of domains to co-occur are NS and MHL. This is due to a high prevalence of connections between biological sciences (NS) and basic medicine and life sciences (MHL) in both core and Exploration.

These results indicate that NFRF is attracting and selecting projects that connect disciplines. Nonetheless, it is worth noting that levels of cross-disciplinary connections in the subject matter of research projects funded by NFRF are lower relative to levels of cross-disciplinary connections observed in NFRF participants and research teams. In other words, disciplinary diversity is better reflected in the composition of NFRF teams than in the subject matter of proposals funded by NFRF.

Figure 11 illustrates connections across disciplines and research domains identified in the topic description of funded projects, comparing projects funded by the agencies’ core programs to Exploration and Transformation projects. The nodes (circles) in each diagram represent disciplines and are marked with CRDC group-level codes. Nodes are colour-coded by research domain (see legend). The size of the node is proportional to the number of observations in the network (i.e., projects in which that discipline is identified). The thickness of the lines connecting the nodes is proportional to the frequency of connections between any two disciplines, reflecting the strength of ties between any two disciplines.

Figure 11 long description

Figure 11 shows three network visuals of the representation of disciplines and connections across disciplines, observed in subject matter of projects funded by core, Exploration and Transformation.

In each panel, nodes represent disciplinary groups and are colour-coded by disciplinary domain: natural sciences (red); engineering (pink); medical, health and life sciences (green); agriculture (purple); social sciences (light blue); and humanities (dark blue). Node size reflects the relative prominence of each disciplinary group across projects. Grey lines connecting nodes indicate connections between disciplines (i.e., where two or more disciplinary groups are represented within the same project).

The core panel displays a network of interconnected disciplinary groups. The Exploration panel shows a network with a greater number of connections among disciplinary groups. The Transformation panel shows a compact network composed of a small number of disciplinary groups with connections among them.

A legend accompanies the figure and identifies the six disciplinary categories and their associated colours, which are used consistently across all three panels.

Integration in project implementation. In addition to integration in design of research projects, there is also some evidence of cross-disciplinary integration as part of project implementation. In Exploration end of grant reports,^{Footnote 88} grantees (n=295) were asked about the extent to which the outcomes of their projects involved integrating elements (ideas, knowledge or methods) from at least two disciplines that do not traditionally collaborate and/or disciplines where collaborations already exist. As shown in Figure 12, most Exploration grantees reported moderate or above on both.^{Footnote 89}

Figure 12 long description

Figure 12 shows a bar graph of grantees who reported the extent to which outcomes of Exploration projects involved integrating elements from disciplines where collaborations already exist, and disciplines that do not traditionally collaborate. For existing cross-disciplinary collaborations, 5% reported “not at all / likely in the future,” 7% reported “slight to some extent” and 88% reported “moderate to great extent.” For new cross-disciplinary collaborations, 10% reported “not at all / likely in the future,” 15% reported “slight to some extent” and 75% reported “moderate to great extent.”

When asked to provide an example, just over half of grantees described integration between disciplines that do not traditionally collaborate (54%), and just under half described integration between disciplines where collaborations already exist (46%). The descriptions provided by grantees who selected the latter option suggest that, while these disciplines may already be commonly collaborating, NFRF had allowed for increased scope, extent or scale beyond what would normally be seen. This is in line with Exploration’s merit review criteria. As noted in the next section, projects that involve integration between disciplines with an existing history of collaboration are considered eligible for Exploration (if the project proposes the application or adaptation of frameworks/tools/methods/techniques from one discipline to solve a problem in another discipline).

Time, effort and skill are needed to develop an integrated team with members from disciplines that do not typically collaborate (see above, under new and sustained relationships). As such, ensuring sufficient grant size and duration of funding is important. As explained by a focus group participant, there needs to be adequate funds and distribution of funds within the team, so that: “all the disciplines are supported through the grant … [to ensure] the interdisciplinarity in the proposal, but also in the actual implementation of the project” [focus group participant, Exploration].

New Frontiers in Research Fund’s selection processes for interdisciplinary research

This section summarizes findings related to NFRF’s merit assessment for IDR.^{Footnote 91} It focuses on process quality in keeping with the formative nature of the evaluation. It was addressed primarily through key informant interviews, document review and analysis of a sample of reviewer scores,^{Footnote 92} with support from the academic literature. A more detailed overview of NFRF’s review process is found in Appendix B.

As important context, the academic literature documents multiple challenges faced by funders to assess the merit of IDR projects. To date, there is no universally accepted standard for IDR assessment. Characteristics intrinsic to IDR itself limit the use of conventional approaches that have been developed for disciplinary-specific assessments. Programs like NFRF have needed to break new ground and need to learn through experience to improve IDR merit assessment practices over time (see, e.g., Molas-Gallart et al., 2012; de Oliveira et al., 2019). NFRF has shown a commitment to reflection and continuous improvement, for example as demonstrated in modifications to NFRF’s processes over time, particularly in Exploration, and reflected in Exploration’s co-chairs’ reports (SSHRC, 2019, 2020b, 2021, 2022).

NFRF reviewers, grantees and unsuccessful applicants were asked in interviews about their experience with NFRF’s review process. Participants were asked about what worked well and what could be improved. While nine participants indicated the review matrices were useful and simple for assessing interdisciplinarity and several participants (7) described NFRF’s guidance and merit criteria for interdisciplinarity as being well developed and helpful, a substantial number (14)^{Footnote 93} described applying them in practice as challenging. For example, in the words of one reviewer, the interdisciplinary component was “one of the trickiest things to evaluate” [NFRF reviewer].

Applicants and reviewers alike described challenges to interpreting the Interdisciplinarity criterion. Reviewers described variation in interpretation by applicants, as indicated in proposals they had reviewed, and also variation in interpretation among their fellow reviewers. Determining the pass/fail threshold for this criterion was identified as particularly difficult. As summed up by one reviewer: “[There is] too much misinterpretation of what qualifies as adequate interdisciplinarity for this program” [NFRF reviewer]. Variation in interpretation of IDR was also raised as a concern by focus group participants who had participated in review panels for Exploration.

Findings from analysis of individual reviewer scores from three Exploration competitions^{Footnote 94} are consistent with the interview and focus group results. Variation is evident in the interdisciplinarity pass/fail scores by individual reviewers (i.e., scores prior to the multidisciplinary review committee deliberation). Over half (57%) of proposals received a mix of “pass” and “fail” scores. There are some indications of an increase in consistency among reviewers over time. The proportion of proposals in which scores align (i.e., all reviewers score as pass or all score as fail) increased from 39% in 2020, to 45% in 2021 and 48% in 2022. With only three years, any conclusion about a trend needs to be tentative. While a nascent upward trend may be indicative of increasing consistency among reviewers over time, score alignment below 50% reinforces key informant perspectives of the value of further efforts to support reviewer interpretation of the criterion and scoring via the matrix. Data from the evaluation and observations in the academic literature suggest a few key considerations, as well as some options to optimize the merit assessment of interdisciplinarity.

Considerations:

• Expectations need to align with the nature of the program. Diversity in perspective is expected and desired from a multidisciplinary panel. In the academic literature, authors note that IDR does not fit within a single discipline or “normal” science, and for this reason it is common for peer reviewers with differing academic backgrounds to misunderstand one another (Leahey et al., 2015) and disagree on the merits of a proposed project (Lamont, 2009; Mansilla, 2006). Alignment should not be expected to be comparable to conventional programs.
• Change takes time. Interview participants pointed out that NFRF is a new program, and the research community has been unsure about NFRF’s expectations. They described a need to “unlearn” conventional practice for proposal development and peer review.^{Footnote 95} NFRF may need to continue to put extra effort into communication about its expectations.
• Additional effort and resources are needed for IDR review. In addition to the above, high diversity among individual IDR projects is noted as a feature of IDR, in the literature and by key informants in interviews. Reviewer participants noted that reviewers are typically not accustomed to the diversity of proposals received by NFRF. Moreover, research proposals centred in a discipline with its own logic and approaches are difficult to compare with others centred in other, very different disciplines. More effort and resources are required to evaluate very different projects equally and fairly (de Oliveira et al., 2019; Moreno et al., 2016).

Strengths to build on:

• Quality of program support. Although not specific to IDR, multiple interview participants highlighted the high quality of the assistance they received from NFRF program staff to support understanding of the program’s expectations; for example, in response to questions when preparing applications, during multidisciplinary review panel discussions, and when implementing their funded projects.
• Richness of the multidisciplinary review panel discussions. The review panel discussions were identified as a key strength in both interviews and focus groups, as they allow for questions, deliberations and consensus as a group. Multiple reviewers took time in interviews to express their appreciation for the quality of these discussions, and for NFRF’s support to them.^{Footnote 96} For example, in the words of one reviewer: “Il y a une réunion et elle permet d’arriver à la décision ... le processus en tant que tel me semble le plus honnête possible. Je n’ai rien à dire là-dessus. Je félicite toute l’équipe” [NFRF reviewer]. While panel discussions support review from an interdisciplinary perspective, reviewer participants in the interviews urged allowing sufficient time, a reasonable number of applications to review and, where possible, face-to-face panel meetings. Focus group participants underscored the value of balanced disciplinary expertise on panels to address interdisciplinary projects.
• Reviewer selection. Ensuring members have sufficient expertise in IDR was recommended by focus group participants. This was also raised in key informant interviews. Participants pointed to the importance of ensuring reviewer understanding of IDR, either as a priority in selecting reviewers with IDR experience and/or retention of reviewers for future competitions. For example, one participant recommended: “… hang on to your good reviewers and your good co-chairs … they are the ones that help these meetings go forward in a good way” [NFRF reviewer]. Time and attention to the composition and functioning of an IDR review panel is also stressed in the literature (see above).
• A level of flexibility. As noted above, the assessment of IDR requires a balance between flexibility, given the diversity of projects,^{Footnote 97} and a need for a systematic, efficient and timely peer review. NFRF’s transparency is evident in the merit review guidelines and matrices, which are available online for all competitions years. The review process is structured and supported by detailed matrices and extended guidance for applicants and reviewers. Reviewers are also given ample discretion in which elements of the Interdisciplinarity criterion they deem relevant to include in their assessment,^{Footnote 98} which allows flexibility. At least two participants cautioned against criteria becoming too rigid, as it could reduce the diversity of projects and limit innovation. However, too much flexibility, combined with variation in understanding about IDR and about NFRF’s expectations for IDR, can create challenges in merit review. Suggestions for improvement follow.
  
• Increase consistency in the scoring process.
  • Revisit the level of discretion provided to reviewers. As noted above, reviewers are offered substantial flexibility in s merit assessment of the Interdisciplinarity criterion. They have discretion with respect to which elements in the matrix they consider in their assessment. A proposal does not have to receive a “pass” on all the elements to receive an overall “pass” score for that criterion. Other elements not included in the matrix can be considered by a reviewer as part of their assessment. While flexibility is important (see above), this level of discretion likely contributes to variation in scoring, as different reviewers may be scoring based on different choices of elements.
  • Provide reviewers with (more) concrete indicators. Several review panel participants focused on the challenge to assess key elements in the merit criteria, i.e., the diversity and novelty of perspectives on a team, and their integration in a project, especially for “borderline” proposals. Options for indicators included the following: disciplinary distance among the team members in lead roles (i.e., NPI and Co-PI);^{Footnote 99} balance in disciplinary representation on the team; looking for evidence that any tools and techniques brought together from different disciplines are being combined in the proposed research, as an indicator of integration; and looking for identification by applicants of risks related to negotiating and integrating differences in perspective among research team members.^{Footnote 100}
  • Provide applicants with case examples of clear “passes.” Providing case examples of clear “passes” (e.g., selecting examples that are at the top of a range as examples), as well as examples of projects that would not meet NFRF’s threshold for IDR, was suggested by reviewers to help applicants better understand the criteria. It was also suggested by applicants for reviewers, to guide them in proposal assessment. A similar suggestion was made in the 2021 Exploration Co-chairs’ Report (SSHRC, 2022).
• Optimize the existing wording of the interdisciplinarity-related criteria.
  • Elements under the Interdisciplinarity criterion can appear inconsistent with each other. In Exploration, for example, one of the elements (“novelty of perspective”) requires integration of two or more disciplines that are not commonly combined for a “pass” score, while another element (“novelty of approach”) states that a “pass” is possible for projects in which the disciplines have a history of collaboration, if the project proposes the application or adaptation of frameworks/tools/methods/techniques from one discipline to solve a problem in another discipline.
  • A pass/fail “Fit to Program” subcriterion was added to Exploration in 2019, aimed to reduce the number of applications in which the proposed collaboration is not novel, i.e., to reduce the volume of applications in which the disciplines involved already “have an established tradition of collaborating” (2018 Co-chairs’ Report [SSHRC, 2019]). The subcriterion states that the proposed project must push the boundaries of what can be funded through the agencies’ existing mandates and suites of programs. The original wording specified fit to program in terms of a proposal’s interdisciplinarity and high-risk nature. However, the wording was changed in 2021 to refer to interdisciplinary and/or high-risk. Although the subcriterion is still labeled “Interdisciplinarity subcriterion,” the wording effectively allows a pass on high-risk features alone. This means that the criterion no longer requires fit to program in terms of IDR, which was the intent behind its addition.
  • Exploration’s high-risk merit criterion incorporates IDR as a potential source of risk to NFRF projects. The element in the matrix for the High-Risk criterion is called “Novel interdisciplinary approaches.” It requires both novelty and integration (novel disciplinary combinations and the integration of disciplines) for a Very Good or Exceptional rating.^{Footnote 101} However, a Fair rating requires neither. As all projects would already have been assessed under the pass/fail Interdisciplinarity criterion before being assessed for high risk, the purpose of the Fair description is unclear.
• Adopt a graduated scale for the Interdisciplinarity criterion. A few reviewers suggested adopting a graduated scale for the Interdisciplinarity criterion. This was proposed as a way to facilitate the review among proposals that vary widely in how they apply IDR, and in their level of interdisciplinarity, with a substantial number considered to be “borderline” or “weak passes.” The 2020 Exploration Co-chairs’ Report (SSHRC, 2021) also suggested adoption of a nonbinary matrix to allow for more nuanced assessment of this criterion.
• Provide applicants with feedback from reviewers. Six applicants urged that feedback be provided to improve their understanding of the Interdisciplinarity criterion. Individualized feedback is resource intensive. NFRF published reports from the co-chairs of Exploration review panels from 2018 to 2021 that provided a synthesis of guidance for all applicants.

EQ2. To what extent is NFRF making progress toward advancing high-risk/novel and anticipated high-reward research?

NFRF Exploration’s selection processes for high-risk / high-reward

This section is specific to Exploration because, while both streams prioritize high-reward research, Exploration focuses more broadly on risk relative to Transformation. The results below are based on data from key informant interviews, grantee end of grant reports, and document reviews, supported by review of academic literature on HRHR research.^{Footnote 102}

NFRF provides descriptions of its expectations for HRHR research in its merit criteria, guidance for reviewers, and other program documentation. An overall description of Exploration’s processes for selection of proposals is provided in Appendix B.

High-Risk. The merit review matrix for Exploration’s High-Risk criterion includes six elements.^{Footnote 103} These elements focus on types of innovation in a project’s approach that can elevate risk for the project; for example: adapting methods, using techniques proven or assumed to be extraordinarily difficult, challenging existing research paradigms, or applying a novel interdisciplinary approach. Examples also include the complexity of the targeted problem that the research aims to address. High-risk is also part of a “Fit to Program” subcriterion introduced in 2019. NFRF’s documentation differentiates high-risk research from incremental research (i.e., research that incrementally advances understanding). High-risk research is described as “outside of the box,” defying current paradigms, and combining ideas from different disciplines in terms of “bold, innovative perspectives.”

As noted in the academic literature on HRHR, different funding bodies have different definitions and approaches for defining HRHR research (Rusco, 2011). Sources of risk documented in the literature, such as the novelty of the research, its position outside of established disciplinary paradigms, or its ambition in tackling challenging and/or complex questions (OECD, 2021) are reflected in NFRF’s criteria.

Also noted in the literature is a caution that funding programs avoid overly prescriptive or restrictive criteria, as they may inadvertently introduce challenges for HRHR. This is similar to advice on assessment for IDR (see above). NFRF’s criteria are clearly not overly prescriptive or narrow, and ample flexibility is offered to the reviewer. As noted above, NFRF’s instructions for the use of its review matrices clearly state that they are intended as a guide. The reviewer is invited to consider all elements or a subset, and can consider relevant elements not listed in the matrix at their discretion. Reviewers are offered discretion in how they arrive at an overall rating. For example, a proposal does not need to be rated Exceptional for all the elements for a reviewer to be able to assign an Exceptional rating for the overall criterion. The matrix provides descriptions of four rating levels for each element: Exceptional, Very Good, Fair and Poor. Reviewers also have the option to select ratings between the four described levels, which provides them with a seven-point scale for their assessment.^{Footnote 104}

In key informant interviews, applicants were asked to describe the nature of risk in their projects, and reviewers were asked about risk in projects they had reviewed as members of the panel. Across subgroups, participants identified a variety of sources of risk, although most responses fell into three broad categories consistent with the academic literature: risks inherent in the novelty of the research approaches, in the level of interdisciplinarity, or in the difficulty of the questions they were trying to address through their research. Yet some participants expressed uncertainty about NFRF’s expectations for risk, and just over half of the interview participants (15/28)^{Footnote 105} identified challenges with assessment of high risk. Overall, participants stressed that interpretation of risk can vary depending on context and perspective. A few stressed the need to exercise judgement when assessing risk, because criteria are not universally applicable, i.e., risk depends on context and varies among FoR. In other words, some of the challenge in assessment is intrinsic to the concept of risk.

Participants identified at least four other challenges for interpreting NFRF’s expectations for risk. These are very similar to the challenges reported above for IDR:

1. People from different backgrounds, training and perspectives tend to prioritize different elements or interpret the nature of risk differently. This means that an IDR team can struggle to find common ground, and that reviewers can arrive at different initial assessments.
2. NFRF is a new program. Researchers and reviewers are accustomed to lower tolerance for risk in core funding programs; it will take time to change long established practices.^{Footnote 106}
3. NFRF’s high-risk criteria were described by some participants as unclear.
4. Reconciling high-risk with NFRF’s feasibility criterion was identified as a particular challenge.

For example, challenges related to diversity of perspectives were described in the words of one reviewer: “People’s evaluations and perception of risk are different. We could hear the same information, and someone evaluates it as high, medium or low” [NFRF reviewer]. An applicant explained: “This has been challenging … this can really vary from person to person. For example, if I’m reading a proposal and you are reading the same proposal, our perceptions can be completely different” [NFRF applicant]. As with the assessment of interdisciplinarity (see above), multiple reviewers pointed to the value of NFRF’s multidisciplinary panel discussions to reach a consensus in the assessment of risk. The multidisciplinary panel is revisited at the end of this section.

Among the 15 participants who identified challenges, 11 described NFRF’s criteria for High-Risk as unclear and/or recommended improvements be made to enhance clarity. This group included reviewers, unfunded applicants and grantees. The four remaining participants (all reviewers) considered the criteria clear but often misunderstood by applicants, containing internal tensions that are difficult to balance;^{Footnote 107} and/or that the scoring process can be confusing.

Among the 11 who described the criteria as unclear and/or recommended improvement, there was a desire for more specificity and/or tighter definitions. A reviewer explained: “There’s a lot of fuzziness in the review, in the writing of the grant. A lot of fuzziness in what they mean. They should tighten up” [NFRF reviewer]. A grantee described their experience: “That was a huge problem that I had when trying to develop my proposal. I mean I have the [grant] … but being a co-applicant on other people’s NFRFs (which were truly very, very risky) and all of the reviewers said it was too risky. Without any true guidelines, how do we know how risky to make it?” [NFRF grantee].

A perceived contradiction between expectations for high-risk and feasibility in NFRF’s criteria was raised by several participants (n=7), including applicants and reviewers, as well as participants in the focus groups. Participants also raised possible implications for the proposals NFRF receives. In the words of a reviewer participant in interviews, it could make some applicants propose less risky projects: “I think there is a degree of confusion between higher risk and feasibility because those two things seem to be really at odds in a sense…. So they (applicants) tend to go light on the high-risk side because they want to demonstrate that it is feasible. But I don’t think that’s really the intent when the program is designed” [NFRF reviewer]. Of note, Exploration co-chairs’ reports from 2019 to 2021 also reported fewer high-risk or “bold” proposals than expected. The concern was echoed by other key informants, including an applicant who described efforts to de-risk their project to improve feasibility after being unsuccessful: “In the last three years we were able to improve the feasibility because we got [other funding].... We built the infrastructure and all that. But is this becoming more like CIHR?.... That’s what we do in CIHR ... CIHR needs to see pilot data or some progress made towards it ... the question is, are we going in the CIHR direction if we are improving and increasing the feasibility?” [NFRF applicant].

There were clear differences in opinion expressed about how NFRF should address the perceived tension between high-risk and feasibility. A few recommended greater focus on feasibility. For example: “It’s backwards. It should be 50% feasibility, and then high-risk, high-reward” [NFRF reviewer]. Others stressed the importance of accepting possible failure in Exploration. One reviewer pointed out that a too heavy emphasis on feasibility could hold advancements back, because what is feasible in research is changing rapidly: “We know that in all areas of science, what might have been feasible even three years ago is not the same as now. Things are evolving very rapidly. So, in some respects, you’ve got to take a bit of a leap of faith too … look at it in the bigger picture perspective … if we’re grounded a lot in feasibility, that might overlook the potential of some that could bring forth novel tools” [NFRF reviewer].

NFRF is still unusual among funding programs in its emphasis on high-risk and, as such, it may take time to normalize this feature of the program. As described by a reviewer: “Because of years and years of working on [other] grants, people are just very afraid to write a proposal where the research might fail. I can see that. They don’t feel like they’re allowed to say that. The ones who get over that, do talk about the risk of failure, are invariably the best proposals” [NFRF reviewer]. Exploration’s online documentation already links high risk to potential failure, in recognition that innovation carries risk, stating an expectation that some funded projects will fail to meet their objectives.^{Footnote 108} Several interview participants (10) raised the willingness to accept failure as something that sets NFRF apart from other programs, for example: “This agency, I believe, funds very innovative research in the Canadian landscape and I get excited reading these applications. It’s okay if some of them fail. That’s what sets this apart” [NFRF reviewer].

Risk of failure is noted in the academic literature in at least two ways: uncertainty in the likelihood of expected outcomes and also the nature of the outcomes that are realized. In other words, with HRHR there is a greater chance of failure to produce intended results and results may be quite different than expected, yet valuable (Boudreau et al., 2016; Cao & Zhang, 2022; Machado, 2021; OECD, 2021; Staudt et al., 2018). Consistent with this, data from Exploration achievement reports shows that fewer than half of Exploration grantees (43%) reported that their objectives were met to a great extent,^{Footnote 109} while a majority (67%) reported that their projects yielded unexpected positive results.

NFRF’s program documentation is clear that the Feasibility criterion is not meant to contradict the High-Risk criterion. The intent for the Feasibility criterion, as described by NFRF, is to focus assessment on whether the project is set up to be implemented successfully, while allowing the nature of the reward to be uncertain. Explanations in the program documentation have varied in detail, but been consistent on this point over the years. For example, from the most recent competitions: “For the Feasibility criterion, the focus is not on the likelihood of success, but on the quality of the project plan and the team’s ability to execute it” (SSHRC, 2025).^{Footnote 110} At least one reviewer described this as the key point that allows feasibility and high-risk to be compatible in the merit review process. While also describing the assessment of feasibility to be very challenging in Exploration, this reviewer explained that it is not necessarily in contradiction with risk: “… on pourrait penser qu’il y a une contradiction avec le risque. Plus le risque est fort, moins ça devrait être faisable. Donc, on pourrait le comprendre de cette manière. Mais ce n’est pas comme ça que je comprends. Pour moi, la faisabilité, c’est est-ce que le projet est monté de sorte qu’on arrive à en tirer quelque chose. Un résultat positif est un résultat intéressant. Un résultat négatif est un résultat intéressant” [NFRF reviewer]. This reviewer suggested that giving the Feasibility criterion a different name could reduce confusion for applicants and reviewers, because it would reduce confusion with conventional feasibility assessment in other funding programs.

Based on the program documentation, NFRF’s intention is not to fund projects with a high implementation risk, i.e., risk of failure during implementation. NFRF’s aim is to focus on projects that are feasible to conduct, likely to be completed, but carry a risk that the proposed reward might nonetheless not be realized. Additional clarification is found under the description of Exploration’s merit criterion for feasibility: “… risk must be related to the idea being proposed, and not to a lack of a concrete plan or inability to execute the activities” (emphasis added)^{Footnote 111}. For this reason, the Feasibility criterion considers elements such as expertise and capacity of the research team; workplan and timeline; aspects of the proposed approach, including equity, diversity and inclusion (EDI) considerations; and suitability of the research environment.

The elements of this criterion reflect ingredients that can aid a team to respond and adapt to the inevitable surprises associated with high-risk research, such as appropriate planning, resources, scoping of the project given duration of the grant, and team skillset. However, this criterion seems to be less clear. A focus on capacity to adapt (versus ability to predict and mitigate^{Footnote 112}) is important in Exploration because of its emphasis on novelty and innovation. More than half of participants (16/28) described novelty as a primary source of risk in NFRF projects. Participants described risk due to novelty leading to potential for failure during project implementation (before completion), as well as relating to the nature of the reward (the project is completed, but the reward is less than or different than was hoped). Some participants attributed risk to the uncertainty and unknowns inherent in exploring untested ideas or uncharted territories; for example, in the words of one grantee: “… we keep running into things we never could have imagined” [NFRF grantee]. Some spoke about a willingness to fund projects without preliminary data as a valuable feature of NFRF: “This is one of the reasons why I think the New Frontiers was so successful—because it’s allowing this high-risk. As I said, if I were going elsewhere, they would say do it first and show us that its working and then we’ll fund you” [NFRF grantee]. Others, however, pointed to a lack of preliminary data as a reason they were not funded by NFRF, contributing to confusion about what’s acceptable in terms of risk; for example: “[The reviewers said] ‘great, you have some candidates, but you still need to provide us more information that these are actually viable … the reward is there’ … but they’re like ‘it’s still too high-risk.’ So how can you have a competition whose sort of tagline is high-risk, high-reward, but then you penalize people for the high-risk aspect?” [NFRF applicant].

Notably, an ability to complete the project within the proposed timeframe and having the required expertise are both included in Exploration’s criteria for feasibility. In other words, not having all the required expertise or uncertainty about being able to complete the project in the time anticipated are not considered acceptable risks under Exploration. Yet, some participants argue that these risks are unavoidable for the types of projects that NFRF is seeking to fund. For example, a few participants raised entering new areas of research in which expertise may not exist. In Exploration end of grant reports, grantees provided examples of how high novelty in their project led to unanticipated challenges, requiring them to pivot and adapt. Examples of adaptations included accessing different types of expertise than expected, forming additional collaborations, and modifying their methodology, all of which required more time than could have been anticipated. Overall, the uncertainty raises the risk of being unable to complete the project during the grant term. Interview participants offered several suggestions for improving risk assessment in NFRF, which are similar to suggestions for the merit review of IDR:

• Seek ways to clarify the intent of the feasibility criterion, and how it differs from what applicants and reviewers are accustomed to in conventional funding programs.
• Strengthen the definition of high risk or provide examples, to improve clarity with respect to acceptable level of risk, and also with respect to types of risk that are acceptable. Include aspects of a project that elevate risk during implementation.
• Retain strong support for high quality multidisciplinary review panel discussions. High quality panel review is enabled through program support and facilitation; synchronous and, if possible, in-person meetings; and sufficient time and attention to each proposal, which is also a function of the number of proposals each reviewer is assigned and the time available to them for review. Caution was raised about too much reliance on disciplinary-specific expertise, in favour of full engagement of multiple perspectives in assessment of risk during panel discussions, and of encouraging proposals be written explicitly for a multidisciplinary panel (versus highly disciplinary-specific content).

Extent to which the New Frontiers in Research Fund’s elements and process optimally support selection of high-reward research

Participants who described challenges to understanding NFRF’s expectations for HRHR focused primarily on the high-risk criterion, detailed above. Overall, the data suggest that the high-reward criteria are less challenging to navigate. Suggestions for improvement focused on ensuring diverse types of rewards continue to be recognized in the merit review process and guidance to applicants.

The interview data show some consensus that NFRF-funded projects should produce benefits beyond academia. However, some participants (n=6) highlighted that interdisciplinary and high-risk research is also needed for advancement of foundational or discovery-oriented academic knowledge that may only later translate to social and economic benefit. Participants provided a variety of examples, such as discovery of new materials or better approaches to working with Indigenous Peoples, which may not yield concrete socio-economic benefits within the timeframe of the project. This means continuing to ensure that merit review recognizes the variety of types of potential high reward (e.g., development of new technologies, changes to laws, new treatments for disease, and changes in social systems). It also means avoiding an assumption that all rewards will be near-term, because the time horizon will vary depending on the novelty of the research, the complexity of the problem, and the change envisioned (e.g., several years, a decade, or even a “generational shift”). Similar to the suggestions above for IDR and high-risk, several participants (9) recommended that NFRF publish case examples. In the words of an interviewee: “It would help proponents and institutions, but it would also help the general community of universities and scholars [to] know to what degree this high standard of high-risk, high-reward is feasibly achieved during the lifetime of the project, and of course beyond the lifetime of the project” [NFRF applicant].

Perspectives on outcomes of mature or complemented projects

Although the evaluation did not measure outcomes of NFRF, due to NFRF’s stage of development (see Background), the evaluation captured perspectives on early outcomes and anticipated longer-term outcomes. All grantees who were interviewed as part of the evaluation (n=15) were asked to reflect on the outcomes of their NFRF projects to date.^{Footnote 113} Results from interviews were complemented by analyses of available end of grant reports submitted by Exploration grantees.^{Footnote 114}

It is important to consider that most NFRF grants within scope were not completed at the time of the evaluation. Funding for the first^{Footnote 115} cohort of Exploration grants (n=157) started in March of 2019. All Exploration grants were provided with an automatic extension for a third year, and 343 Exploration grants from the 2018 and 2019 competitions were provided with a one-year extension due to the pandemic. A total of 259 Exploration grants had completed their projects at the time of the evaluation. Given the six-year duration of Transformation grants, all were in their early stages at the time of the evaluation.

Delays in research projects due to the pandemic or other reasons (e.g., unanticipated challenges, or complexity of the research project) were raised in both interviews and in Exploration grantee reports, and were considered to be common for this type of research project, based on academic literature (see above).

As most of the projects were still ongoing, grantee participants in interviews (of both streams) mostly described the progress they had made to date. Many highlighted milestones in their projects such as acquiring preliminary data and results, building team capacity or research capacity, and training students and other research personnel on how to conduct IDR. These interviewees indicated that their NFRF projects had provided their students and trainees with unique opportunities to develop skills and experience with this type of research, aligned with NFRF’s intended outcomes for training and talent development in IDR and HRHR research. Descriptions of capacity development in teams are not surprising, considering that the majority of NFRF teams involve researchers working together for the first time (see the section on novel collaborations, above). A few grantee participants who were in relatively earlier stages of their projects focused on the establishment of processes and structures that facilitate collaborations within the team or with external stakeholders or partners (e.g., industry partners or Indigenous communities).

Some grantees noted specific outputs of their NFRF-funded research, such as publications and other forms of knowledge dissemination (e.g., presenting at conferences), based on either the preliminary results of their ongoing projects or final findings of completed research. End of grant reports provided additional information on outcomes of Exploration projects as reported by grantees (n=295). Most grantees (75%) indicated that the objectives and intended outcomes of their research projects were met to a moderate or great extent.^{Footnote 116} About a fifth of grantees (23%) reported their stated goals were somewhat^{Footnote 117} met. Additionally, two-thirds of Exploration grantees (67%) indicated that their research projects yielded unexpected positive results. Grantees described encountering results or discoveries that were not part of their initial research plan or expectations, which often led to new research directions, projects or applications. Some grantees mentioned that their unexpected results frequently prompted collaborations with other researchers from different disciplines or sectors, and led to enhanced research outcomes. Given the high-risk nature of most NFRF research projects, it is expected that a certain number of projects produce significant yet unpredicted advances in knowledge, methods and/or technologies that lead to further research.

Grantees reported on their future plans for their research. About three-quarters of grantees indicated that they will seek funding from one of the tri-agencies (excluding NFRF). This result may indicate that Exploration grants may have helped “de-risk” some research projects such that they are able to seek funding from the core programs to build on their NFRF-funded research. About two-thirds of grantees indicated they will seek another Exploration grant, while about half indicated they will apply for a Transformation grant. These results are in line with analyses of participation histories reported earlier in this report (see the section on sustainability of relationships, above).

More than half of the Exploration grantees reported other funding sources in terms of their future plans. Some examples provided included the Canada Foundation for Innovation, Mitacs, the private sector, and research funding provided by their institutions.

Figure 13 long description

Figure 13 shows a bar graph of planned sources of funding reported by Exploration grantees. For research contract, 77% reported no and 23% reported yes. For another country or countries, 68% reported no and 32% reported yes. For another funding organization, 42% reported no and 58% reported yes. For CIHR/NSERC/SSHRC grant, 24% reported no and 76% reported yes. For Transformation grant, 51% reported no and 49% reported yes. For Exploration grant, 38% reported no and 62% reported yes.

EQ3. What is helping or hindering the New Frontier in Research Funding’s progress to fill the gaps it was created to address? What could be improved?

NFRF was created to fund interdisciplinary and HRHR research led by Canadian researchers. NFRF aims to provide opportunities to conduct this type of research at an advanced level, to build Canadian researchers’ expertise and international reputation for IDR and HRHR. Novel and/or transformative insights and discoveries are expected to result from the projects funded by NFRF.

Findings on NFRF’s contributions to advancing interdisciplinary and HRHR research are covered above. A brief summary is provided at the end of this section. Factors affecting achievement to date on some related agency-level priorities are summarized next, including support for EDI, official languages and early career researchers (ECRs).

Equity, diversity and inclusion

NFRF has actively supported the advancement of EDI and was the first program to adopt EDI criteria in merit criteria across the tri-agencies. NFRF has incorporated EDI principles in terms of composition of research teams and how they function (EDI in research practice, EDI-RP) as well as in the design of the research itself (EDI in research design, EDI-RD). There is some evidence that NFRF may be contributing in terms of advancing EDI. Areas for improvement were identified to ensure meaningful collaboration with communities.

About one-third of project leads (NPIs and co-PIs) funded by NFRF identify as women (36%). For context, these results are compared with the representation of project leads by gender in the core between 2019 and 2023.^{Footnote 118} Representation of women among NFRF project leads is similar to that of CIHR-funded project leads (37%), higher than that of NSERC (23%) and lower than that of SSHRC (46%).^{Footnote 119} Of note, women had somewhat higher representation among NFRF leads of NS-related projects (33%) compared to their corresponding representation among leads of core NS-related projects (27%), a difference of six percentage points. For engineering-related projects, women had somewhat higher representation among NFRF project leads (30%) compared to their corresponding representation among leads of core engineering-related projects (21%), a difference of nine percentage points. No notable differences in success rate by self-identified gender^{Footnote 120} were observed.

Representation of members of visible minorities^{Footnote 121} was somewhat higher among NFRF project leads (34%) compared to core project leads (27%), a difference of seven percentage points.

Indigenous Peoples represent 2% of Exploration-funded project leads and 6% of Exploration-funded team members. In Transformation, Indigenous Peoples represent 10% of funded team members. For reference, representation of Indigenous Peoples among core-funded project leads ranged across the tri-agencies between 1% and 3%.

Individuals who self-identified as having a disability represent 4% of Exploration-funded project leads and 2% of Exploration-funded team members. These proportions were similar among all Exploration project leads (4%) and team members (3%). Representation of individuals with a self-identified disability was similar among CIHR-, NSERC- and SSHRC-funded project leads and team members (2-5%).

In end of grant reports, most Exploration grantees reported that the EDI measures they implemented resulted in positive impacts for the team or the project itself. A majority (84-86%) reported that their EDI measures had increased team diversity, improved team cohesion and/or strengthened collaborations among team members to a moderate or great extent.^{Footnote 122} A somewhat smaller proportion (70-73%) reported their EDI plans having an impact on the research to a moderate or great extent (e.g., inspired new avenues / new dimensions). Fewer than half of Exploration grantees (43%) reported that they have successfully implemented their planned gender-based analysis plus^{Footnote 123} elements to a moderate or great extent.

Additional considerations in terms of design elements related to EDI were identified by interview and focus group participants. The short project timelines of Exploration grants were identified^{Footnote 124} as a feature that could affect meaningful EDI implementation, particularly for projects requiring engagement with diverse communities. These limitations are important when sustained collaboration with underrepresented groups is necessary to build trust and mutual understanding, and also important for the research objectives (e.g., collaborations with Indigenous communities).

Focus group participants noted that some grant proposals appear to adopt the researchers’ institutional EDI policies without demonstrating a genuine understanding or commitment to them. Participants recommended clarifying guidelines to prioritize substantive and meaningful inclusivity over performative compliance.

Official languages

The majority of Exploration (97%) and Transformation applications (99%) were in English. This is somewhat higher than the proportion of English applications in the core over the same time period (91%).^{Footnote 125}

In terms of preferred official language,^{Footnote 126} individuals who identify their preferred language as English have higher application rates to Exploration and Transformation relative to their corresponding representation in the core overall. In contrast, lower application rates are observed for individuals who identify their preferred language as French. There are no observed differences in terms of success rate by preferred official language for either Exploration or Transformation.

Specifically, those identifying English as their preferred language represent 95% of Exploration-funded project leads, while those identifying French represent the remaining 5%. Similarly, those identifying English represent the majority of Exploration-funded team members (92%), while those identifying French represent the remaining 8%.^{Footnote 127} A larger majority of Transformation project leads identify English as their preferred language. Similarly, individuals who identify English as their preferred language represent 96% of Transformation team members, while those identifying French represent the remaining 4%.

For reference, among core-funded project leads, representation of individuals who identified their preferred language as English was somewhat lower (87%), and representation of those identifying French was somewhat higher (13%) relative to Exploration and Transformation. Similarly, among core-funded team members, representation of individuals who identified English as their preferred language was somewhat lower (82%) and representation of those identifying French was somewhat higher (17%). The higher representation of individuals who identify French as their preferred language among core project leads and team members is primarily driven by their higher representation among SSHRC-funded project leads and team members. Representation of individuals who identify French as their preferred language in Exploration and Transformation was similar to their corresponding representation among CIHR- and NSERC-funded project leads and team members.

Opportunities for early career researchers

The importance of supporting ECRs is recognized by the tri-agencies and the CRCC. Support for ECRs is also considered an important element of NFRF, and links directly to NFRF’s objective to provide opportunities for the development of expertise in interdisciplinary and HRHR/transformative research.

Overall, NFRF is clearly providing opportunities for ECRs to lead research projects, with about two-thirds (58%) of Exploration-funded projects having an ECR as the NPI. Notably, ECRs represented 100% of project leads in 2018, since that competition was open for ECRs only, although about half (51%) of 2022 competition projects had an NPI as the ECR.^{Footnote 128} The relatively higher representation of ECRs among Exploration applicants is not surprising, given that this stream encourages applications from this group of researchers.^{Footnote 129}

In contrast, only 9% (n=47) of Transformation proposals have been submitted by NPIs who are ECRs, none of which were successful (for context, success rates for Transformation proposals where the NPI is an established researcher ranged between 2.2% and 3.7%). These results may relate to the extensive experience needed to lead large teams/networks in Transformation, as recognized by some key informants in interviews, and the very competitive nature of Transformation.^{Footnote 130 ;Footnote 131}

Smaller academic institutions

Three-quarters (74%) of Exploration grants in scope have been hosted at large institutions. Of the remainder, 11% are hosted at other institutions,^{Footnote 132} and a smaller proportion are hosted at small (8%) and medium-sized (7%) institutions. Success rates have been the highest for projects hosted at large institutions (20%), and somewhat lower for proposed projects hosted at small institutions (13%).^{Footnote 133}

About two-thirds (62%) of Transformation grants (n=8)^{Footnote 134} have been hosted at large institutions, and 23% (n=3) have been hosted at other institutions. To date, there have been two Transformation projects hosted at medium-sized institutions, but none at small institutions.

In key informant interviews, several participants (n=12) highlighted that the larger universities have more capacity to support researchers to prepare a proposal and/or during the conduct of the research, including professional support. Participants described this as an enabler for researchers at large institutions, especially with respect to proposal writing. For example, institutional staff are involved in raising awareness of and encouraging applications to NFRF; providing support for writing sections of proposals; reviewing applications and providing feedback; arranging with current grant holders to provide new applicants with peer support; hiring external consultants; and providing teaching relief so applicants can focus on proposals and/or on their research if successful. Suggestions to offset differences in capacity in smaller institutions were offered by interview participants and were very similar to those noted above with respect to supporting disciplinary representation: supporting teaching relief; providing adequate time between stages of the application process for preparation by researchers with lower levels of support; and working with the institutional administration to encourage applications and provide accurate and timely information about NFRF competitions.

Summary of findings: New Frontiers in Research Fund’s contributions to advancing interdisciplinary and high-risk, high-reward research

Earlier sections of this report document findings on other enablers and potential hinderances to NFRF’s objectives, focusing specifically on progress to support interdisciplinary and HRHR research. A short summary follows.

NFRF is a young program with a mandate to foster a type of research that is otherwise constrained by conventional norms, structures and incentives in the research system. Overall, NFRF has already demonstrated a clear departure from “business as usual.”

Various enablers were identified in the evaluation. For example, interview participants voiced appreciation for the quality of support provided by NFRF program staff, including timely response to their many questions. Participants in NFRF’s merit review panels highlighted the importance of time and support by staff to enable rich discussions and deliberations of the multidisciplinary review panels.

Other enablers include strong design elements aligned with research on IDR and HRHR, embedded in NFRF at multiple levels, and recognized by key informants to be attracting and enabling the types of projects NFRF was created to fund. NFRF’s identity as a home for IDR and HRHR, and its relevance to researchers, was a dominant theme in interviews and focus groups.

NFRF’s objective is not solely to fund impactful research, but also to support Canadian researchers to acquire expertise and international reputation in IDR and HRHR. NFRF’s role to advance understanding and acceptance of these forms of research in Canada’s research ecosystem is important in that regard. In interviews, participants described a learning process for their institutions, for their peer researchers, and for themselves on how to envision, design and carry out this type of research, facilitated by NFRF. For example, in the words of one reviewer: “Through [my experience] reviewing and being part of these panels … I feel like I have a huge window, as a result, as a scholar, into what interdisciplinarity is. And it has helped me in my own research as well” [NFRF reviewer]. NFRF’s value to Canada’s reputation in the international research community was a point of consensus among focus group participants.

NFRF appears to enable or incentivize new collaborations at the federal funding level, and there is evidence that these collaborations are being sustained into subsequent research projects. In Exploration end of grant reports, nearly all grantees (97%) reported that they were more likely than before (i.e., prior to their Exploration grant) to pursue other interdisciplinary projects in the future.

Participation history of applicants supports interview findings that NFRF is attracting researchers with interest in IDR who are looking for a home for this kind of research. For example, as shown in Figure 14 below, a substantial proportion of NFRF applicants (29%) have a history of applying for and/or being funded by core programs from multiple agencies, indicating that their work relates to the mandates of more than one agency.

Figure 14 long description

Figure 14 shows two pie graphs of NFRF applicant participation history in the three agencies’ core programs between 2010 and 2022. In the first pie graph, 30% reported none, 29% reported multi-agency, 17% reported NSERC only, 14% reported CIHR only, and 11% reported SSHRC only. The second pie graph breaks down those who reported multi-agency, with 15% reporting CIHR and NSERC, 7% reporting CIHR and SSHRC, 4% reporting NSERC and SSHRC, and 3% reporting all agencies.

NFRF is positioned to leverage a synergy between interdisciplinarity and risk, which can be self-reinforcing to encourage innovation. For example, as noted above, openness to risk can provide researchers with freedom and incentive to expand interdisciplinary collaborations, and these collaborations are a source of novel ideas. Overall, NFRF attracts a striking diversity of proposed research projects, relative to the agencies’ core programs. NFRF-funded teams are substantially more diverse in terms of disciplinary composition than in the agencies’ core programs, and clearly include novel (uncommon) collaborations across research disciplines. There is also much higher density in connections across disciplines and research domains than found in the core. This higher density demonstrates a contribution by NFRF to expanding cross-disciplinary research networks in Canada.

While findings show clear progress, there are opportunities to further develop NFRF’s processes, in ways that can increase NFRF’s ability to achieve its intended long-term impact.

Overall, NFRF is attracting researchers from some disciplines more than others. This leads some to perceive NFRF as having an orientation to specific fields of research, which could further influence application rates and NFRF’s progress on IDR. Even with respect to “boundary disciplines,” NFRF is attracting researchers with backgrounds in some “boundary” disciplines at higher rates (e.g., biology and/or biomedical engineering), but not all. Researchers with backgrounds in other boundary disciplines (e.g., psychology) are not standing out in terms of participation in NFRF.

Variation is evident in how applicants and reviewers interpret some of NFRF’s merit criteria, particularly Interdisciplinarity, High-Risk and Feasibility criteria. A range of collaboration types have met NFRF’s threshold for interdisciplinarity. Relatedly, disciplinary diversity is better reflected in the composition of NFRF teams than in the subject matter of funded proposals, where the integration of disciplinary perspectives becomes visible.

Suggested areas for improvement

Suggestions and areas for improvement documented in the report are summarized below. These are intended as options for the program to consider when addressing recommendations.

Enhanced communication to applicants and reviewers

• Provide targeted communications to clarify NFRF’s expectations for: (a) disciplinary integration at the project level (i.e., beyond a multidisciplinary approach or as reflected only in the composition of the team), and (b) feasibility (i.e., with emphasis on capacity to adapt, given the uncertain nature of HRHR projects).
• Continue to emphasize to applicants that their proposal needs to communicate to a multidisciplinary review panel (e.g., communicate the big picture of the research project), as part of an interdisciplinary approach to merit review.
• Provide examples (e.g., case examples that are clearly above and below NFRF’s threshold level for interdisciplinarity) to aid with understanding of expectations for pass/fail.
• Provide feedback to applicants, for example, through an overall summary report such as the Exploration co-chairs’ reports that were published prior to 2021 (e.g., SSHRC, 2020a).

Opportunities to enhance conceptual clarity of merit criteria

Enhancing conceptual clarity of merit indicators may aid consistency in how they are understood and scored by reviewers. The adoption of overly specific (i.e., too narrow) language in the merit criteria could cause other challenges, since language and terminology vary across disciplines. Nonetheless, there are opportunities to improve clarity while retaining breadth/applicability to the diverse research ideas attracted by NFRF. Some specific considerations are:

• Enhance consistency/clarity in the merit assessment of interdisciplinarity by:
  • optimizing the existing wording of interdisciplinarity-related elements in the merit assessment matrices. Specific examples of updates that would reduce apparent inconsistency between elements and increase clarity are provided in this report’s section on NFRF’s selection processes for IDR;
  • organizing existing elements under the Interdisciplinarity criterion to allow reviewers to score on each key characteristic of IDR separately (i.e., diversity, novelty, integration, approach, design). For example, novelty and integration are currently combined under “Novelty of Perspective” in Exploration and could be separated, as they are in the grid for Transformation, to simplify and aid reviewer consistency in scoring;
  • collecting individual reviewer scores on each element to monitor overall patterns. These data could provide more specific information for the program on areas of strength or weaknesses in proposals related to IDR, and specific elements in the merit criteria that may need clarification for reviewers.
• Review the wording related to the Feasibility criterion, both in the description of its elements and other program documentation and reviewer guidance, to aid understanding of its intent, ensure key ingredients are present for response to the unexpected in novel/high-risk research, and be clear that prediction and advance mitigation of all risks is not assumed to be possible.

Consider providing support for applicants to develop strong teams and projects via pilot grants

Developing authentic/strong IDR projects with high levels of integration of perspectives from multiple disciplines requires involvement of researchers in identifying the research issue and developing the research project (i.e., projects that are interdisciplinary by design). It is evident that involvement of team members in developing the research proposal varies in depth and extent across projects proposed to and funded by NFRF. Incentives are needed to get buy in and investment of time and effort from co-applicants. Exploration grants have relatively short durations (two years), while Transformation grants are longer (six years) though very complex (e.g., very large teams ). Relationship building requires time before collaborations become productive.

NFRF may consider offering “stage one” funding and time allocation to support developing strong interdisciplinary proposals and community relationships in advance of an Exploration or Transformation project. For instance, providing initial funding may help incentivize higher levels of involvement from co-applicants in the development of the research proposal (and consequently, higher levels of integration in the design of research projects).

Consider providing support for NFRF grantees in early stages of their grants

Given that IDR and HRHR research are still not “business as usual” for the community, many grantees are expected to face unique challenges/barriers in early stages of implementing their research projects. After a grant is awarded, project teams need to be able to make progress quickly. This may be especially important for grantees who are new to this type of research, and/or grantees who lack resources within their own institutions equipped to advise grantees (e.g., institutions with no/few past NFRF grantees to offer peer support, or less resources/support from the vice-principal research offices). NFRF may consider opportunities to help grantees in the early stage of their grants, specifically to overcome barriers in their environments that are related to the novelty of the type of research NFRF funds and can hinder meaningful collaboration. For instance, NFRF may consider extending existing opportunities^{Footnote 135} for knowledge sharing to allow new grantees to learn from previous cohorts of NFRF grantees.

Continue to offer flexibility for funded projects, and consider additional flexibility for projects that are highly novel

Generally, projects that involve researchers from different areas of research and that incorporate novel approaches require longer timeframes to “ramp up” for implementation relative to conventional research. In addition, the novel nature of this type of research increases the risk of unanticipated challenges and, therefore, may require more time than originally anticipated to be feasibly implemented. This is especially relevant considering that most NFRF grants involve researchers who are formally collaborating on a research project for the first time. For these reasons, it is important for NFRF to continue to offer flexibility in Exploration’s duration (e.g., the automatic third year, no cost extension).

Additionally, as noted above, research projects supported by NFRF vary in terms of their nature (e.g., unique needs and timeframes for implementation and anticipated outcomes). Some projects may require additional time to be successfully implemented, due to their highly novel nature and/or because they require meaningful engagement with communities. In such cases, offering longer grant durations will better enable these projects to meet NFRF’s Feasibility criterion, be successfully implemented and achieve their intended outcomes.

Recommendations

Recommendation 1. If supporting interdisciplinary and high-risk, high-reward research remain priorities for the government, the program warrants continuation. NFRF is meeting a need and addressing an identified gap in terms of support for these areas of research.

The importance of the type of research NFRF is designed to fund continues to be highlighted as relevant to the research system. NFRF is well situated to address an identified gap, and is considered highly relevant by members of the research community. NFRF is clearly contributing to closing that gap, and is providing unique opportunities for Canadian researchers to pursue interdisciplinary and high-risk research with anticipated high reward.

It is important for NFRF to remain clear on its mandate to fund interdisciplinary and high-risk research (with anticipated high reward). There is a valuable synergy between NFRF’s primary objectives (interdisciplinarity, high risk and high reward). Nonetheless, tensions may arise if the program is asked to support some of its objectives in isolation (e.g., if NFRF is also asked to fill a gap in existing funding for high-risk, monodisciplinary research).

Recommendation 2. Build on existing strengths in the level of program support to reviewers and applicants.

NFRF is still unconventional among funding programs in terms of its emphasis on interdisciplinary and high-risk research. Normalizing these unique features of NFRF and allowing for learning in the research community are expected to take time and investment.

Maintaining high levels of program support is essential to facilitate understanding of NFRF’s expectations, as they are different from those of conventional funding programs, and support by program staff may need to continue at a high level. Maintaining strong support for the deliberations of the multidisciplinary review panel is especially important to help the multidisciplinary panel provide merit assessments from an interdisciplinary perspective. High quality panel review is enabled through program support and facilitation via synchronous communication and, if possible, in-person meetings. Allowing sufficient time and attention to each proposal is also important for high quality merit assessment, which is a function of the number of proposals each reviewer is assigned and the time available to them for review.

Recommendation 3. Revisit the Interdisciplinarity, High-Risk and Feasibility merit criteria and scoring process to provide more clarity and consistency.

The selection of research projects that are most aligned with NFRF’s funding objectives will determine NFRF’s outcomes. The merit criteria and scoring process are the instruments through which NFRF’s expectations are implemented. Continuing efforts to optimize the merit review criteria will support understanding of NFRF’s expectations and aid reviewers and applicants in determining threshold levels for interdisciplinarity and risk (i.e., how much is enough). This is important to facilitate merit assessment of “borderline” proposals, and ensure continued alignment of funded projects with NFRF’s mandate. A primary point of focus is to improve the reliability of the merit review matrices, and consistency in how they are applied, as they are the primary instrument used by both applicants and reviewers. To this end, the program is encouraged to:

1. Refine the Interdisciplinarity criterion to provide more clarity and to support consistency in scoring. In order to ensure continued alignment of funded proposals with NFRF’s expectations for IDR, it is important to continue to emphasize the distinction between interdisciplinary and multidisciplinary research, as this is a key point of distinction for NFRF from other funding programs. Additional options for consideration are documented in the report (see Suggested areas for improvement). They include reviewing the elements in the criterion to reduce the appearance of inconsistency, and simplifying elements for scoring to aid applicants and reviewers in determining what warrants a pass score for Interdisciplinarity.
2. Provide enhanced clarity for the High-Risk and Feasibility criteria. Perceived conflict between NFRF’s High-Risk and Feasibility criteria is evident, and reflects challenges with interpretation by applicants and reviewers. This is partly because NFRF’s Feasibility criterion departs from the way feasibility is approached in merit review in conventional programs. As noted earlier in the report, misapprehension of the Feasibility criterion may lead applicants to constrain the level of risk in their proposals. Review of the wording of the elements of the Feasibility criterion is recommended to clarify its intent, i.e., to ensure the project includes key ingredients to enable response to the unexpected, rather than expecting prediction and mitigation of all risks, given the uncertain nature of HRHR projects.
3. Revisit Exploration’s “Interdisciplinarity subcriterion: Fit to Program.” A pass/fail subcriterion titled “Interdisciplinarity: Fit to Program” was added to Exploration in 2019, aimed at reducing applications in which the proposed collaboration is not novel, i.e., applications in which the disciplines involved already “have an established tradition of collaborating” (2018 Co-chairs’ Report [SSHRC, 2019]). The original wording specified fit to program in terms of a proposal’s interdisciplinarity and high-risk nature. However, the wording was changed in 2021 to refer to interdisciplinary and/or high-risk. Although the subcriterion is still labeled “Interdisciplinarity subcriterion,” the wording effectively allows applications to receive a pass on high-risk features alone. This could result in deviation from the original intent of this subcriterion. NFRF’s mandate is to support interdisciplinary and HRHR research, and the title and description of this subcriterion (fit to program) and its elements should remain aligned with this mandate.
4. Revisit the level of discretion offered to reviewers when using the criteria. NFRF allows some discretion to reviewers in terms of deciding which elements to consider when determining the overall score (pass/fail) for the Interdisciplinarity and High-Risk criteria. Academic literature supports providing such flexibility in the merit review of this type of research and suggests that merit criteria should not be too rigid but, rather, adaptive to diverse forms of interdisciplinary and HRHR research. Nonetheless, too much flexibility contributes to confusion and variation in assessment. It may not be necessary to allow the current level of discretion, while still allowing for flexibility. NFRF should introduce boundaries or more guidance on what reviewers include in their merit assessment beyond what is identified in the matrix. Other options for consideration are documented above in the section “NFRF’s selection processes for interdisciplinary research.”

Recommendation 4. Continue to reinforce that the program is open to all research domains through program communications and documentation, and monitor participation and success rates by discipline.

The predominance of some natural sciences and engineering disciplines in NFRF teams and projects is clearly observed. In contrast, lower participation rates are observed for social sciences and humanities. These trends are generally consistent across different lines of evidence. Relative to their respective representation in core teams and projects, these differences seem to be mainly attributable to higher or lower application rates for researchers from specific fields of research (e.g., biomedical engineering), rather than differences in success rates. Multiple other factors are also expected to influence these trends (e.g., differences in characteristics of core programs in the respective domains).

It is important to recognize that NFRF does not have specific objectives for ensuring equal representation by domain, nor a specific expectation for representation of domains in NFRF to be proportional to their respective representation across core programs overall. Nonetheless, it is important to continue to ensure that NFRF is open to all research domains, by monitoring application and success rates and for the presence of any barriers specific to some research areas.

References

Advisory Panel on Federal Support for Fundamental Science. (2017). Investing in Canada’s Future: Strengthening the foundations of Canadian research. In Canada’s Fundamental Science Review. https://ised-isde.canada.ca/site/canada-fundamental-science-review/sites/default/files/attachments/2022/ScienceReview_April2017-rv.pdf

Bark, R. H., Kragt, M. E., & Robson, B. J. (2016). Evaluating an interdisciplinary research project: Lessons learned for organisations, researchers and funders. International journal of project management, 34(8), 1449-1459. https://doi.org/10.1016/j.ijproman.2016.08.004

Boon, M., & Van Baalen, S. (2019). Epistemology for Interdisciplinary research—shifting philosophical paradigms of science. European Journal for Philosophy of Science, 9(16). https://doi.org/10.1007/s13194-018-0242-4

Boudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2016). Looking across and looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation in science. Management science, 62(10), 2765-2783. https://doi.org/10.1287/mnsc.2015.2285

Brennan, M., & Rondón-Sulbarán, J. (2019). Transdisciplinary research: Exploring impact, knowledge and quality in the early stages of a sustainable development project. World Development, 122, 481-491. https://doi.org/10.1016/j.worlddev.2019.06.001

Bromham, L., Dinnage, R., & Hua, X. (2016). Interdisciplinary research has consistently lower funding success. Nature, 534(7609), 684-687. https://doi.org/10.1038/nature18315

Brown, R. R., Deletic, A., & Wong, T. H. (2015). Interdisciplinarity: How to catalyse collaboration. Nature, 525(7569), 315-317. https://doi.org/10.1038/525315a

Bruun, H., Hukkinen, J. I., Huutoniemi, K. I., & Thompson Klein, J. (2005). Promoting interdisciplinary research: The case of the Academy of Finland. Academy of Finland. http://hdl.handle.net/10138/232267

Cao, L., & Zhang, Z. (2022). Developing Science and Technology Policies for High Risk-High Reward Research. Bulletin of Chinese Academy of Sciences (Chinese Version), 37(5), 661-673. https://doi.org/10.16418/j.issn.1000-3045.20220108001

Choi, B. C., & Pak, A. W. (2006). Multidisciplinarity, interdisciplinarity and transdisciplinarity in health research, services, education and policy: 1. Definitions, objectives, and evidence of effectiveness. Clinical Investigative Medicine, 29(6), 351–364. https://domteamscience.stanford.edu/wp-content/uploads/2023/02/Multidisciplinarity-Interdisciplinarity-and-Transdisciplinarity-in-Health-Research-Services-Education-and-Policy-1.-Definitions-Objectives-and-Evidence-of-Effectiveness.pdf

de Oliveira, T. M., Amaral, L., & Pacheco, R. C. D. S. (2019). Multi/inter/transdisciplinary assessment: A systemic framework proposal to evaluate graduate courses and research teams. Research Evaluation, 28(1), 23-36. https://doi.org/10.1093/reseval/rvy013

Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs—principles and practices. Health services research, 48(6pt2), 2134-2156. https://doi.org/10.1111/1475-6773.12117

Gleed, A., & Marchant, D. (2016). Interdisciplinarity: Survey report for the Global Research Council 2016. DJS Research. https://globalresearchcouncil.org/fileadmin/documents/GRC_Publications/Interdisciplinarity_Report_for_GRC_DJS_Research.pdf

Hoffmann, S., Pohl, C., & Hering, J. G. (2017). Exploring transdisciplinary integration within a large research program: Empirical lessons from four thematic synthesis processes. Research Policy, 46(3), 678-692. https://doi.org/10.1016/j.respol.2017.01.004

Huutoniemi, K. (2012). Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science, 42(6), 897-921. https://doi.org/10.1177/0306312712458478

Klein, J. T., & Falk-Krzesinski, H. J. (2017). Interdisciplinary and collaborative work: Framing promotion and tenure practices and policies. Research Policy, 46, 1055-1061. https://doi.org/10.1016/j.respol.2017.03.001

Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Harvard University Press.

Leahey, E., Beckman, C. M., & Stanko, T. L. (2017). Prominent but less productive: The impact of interdisciplinarity on scientists’ research. Administrative Science Quarterly, 62(1), 105-139. https://doi.org/10.1177/0001839216665364

Leigh, J., & Brown, N. (2021). Researcher experiences in practice-based interdisciplinary research. Research Evaluation, 30(4), 421-430. https://doi.org/10.1093/reseval/rvab018

Lengwiler, M. (2006). Between charisma and heuristics: Four styles of interdisciplinarity. Science and Public Policy, 33(6), 423-434. http://dx.doi.org/10.3152/147154306781778821

Lyall, C. (2019). Being an interdisciplinary academic: How institutions shape university careers. Springer.

Lyall, C., & Fletcher, I. (2013). Experiments in interdisciplinary capacity-building: The successes and challenges of large-scale interdisciplinary investments. Science and Public Policy, 40(1), 1-7. https://doi.org/10.1093/scipol/scs113

Machado, D. (2021). Quantitative indicators for high-risk/high-reward research. OECD Science, Technology and Industry Working Papers, 2021(7), 2-12A. https://dx.doi.org/10.1787/675cbef6-en

Mansilla, V. B. (2006). Interdisciplinary Work at the Frontier: An empirical examination of expert interdisciplinary epistemologies. Issues in Integrative Studies, 24, 1-31.

Moreno, M. D. C. C., Kynčlová, P., & Werthner, H. (2016). A multiple-perspective analysis of doctoral interdisciplinarity. In 2016 15th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1-11). IEEE. https://doi.org/10.1109/ITHET.2016.7760722

NASA. (2018). Gaudi: High-Risk, High-Reward (HRHR) Research Portfolio. Retrieved from, https://smd-cms.nasa.gov/wp-content/uploads/2023/04/Gaudi_hrhr_APAC_April2018.pdf

Newman, J. (2024). Promoting interdisciplinary research collaboration: A systematic review, a critical literature review, and a pathway forward. Social Epistemology, 38(2), 135-151. https://doi.org/10.1080/02691728.2023.2172694

Nowell, L., Paolucci, A., Dhingra, S., Jacobsen, M., Lorenzetti, D. L., Lorenzetti, L., & Oddone-Paolucci, E. (2022). Interdisciplinary mixed methods systematic reviews: Reflections on methodological best practices, theoretical considerations, and practical implications across disciplines. Social Sciences & Humanities Open, 6(1), 100295. https://doi.org/10.1016/j.ssaho.2022.100295

Organisation for Economic Co-operation and Development. (2021). Effective policies to foster high-risk/high-reward research. OECD Science, Technology and Industry Policy Papers, 112. https://www.oecd.org/content/dam/oecd/en/publications/reports/2021/05/effective-policies-to-foster-high-risk-high-reward-research_131670a3/06913b3b-en.pdf

Petts, J., Owens, S., & Bulkeley, H. (2008). Crossing boundaries: Interdisciplinarity in the context of urban environments. Geoforum, 39(2), 593-601. https://doi.org/10.1016/j.geoforum.2006.02.008

Ramadier, T. (2004). Transdisciplinarity and its challenges: The case of urban studies. Futures, 36(4), 423-439. https://doi.org/10.1016/j.futures.2003.10.009

Rodela, R., & Alašević, D. (2017). Crossing disciplinary boundaries in environmental research: Interdisciplinary engagement across the Slovene research community. Science of the Total Environment, 574, 1492-1501. https://doi.org/10.1016/j.scitotenv.2016.08.144

Rusco, F. (2011). High-Risk Series: An Update (Report No. GAO-11-127R). United States Government Accountability Office. https://www.gao.gov/products/gao-11-278

Social Sciences and Humanities Research Council. (2018a). Canada Research Coordinating Committee Consultation: Summary Report on Proposed Measures to Strengthen Science in Canada. https://publications.gc.ca/collections/collection_2019/crsh-sshrc/CR22-109-2018-eng.pdf

Social Sciences and Humanities Research Council. (2018b). Manual for Multidisciplinary Review Panel: 2018 Exploration. https://sshrc-crsh.canada.ca/funding-financement/nfrf-fnfr/exploration/2018/reviewers_manual-guide_de_l_evaluateur-eng.aspx

Social Sciences and Humanities Research Council. (2019, July 03). 2018 New Frontiers in Research Fund—Exploration stream competition: Co-Chairs’ Report. https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/exploration/2018/co-chair_report-rapport_des_copresidents-eng.aspx

Social Sciences and Humanities Research Council. (2020a, May 14). Feedback to Applicants: 2018 Exploration. https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/exploration/2018/feedback_to_applicants-observations_destinees_aux_candidats-eng.aspx

Social Sciences and Humanities Research Council. (2020b, June 24). Co-chairs’ Report: 2019 Exploration. https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/exploration/2019/co-chair_report-rapport_des_copresidents-eng.aspx

Social Sciences and Humanities Research Council. (2021, May 3). 2020 Exploration Multidisciplinary Review Panel Co-chairs’ Report. https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/exploration/2020/co-chair_report-rapport_des_copresidents-eng.aspx

Social Sciences and Humanities Research Council. (2022, May 11). 2021 Exploration Multidisciplinary Review Panel Co-chairs’ Report. https://www.sshrc-crsh.gc.ca/funding-financement/nfrf-fnfr/exploration/2021/co-chair_report-rapport_des_copresidents-eng.aspx

Social Sciences and Humanities Research Council. (2025). 2025 Exploration: Merit Indicators for the Review Process. https://sshrc-crsh.canada.ca/funding-financement/nfrf-fnfr/exploration/2025/merit_indicators-indicateurs_du_merite-eng.aspx

Standing Committee on Science and Research. (2022). Successes, challenges and opportunities for science in Canada. https://www.ourcommons.ca/Content/Committee/441/SRSR/Reports/RP11841016/srsrrp01/srsrrp01-e.pdf

Standing Committee on Science and Research. (2023). Pursuing a Canadian moonshot program. https://www.ourcommons.ca/Content/Committee/441/SRSR/Reports/RP12490814/srsrrp04/srsrrp04-e.pdf

Staudt, J., Yu, H., Light, R. P., Marschke, G., Börner, K., & Weinberg, B. A. (2018). High-impact and transformative science (HITS) metrics: Definition, exemplification, and comparison. PloS one, 13(7), e0200597. https://doi.org/10.1371/journal.pone.0200597

Villeneuve, D., Durán-Rodas, D., Ferri, A., Kuttler, T., Magelund, J., Mögele, M., Nitschke, L., Servou, E., & Silva, C. (2020). What is Interdisciplinarity in Practice? Critical Reflections on Doing Mobility Research in an Intended Interdisciplinary Doctoral Research Group. Sustainability, 12, 197. https://doi.org/10.3390/su12010197

Von Wehrden, H., Guimarães, M. H., Bina, O., Varanda, M., Lang, D. J., John, B., ... & Lawrence, R. J. (2017). Interdisciplinary and transdisciplinary research: Finding the common ground of multi-faceted concepts. Sustainability Science, 14(3), 875-888. https://doi.org/10.1007/s11625-018-0594-x

Appendix A—Details on Methods

Overview of design

A mixed methods design was used for this evaluation, in which qualitative and quantitative methods were implemented in stages, allowing results to inform subsequent stages in an iterative way (Fetters et al., 2013). Nearing the end of the study, results from all individual lines of evidence were again compared for convergence, allowing for reassessment and further development of some results.

The quantitative methods made use of the agencies’ flagship program data (core programs) as a baseline or “business as usual” comparator, given that one of the primary objectives behind NFRF’s creation was to fill a gap in the agencies’ regular funding programs. The use of the core as a naturally occurring comparison group carries limitations, because differences between a baseline and program group may occur for a variety of reasons, including reasons unrelated to the program, not all of which can be identified or controlled for. When used in evaluations, such comparisons are typically accompanied with complementary methods, to help identify and account for alternative explanations for differences between the groups. The use of multiple methods and their sequencing allowed for analysis of this kind. Overall, the purpose is to assess whether observed effects would have occurred without the program being evaluated. The findings and recommendations in this report are based on this overall evaluative synthesis.

Social network analysis and administrative data analysis

Scope. Social network analysis (SNA), as well as conventional descriptive analyses, was conducted using administrative data on NFRF and core programs. This includes data on N=156,890 projects and N=83,845 unique individuals ^{Footnote 136} across NFRF and core funding opportunities (Fos) in scope. FOs and timeframes in scope are shown in the table below.

Agency	Funding opportunity	Timeframe covered
Tri-agency	NFRF Exploration	2018-22
Tri-agency	NFRF Transformation	2020-22
CIHR	Project Grants	2016-23
CIHR	Foundation Grants	2014-18
CIHR	Operating Grants	2010-15
NSERC	Discovery GrantsFootnote 137	2010-23
NSERC	Discovery Horizons	2022-23
NSERC	Collaborative Research and Development Grants	2010-19
NSERC	Alliance	2019-23
SSHRC	Insight Grants	2012-23
SSHRC	Insight Development Grants	2011-23
SSHRC	Partnership Grants	2011-23
SSHRC	Partnership Development Grants	2010-22
SSHRC	Partnership Engage Grants	2017-23

Research domain and discipline classification. Division- and group-level field of research (FoR) codes from the Canadian Research and Development Classification (CRDC) (v.1) were used as a common classification system across the dataset. Natural language processing was used to identify the research domains (CRDC divisions) and disciplines (CRDC groups) relevant to funding applications and researcher disciplinary background in the dataset.

Of note, in its eligibility criteria, NFRF references the mandates of the three agencies: the Canadian Institutes of Health Research (CIHR), Natural Sciences and Engineering Research Council (NSERC) and Social Sciences and Humanities Research Council (SSHRC). Alignment between the CRDC domains and agency mandates is not perfect. Some disciplines are internally diverse, i.e., the work of subgroups of researchers within the same discipline can fall under the mandate of different agencies, and some individual researchers also work across areas of more than one agency. A few examples include biological sciences (medical health and life sciences [MHL] and natural sciences [NS]), psychology (social sciences [SocSci], humanities and MHL) or computer science (NS, MHL, SocSci and humanities). These are referred to as “boundary disciplines” in this report. Efforts were taken to consider boundary disciplines in the analysis and to draw attention to them in this report, because projects may be consistent with NFRF criteria (crossing agency mandates) while not crossing CRDC division- or group-level categories.

Disciplinary classification process. Natural language processing was used to identify the research domains (CRDC divisions) and disciplines (CRDC groups) relevant to funding applications and researcher disciplinary background in the dataset. To classify funding applications, three types of input data (training data) were used for the machine learning: funding proposal abstracts; proposal titles; and applicant-submitted, discipline-related keywords. Classification for individual-level research background used self-reported, discipline-related keywords on funding applications, as well as individual participation history across NFRF and core programs between 2011 and 2023.

The training data and output data from the machine learning process were human validated as a quality check, to increase confidence that the disciplinary classifications correspond to the disciplines central to a researcher’s expertise and the subject matter of their research projects/proposals. Human validation was performed by 12 analysts, drawn from evaluation and data teams of the three agencies, supported through consultations with subject matter experts. It is important to recognize that the machine learning process carries limitations; for example, sensitivity is greater where there is more input data (training data) available. As the algorithm was trained using 12 years of core data, from 2011 to 2023, it may be less sensitive in detection of novel areas of research (e.g., nanomedicine, Indigenous research methodologies) that may be less represented in the training dataset.

Diversity and novelty indices. Two main quantitative indices of interdisciplinarity were calculated for assessing interdisciplinarity: diversity and novelty. These indices were calculated at three levels: individuals, teams and projects.

• Disciplinary diversity assessed the extent to which connections among domains/disciplines were observed at that level of analyses.
• Novelty assessed the extent to which discipline combinations are rare versus common within the core. Novelty scores were assigned on a continuous scale from 0 to 1, based on prevalence in the core. For example, although both of the following pairs cross domains, a collaboration between clinical medicine and psychology is relatively common in core teams (resulting in a lower novelty score of 0.00054), while a collaboration between civil engineering and psychology is less common in the core (resulting in a higher novelty score of 0.02083).

Existing and new relationships. NFRF’s role in fostering new relationships among researchers was explored using the SNA dataset. A pair of researchers was described as an existing relationship if they had a joint application or grant from one of the core programs prior to their involvement in a NFRF application/grant. Otherwise, the relationship between this pair of researchers (as part of the NFRF application/grant) was categorized as a new relationship. NFRF’s role in developing relationships among researchers was also explored in depth via the key informant interviews (see below).

Key informant interviews

A total of 41 interviews was conducted with 42 grantees, unsuccessful applicants and review panel members who participated in Exploration and/or Transformation streams between 2018 and 2022. Interviews were 45 to 60 minutes in length, conducted virtually. Full transcriptions were produced for each interview for coding and analysis.

Sampling. Purposeful sampling was used to select information-rich cases, that is, participants with knowledge and experience of the NFRF Exploration and/or Transformation streams who could provide in-depth information relevant to the evaluation questions and indicators. For example, the NPI was invited for all unsuccessful applicant / grantee interviews, as they were more likely to provide insight regarding questions of interest. Applicants that did not reach the full application stage (beyond notice of intent [NOI] / letter of intent [LOI] stages) were excluded. Other selection criteria included the number of competitions a reviewer was involved in, and participation history in the core and Tri-Agency Interdisciplinary Peer Review Committee. The 2018 Exploration competition was excluded,^{Footnote 138} due to differences in its characteristics from subsequent Exploration competitions. Within the above criteria, selection focused on ensuring representation of different research domains, official languages, early career researcher / established career status, regions, and institution sizes. Overall, the strategy focused on diverse perspectives on aspects of NFRF’s grant proposal and merit review processes; dynamics of collaborations within NFRF research teams; the nature of interdisciplinarity, risk and novelty in NFRF research projects; and other NFRF design elements.

Instruments. An interview guide for each participant group was created by the SSHRC/NSERC Evaluation Division. The guides were piloted by the external consultant supporting this component of the evaluation. Guides were adjusted following three pilot interviews and based on feedback from the consultant.

Analysis. Analysis was conducted in two phases. First cycle coding was conducted by the external consultant in NVivo on 70% of the interviews (n=29). First cycle coding followed a framework developed from the evaluation indicators and independent coding of the first three transcripts by three analysts. Results were reviewed in consensus discussions between the Evaluation Division and the consultants, to finalize the framework for first cycle coding. Second cycle subcoding was conducted by two analysts in the Evaluation Division using NVivo. This coding was inductive using an analytical approach.^{Footnote 139} Larger discussions with the evaluation team, the external consultants supporting the interviews component, and the external consultants supporting the SNA component led to coding at greater depth. Themes developed through coding were assessed for internal coherence, redundancy and saturation. During this process, emergent codes were mapped back to the evaluation questions and indicators. A technical report was reviewed by both teams of external consultants as a quality check.

Focus groups

Two focus groups were held with members of the multidisciplinary/multisectoral review panel for Exploration (n=5) and Transformation (n=2). Participants were selected from reviewers who had not already participated in key informant interviews and who participated in the competitions between 2018 and 2022. These focus groups were held at the end of the evaluation as a validation check on interpretation of results. They were led by the external consultant supporting the key informant interviews. Focus group guides were developed collaboratively between these consultants, the Evaluation Division, and the consultants supporting the SNA. Network maps created through the SNA process were used to focus the discussion, and focus group participants were invited to offer their interpretation and critique. Thematic analysis of the focus group data was conducted by the external consultant.

Literature and document reviews

Two literature reviews were conducted. The first focused on academic and grey literature related to interdisciplinary research (34 articles), and the second on high-risk/high-reward and transformative research (45 articles). Articles were selected using a systematic search and screening process in Covidence. The literature reviews helped contextualize NFRF’s definitions of these constructs, to inform their operationalization to support other lines of evidence, and to contextualize results.

A review of program documentation included documentation related to NFRF’s purpose, intended outcomes, and application and review processes.^{Footnote 140} Exploration end of grant reports were also reviewed (n=295^{Footnote 141}). The review of these documents allowed for documentation of any changes over time to NFRF’s guidance to applicants and reviewers, design elements and processes (e.g., application and selection processes), and participant-generated reports (e.g., from co-chairs and grantees) relevant to the evaluation questions.

Results from the literature review, the document review and the key informant interviews were assembled into analytic matrices for triangulation; for example, to compare and contrast characteristics of interdisciplinary research referenced in the NFRF program literature to those identified in the academic and grey literature and described by key informants.

Appendix B—Summary of New Frontiers in Research Fund Proposal Review Process

Exploration

The Exploration stream held its first competition in 2018. All Exploration competitions have had a Notice of Intent (NOI)^{Footnote 142} and Full Application stage.^{Footnote 143}

Assessments of full applications in each competition have been conducted by a multidisciplinary review panel, with each application assigned five members from the panel. Two to three members are assigned based on their research expertise, matching the primary areas of research of the proposal, while the remaining members provide assessments from a non-expert perspective. Proposals with the highest rankings (one-third of proposals) are discussed by the committee at a panel meeting, and the panel arrives at a final assessment. A list of recommended applications is sent to NFRF’s steering committee, which makes the final decision on which applications will receive funding.

Modifications to elements and the process have taken place over time. A few examples include the double-anonymous external expert reviewers introduced as part of the merit review process in 2019 and removed in 2023; a Letter of Intent (LOI) stage introduced in 2019 and removed in 2020; variation in competition timelines (a gradual increase in the overall timeline); and edits to merit criteria language.

Transformation

The Transformation stream has held three competitions to date, all of which included three application stages: an NOI, then an LOI to apply, then Full Application. NOIs have been used only for administrative purposes, to help the NFRF program team identify reviewers.

Each LOI submission is assigned to five members from a multidisciplinary/multisectoral review panel for review. Language abilities and expertise are taken into account when assigning LOIs. The most meritorious proposals are invited to submit a full application. Full applications are reviewed by an expert panel, consisting of five to seven expert reviewers, and then evaluated by the multidisciplinary/multisectoral review panel.

A subset of teams from the highest ranked applications (which were deemed to meet all criteria/objectives at the Full Application review stage) are invited to present their proposals to a jury composed of 10 to 15 international and Canadian members representing diverse sectors.^{Footnote 144} At this stage, proposals are identified by the jury to recommend for funding. The NFRF Steering Committee makes the final decisions on which applications to fund.

The responsibilities of the multidisciplinary review panel, NFRF staff, the international and multisectoral jury, the steering committee, and external reviewers did not change throughout the years. The merit criteria language, especially on the Interdisciplinarity criterion, was mostly consistent.