What Does Research Impact Actually Do?

The concept of research impact pervades contemporary academic discourse. Its prominence can be viewed as part of an increasing demand for academic institutions and individual researchers to demonstrate the practice and policy implications of their work. Yet, despite its widespread usage, research impact remains an unclear and contested concept.

Search for a conceptual definition of research impact

As Australia-based researchers with an interest in translating mental health research into policy, we reviewed literature to find a conceptual definition that would guide academic inquiry into how mental health research impacts mental health policy. We chose the systematic review method because this is generally considered the “gold standard” of academic inquiry in health and medical sciences, and involves the methodical collection and critical analysis of literature in a way that facilitates quantification of findings. Thus, using a systematic review methodology enabled us to identify and quantify the prominence of different definitions and the presence and absence of research impact definitions in the literature.

Often cited, rarely defined

We reviewed 886 academic and grey literature sources (websites, online reports, and conference papers). In these, we identified 108 definitions in 83 publications. We found that only 23% – or 45 out of 200 – peer-reviewed articles to discuss research impact actually defined it. Literature that discusses and defines research impact was found to be growing, with 76% of sources having a publication date of 2011 or later.

Bureaucratic foundations

Our review identified that the concept of research impact appears to be a term of research governance rather than an academic conceptualisation. We found that 76% of the definitions either cited or were created by research organisations or funding institutions. Peer-reviewed journal articles that defined research impact often sourced definitions from grey literature.

Half of research impact definitions are from the UK

Our review also identified the United Kingdom as the epicentre of the research impact concept, contributing half (51%) of the definitions. The remainder were found in Australian (22%), other European (16%), and North American literatures (12%, combining the United States of America and Canada). These findings confirm prior research on the research impact agenda that locates its origins in the political history of the UK.

Four types of definitions

We identified four types of research impact definitions:

• as “the demonstrable contribution that excellent research makes to society and the economy”(the Research Councils United Kingdom)
• as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (the Higher Education Funding Council for England and the Research Excellence Framework)
• as measurable influences in the form of quantifiable data such as citation frequency (bibliometric definitions)
• as the influences of research results on the knowledge and actions of researchers and policymakers (use-based definitions).

Four core elements of definitions

We identified four domains that underpinned the research impact concepts reviewed (Figure 1):

• contribution (the areas of research influence; e.g. economy, policies)
• avenues of impact (processes by which research could have impact; e.g. effects on knowledge, attitudes)
• change (synonyms used to describe “effects” or “benefits”)
• levels of impact (e.g. national, international).

Figure 1: Concepts that underpin research impact definitions.

Policy was expressly noted in approximately half (52%) of definitions. These definitions mostly described research impact on policy in terms of the impacts of “good-quality” academic activities and in relation to the impacts of research in changing the awareness, behaviour, or attitudes of policymakers. The definitions we reviewed tended not to describe impacts on specific policy decisions.

Critique of bureaucratic definitions

Investigating the complexity of the concept of “research impact” helps highlight underlying tensions between its uses in research governance and in dedicated academic research on knowledge translation and implementation. This tension may mirror the different practices and discourses that exist between research and policy realms, that others have noted.

One key tension is the bias, from a research perspective, of utilising research governance definitions for academic inquiry that relates to the ways these definitions unproblematically equate “impact” with “benefit”. Adopting a positive definition means that negative impacts of research would be ignored, and thus not contribute to collective understanding of the processes of knowledge translation and implementation.

A proposed academic conceptualisation of research impact

On the basis of our review, and given our specific interest in understanding the impact of mental health research on mental health policy, we propose the following definition of research impact as:

“A direct or indirect contribution of research processes or outputs that have informed (or resulted in) the development of new (mental) health policy/practices, or revisions of existing (mental) health policy/practices, at various levels of governance (international, national, state, local, organisational, health unit).”

Whilst our focus is mental health, the definition is readily adaptable to other areas of health and policy. This definition aims to capture the concepts, avenues, and levels of research impact and be inclusive of the hard-to-measure, and often non-linear, ways in which research can inform policies. It entails an examination and attention to both policy content and policy processes.

Towards conceptual clarity and academic understanding of an academic zeitgeist

Research impact is an increasingly used, but often undefined term. There are some important limitations in predominant definitions of research impact that we need to be aware of that may lead to bias if we are to advance academic understandings of how research informs policies and practice. We have proposed a conceptual definition of research impact on policy that we think is suited to various health disciplines and contexts to facilitate a better understanding of what research impact means. The next step is to apply, evaluate, and improve this definition.