By Ann-Louise Caress

What is the Research Excellence Framework?

The ‘Research Excellence Framework’ or ‘REF’ is a periodic activity undertaken to assess the quality of research in universities in the United Kingdom (see www.ref.ac.uk). It was formerly known as the ‘Research Assessment Exercise’ (RAE). The REF is very important because it compares research quality across comparable subjects and determines the level of funding available to universities from the four UK higher education funding bodies. The next REF will take place in 2014, so UK universities are well underway towards developing their submissions. These will be assessed through rigorous review by subject-specific panels, which may include lay representation. Submissions are assessed on the type and quality of research publications; volume and source of research funding and quality of the research environment. For the first time, the REF in 2014 will also include assessment of the ‘impact’ of research.

REF ‘impact criteria’

Given that this is the first time that ‘impact’ has been included in one of these nationwide assessments of research quality in the UK, exactly what constitutes a good example of ‘impact’ is still evolving. REF guidance defines ‘impact’ as follows:

‘For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’  (HEFCE et al 2012a)

The emphasis on impact being ‘beyond academia’ is important. It means that researchers are now required not just to demonstrate ways that their research is highly regarded by fellow researchers, but also how it is ‘making a difference’ more broadly. In order to do this, each REF submission will have to include several ‘impact case studies’.

The broad criteria for assessing these case studies will be the ‘reach’ and significance’ of the impact reported (HEFCE et al 2011). This relates to how widespread the benefit of the ‘impact’ is and how important or meaningful it is to relevant stakeholders.

Does public involvement have a role in REF ‘impact’ case studies? 

A role for patient and public involvement in, or engagement with research in creating or demonstrating ‘impact’, as defined by the REF, has been identified:

‘Impacts or benefits arising from engaging the public with the submitted unit’s research will be included’ (HEFCE 2011)

Although this definition refers only to ‘engagement’, guidance for the REF panel to which much health and social care research will be submitted does make explicit mention of ‘involvement’ and emphasises that:

‘There must be a clear link between the research and the engagement or involvement activity’ (HEFCE 2012b)

It further states that:

‘The activity should go beyond ‘business as usual’ engagement or involvement (for example, there was active involvement of service users and/or the public, the activity informed the focus of the research or created widespread interest, was particularly innovative, or created legacy resources)’ (HEFCE 2012b)

Challenges

The REF definition of ‘impact’ (see above) is clear that it cannot be on the research process/research itself, but must have a wider remit. This presents a considerable challenge, given that there is a limited body of evidence regarding the ‘impact’ of patient and public involvement/engagement on ‘outcomes’ as opposed to ‘process’ in research (Staley 2009).

Demonstration of causal links is challenging in many fields, but is especially the case for patient and public involvement in/engagement with research. How is it possible, for example, to ‘prove’ that such involvement was the ‘vital ingredient’ that resulted in the ‘impact’ (and, indeed, should one even strive to demonstrate this)?

All impact case studies must be supported by relevant ‘evidence’. This includes, but is not limited to, publications. Other examples of relevant ‘evidence’ may include demonstrable changes in policy or practice, reports in the mass media or testimonials from relevant stakeholders. Given its novelty in assessment of research quality, many researchers are struggling to provide sound evidence of impact, as this has often not been adequately captured/recorded to date – and patient and public involvement/engagement is by no means immune from this problem!

References

Higher Education Funding Council for England (HEFCE), Scottish Funding Council (SFC), Higher Education Funding Council for Wales (HEFCW)  and Department for Employment and Learning Northern Ireland (DELNI) (2011) Decisions on Assessing Research Impact’ (REF 01.2011) www.ref.ac.uk/media/ref/content/pub/decisionsonassessingresearchimpact/01_11.pdf

 

HEFCE, SFC, HEFCW, DELNI (2012a) Assessment framework and guidance on submissions (02.2011 updated version) www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf

HEFCE, SFC, HEFCW, DELNI (2012b) Main Panel A Criteria (01.2012) www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2A.pdf

Staley K (2009) Exploring Impact: Public involvement in NHS, public health and social care research. INVOLVE, Eastleigh www.invo.org.uk/posttypepublication/exploring-impact-public-involvement-in-nhs-public-health-and-social-care-research/

Contact: Ann-Louise Caress, PhD, BNurs, RGN, RHV, Professor of Nursing, School of Nursing, Midwifery and Social Work, The University of Manchester and University Hospital of South Manchester NHS Foundation Trust and INVOLVE member
Email: ann.caress@manchester.ac.uk