Training in critical appraisal skills: Making sense of the evidence in dementia – the ALOIS community

Summary
This one-day skills workshop was for carers of people with dementia. It was linked to a specific engagement project managed by the Cochrane Dementia Group, which recruits and trains lay volunteers, specifically targeting carers and former carers of people with dementia, to update and maintain an online register of controlled dementia trials. This trials register is called ALOIS after Alois Alzheimer. The ALOIS community volunteers read reports of dementia research and extract key pieces of information to enter into the trials register, a task referred to as ‘coding’.

What was the aim of the training?

The workshop had dual aims:

1. to enable carers to understand more about research and specifically how to tell good research from bad
2. to recruit volunteers to the ALOIS coding task, as described above.

People could take part in the workshop regardless of their intention to volunteer for the coding task. Also, taking part in the training was not a requirement for volunteering – many of the volunteers have access to this opportunity through other avenues.

Who was the target audience?

Although the target audience were carers and former carers of people with dementia, the workshop was open to anyone with an interest in dementia research. All members of the Cochrane Dementia and Cognitive Improvement Review Group (the team which set up the trials register) took part in the workshop, learning alongside the carers.

What did the training involve?

The workshop used a step-by-step approach to build up participants’ confidence in critically appraising research articles. The trainers approached the workshop on the basis that people already have critical skills and the workshop was to help them learn how to apply these skills to research reports. The workshop activities are outlined below:

Step 1

Using a spoof example, participants were asked to think about their views of research and the core principles required when reporting research. The example was a (pretend) new wonder drug for hangovers called ‘LOMA – Less of the Morning After’. The aim was to make the first session accessible and fun.

Step 2

Using examples of real research which had been reported in the national press, the session continued discussion about how to critique research and focused on how these real examples stood up to further scrutiny. Participants were asked to think about what else they would want to know about the research if they read about it in the newspaper, such as what was missing in the report that would enable them to decide if it was good quality or would be useful to them.

Step 3

Using the standard Critical Appraisal Skills Programme (CASP) 11 questions for randomised controlled trials, participants worked in small groups to critique a published research paper, which had been referred to in Step 2.

Other workshop content included:

• thinking about why good quality research is important to carers of people with dementia
• a ‘map’ of a research article – getting orientated to research articles and the different elements you expect to see
• practical information about how the coding task would be carried out for the ALOIS community.

What were the outcomes?

About half of the carers taking part in the workshop decided to volunteer for the ALOIS project – reading trial reports and extracting information such as number of participants, type of intervention and diagnostic criteria. Their contribution is improving the ability of the Cochrane review team to undertake the systematic review in a timely way as the trial reports are entered into the database more quickly.

As people become more confident with the task, they are encouraged to undertake other tasks for the Cochrane Dementia Group, such as acting as consumer referees of the Group’s reviews and protocols, which is an integral part of the Cochrane peer review process.

Who developed the training?

Sally Crowe was the trainer commissioned to deliver this workshop. She has many years’ experience of training members of the public in critical appraisal skills and has developed materials with Amanda Burls at CASP. The content of this workshop was influenced by feedback received from participants in previous workshops.

How do you support carers after the training?

Those carers who volunteered for the coding task are supported by the ALOIS study manager, Anna Noel-Storr. A comprehensive ALOIS coding manual gives step-by-step guidance for new volunteers and they submit their coding to Anna, who gives detailed and constructive feedback via email on how they’ve done. She also makes any corrections needed to the record and then publishes it on ALOIS. The number of corrections needed falls dramatically as volunteers get familiar with the task. For many volunteers Anna no longer checks their records in detail before publishing them.

Online e-learning materials are being developed both to encourage people to volunteer and to support volunteers. These materials include an online coding tutorial using a real dementia research paper as a worked example. There are more details about the e-learning materials below.

Was the training evaluated?

Participants were contacted to give feedback about their experience of the workshop. They enjoyed the day and of those who have gone onto code research reports for ALOIS, enjoy this task also. Some sample quotes from participants include:

“Rewarding”…”interesting”…”informative”…”challenging”…”educational”

“…a great way to become and stay more informed about the current research that is being done in the area of dementia and cognitive enhancement”

“I feel that I am making a contribution to a very important cause and a very worthwhile project”

Learning points

• When training members of the public in critical appraisal skills, the choice of topic for the practice session is important.
• Sometimes it can be helpful to use research examples that are different from the participants’ specific area of interest – this enables people to get really engaged with the critical appraisal process, rather then get distracted with the topic of the paper.

Contact for more information:

Sally Crowe, Director, Crowe Associates

Dr Amanda Burls, Director of Postgraduate Programmes in Evidence-Based Health Care, Department of Primary Care Health Sciences, University of Oxford
Email: amanda.burls@phc.ox.ac.uk

Caroline Struthers, Project Manager, ALOIS Community Volunteer Project
Email: caroline.struthers@ndm.ox.ac.uk

Links to useful reports/documents

See further information about the e-learning materials that are being developed.

May 2012