The importance of critical appraisal
by Dr. Helen Aveyard, Professor Nancy Preston, Professor Morag Farquhar, co-authors of How to read and critique research (2023).
Missed the webinar “Analyzing Published Literature Across Paradigms and Disciplines” with Dr. Helen Aveyard, Professor Nancy Preston, Professor Morag Farquhar? View it here and find related resources.
Critical appraisal of research papers is a component of everyday academic life, whether as a student as part of an assignment, as a researcher as part of a literature review or as a teacher preparing a lecture. For health care professionals it is also a component of evidence-based practice – assessing the quality of studies and research papers so that we can make a judgement about whether to change clinical practice based on their findings. Yet we sometimes struggle to get started on this and to see the purpose of appraising a paper, especially one that has gone through peer review in a reputable journal. We might wonder why we can’t take a paper’s findings at face value. There are many reasons why we should give the research papers we use in our academic and clinical lives the critical reflection they require – here are some of them:
Firstly, very occasionally, papers are published that are misleading. The most obvious example of this is the (now retracted) paper raising the suggestion of a link between the MMR vaccination and autism. The possibility of a link was suggested in the paper, but the evidence was weak and the sample size incredibly small. A quick glance through the paper is sufficient to determine that the evidence presented is lacking. When students are shown the “facts” outlined in the paper, they can invariably identify the serious flaws in the research. However, the media scare that followed, and the response of some health care professionals, illustrates that scrutiny of research before drawing a conclusion may not happen as often as it should. It also illustrates how important such scrutiny is. We must look carefully at the quality of evidence that is put before us to prevent misinterpreting the “evidence” and taking the conclusions of a study at face value.
Secondly, whilst it is rare for published research papers to be so misleading, it is still important to assess the value of the paper to your academic and clinical work. The vast majority of papers represent research that has been undertaken in good faith and published after a rigorous peer review process. The purpose of critical appraisal in most situations is to ascertain the extent to which a research paper contributes to an argument we are making, or its applicability to our clinical practice. We are looking at the strengths and weaknesses of a study and its reporting (the paper) – this is not to discredit the study, but to understand the strength of evidence it provides and how it might help develop our arguments or inform our clinical practice. We need to appraise papers to ensure that evidence is used proportionately and appropriately. As a general rule, we give more weight to stronger papers and less weight to those that are weaker. For example, a small-scale study reporting cases of dementia in footballers probably provides weak evidence. Evidence derived from a very small sample is rarely generalisable and only one step up from anecdotal. It is not possible from a study like this to say that football has caused dementia: many other things might have done. Yet a much larger study, involving more participants over a longer period of follow-up, possibly with a control group, is likely to provide evidence suggesting a link that is far more conclusive. We can be more certain about, or confident in, the results. This level of confidence is reflected in the weight we would give this evidence in our writing or thinking. Critical appraisal will help us to identify the aspects of a study that make it strong or weak. We should reflect this level of certainty – and the product of our critical appraisal – in our own academic writing and in our clinical thinking.
In some instances, the strengths and weaknesses of a study are straight forward to identify. The example of the link between repeatedly heading a football and later dementia is a recent example of how a larger study provides stronger evidence than a smaller study. For a lot of research, the appraisal is more subtle. For example, you might come across a study undertaken on a population similar, but not identical to, that in whom you are most interested, and you need to decide if the results will apply to your setting. You might come across a study undertaken in a different geographical location and you need to decide if the findings will apply to your setting. The context in which a study is undertaken, and the prevailing culture, can have a big influence on the findings of a study and this needs to be considered when reading a study. For example, the attitudes of people towards end of life decisions in a country which has a history of acceptance of euthanasia are likely to be different from those in a country where euthanasia remains illegal and probably slightly taboo. Such are important considerations when reading a study and deciding on its relevance for the context in which you are studying and working.
Many people use tools – critical appraisal tools – to identify the strengths and weaknesses of research. There are many appraisal tools to choose from (for different types of studies) and these are especially popular in health care research. Such tools are by no means obligatory, but they can help novice researchers to systematically identify key aspects of the research. They can ensure you include areas to critique that you might not have thought of.
Learn more!
Our book How to read and critique research is a book full of critical appraisal. We have selected nine research papers and undertaken a critical appraisal of each. Our appraisals are not structured around specific appraisal tools but identify the main aspects of the research to critique based on our many years of critical appraisal work. The aim of each chapter is to illustrate a different research method and explore the rationale for the different steps that the researchers took to answer the research question they posed. We also wrote to the authors of the papers to see if they had anything to add to our critique and, where we received this, it is included in the text too. We hope that this book is a useful contribution to discussion about critical appraisal and provides a clear illustration about how it can be achieved.
Use the code MSPACEQ223 for a 20% discount, valid through June 2023.
More Sage Research Methods Community Posts about Reviewing Scholarly Literature
The process for researching literature on research methods is somewhat different from the process used for researching literature about the topic, problem, or questions. What should we keep in mind when selecting methods literature?