AI and literature review: help or hindrance?

Guest post by Gary Thomas, emeritus professor of Education at the University of Birmingham, UK

The new AI tools for conducting a literature review provide a remarkable resource both for professional researchers and students. They can rapidly scan millions of papers, provide digests and summaries in seconds, link cognate literature, and even draw maps of the literature showing how one piece relates to another.

But we – professional researchers and students – can’t rely on them too heavily. There’s a downside to the use of AI, as I try to make clear in my new book How to Do Your Literature Review. A literature review is not just a compilation of related references and sources. The best literature reviews use human intelligence (HI) to scaffold meaning around the found literature. They go beyond summarising and list-making, striving instead for storytelling, weaving linked ideas into a narrative – a narrative that contributes meaning and value to a research study.

It is this narrative-making that will give the human literature reviewer a huge superiority over the AI literature reviewer. The good human reviewer’s review will be full of ‘however’ and ‘whereas’ and ‘by contrast with’ and ‘the balance of opinion’ and ‘we might conclude tentatively that’. Even the best AI’s offer, even if it’s presented in prose like a story, won’t be much more than an expanded list.

So, there are good and not-so-good sides to the host of new AI-based reviewing tools that are now available. How should we summarise these?

Positives

• They are good at finding relevant papers and those that may be tucked away in a cobwebby corner of the literature somewhere. It’s easy to miss things when doing a search, and the AI tools will emerge with papers that you may not have come across or considered relevant.

• They are great for snowballing backwards and forwards.

• They review a range of databases (e.g. PubMed, ERIC, PsycInfo) saving you the trouble of going to each individually. Moreover, in reviewing a range of these at the same time, they may unearth material that wouldn’t have been revealed if you had relied on a subject-specific database.

Negatives

• They may miss some obvious finds and connections, perhaps because an author is using an unusual vocabulary, or hasn’t adopted a widely used keyword, or hasn’t connected explicitly with the existing literature.

• They may latch on to a word in a search query or title that isn’t centrally relevant and go over the top on offering ‘finds’ based on that word.

• They are not good at telling a story. Even when they claim to be offering some kind of prose digest, this is more like a list – albeit a list that is made to look like prose – than a narrative. They cannot see the threads, discrepancies, anomalies and themes that you would see using your human intelligence.

• Relying on them too closely can mean that the form, structure and priorities of the existing literature can be privileged over a thoughtful review of what is important for the issue that û  you are exploring. Your mindmap or storyboard should be your compass in navigating the area you are investigating, not the thought-lines imposed by existing work.

I show how to use some of the AI tools in ‘How to Do Your Literature Review’. But as I try to make clear in the book, researchers and students have the power, using their own human intelligence, to develop narratives – storyboards – in ways that AI cannot. AI can certainly help with the searching, but it can’t do the thinking. In the book, I outline a number of scaffolds for structuring a literature review – funnelled, thematic, chronological, and others – that will give structure and meaning to synthesis of the literature in ways that AI can’t.

Gary Thomas is an emeritus professor of education at the University of Birmingham. His teaching and research have focused on inclusion, special education, and research methodology in education, with a particular focus on case study. He has conducted research funded by the AHRC, the ESRC, the Nuffield Foundation, the Leverhulme Trust, the Department for Education, Barnardos, local authorities, and a range of other organisations. He has coedited the British Educational Research Journal and is currently an executive editor of Educational Review. He is author of many books, most recently ‘How to do Your Literature Review’ published by Sage.

If you want to hear more from Gary Thomas about ways to engage students in doing their literature reviews, join our webinar: 'What is a literature review in the age of AI?' on 27 March, 5pm BST.

Find out more about his book 'How To Do Your Literature Review', and request an inspection copy.