Literature review in the age of AI: Common questions
Guest post by Gary Thomas, emeritus professor of Education at the University of Birmingham, UK
Join Gary Thomas, author of ‘How to Do Your Literature Review’ as he discusses questions which arose in his recent webinar, ‘What is a Literature Review in the Age of AI?’. A link to the webinar recording can be found at the end of this blog post.
Q. What does 'snowballing' mean in relation to literature reviews with AI?
A. If you find a source from Google Scholar (or another search engine or AI tool) that looks just right, you can ‘roll’ backwards or forwards in time from that key reference to gather new material – either material that the authors themselves drew on (backward snowballing) or forward in time to material where authors have cited your key reference (forward snowballing).
To backward snowball (in Google Scholar), click on the article and reach either the publisher’s abstract page of it or the full reference. The abstract page will often also give the full reference list. Now,
examine the reference list to see if there are articles or books that look to be of particular interest to you
follow these up with a separate search.
To forward snowball, click on ‘Cited by’ under the article, which will give a list of all the other works that have cited this article.
Look through these articles that have cited your key article.
Do any look especially interesting?
Have any been highly cited?
AI applications such as ResearchRabbit and Connected Papers allow a rather more sophisticated form of snowballing. To do this in ResearchRabbit click on ‘Earlier Work’, or ‘Later Work’. This is all covered in more detail in How to do your literature review, Chapter 5.
Q. When using an AI to help with a literature review, do you cite the text as normal, or include the AI in it?
A. Cite the text as normal (using APA, Chicago or whatever), but acknowledge the AI in discussing the methodology of your review. In other words, at the outset, say that you used AI machines x, y and z and specify the keywords that you used and the questions that you asked. But do not use the text that AI generates in its summaries. Rather, use AI as a tool and say how you used it. Did you use AI to brainstorm your research question, to scope the area, to make searches, to screen for quality of finds? Tell the reader. This is all covered in more detail throughout How to do your literature review, and particularly in Chapters 5 and 10.
Q. Do you have any advice for teachers when students are relying so heavily on AI? How do we counteract that and make sure they're using it right?
I think you need a two-pronged approach. First, you need to appeal to students’ understanding of the fact that human intelligence (HI) is likely to produce a more targeted review – a review which fillets out the irrelevant and focuses on the meaningful, and which adopts one of the scaffolds I outline in Chapter 6 of the book. These will give meaning and structure to a review. They will take the review beyond list-making and produce a top-quality review. An AI generated review will not get much beyond a C. Second, you need to warn students about tools such as Turnitin, Originality.ai, and Copyleaks, which ‘set a thief to catch a thief’, offering AI detection capabilities to identify AI-generated content in essays, assignments and literature reviews. Students need to be warned that copying AI-generated content will be treated as plagiarism.
Q. How do you see AI playing a role in assessing publication quality?
A. Tools such as Elicit.com do offer some kind of assessment of quality, but its developers are open about the limited satisfactoriness of this. They will certainly tell you about the methodology used, sample size, etc, but they cannot tell you about whether these were appropriate for the questions being asked in the study. With question A, a case study with one person may be far more appropriate than a study using a million people, whereas with question B a large sample may be necessary. AI tools will, until they become much, much more sophisticated, default to the norm: Does this study do what all the others do? Does it have the stereotypical structure – tick, tick, tick? It will be difficult for AI to appreciate the breakthrough using out of the ordinary methods, or anything that challenges the paradigm or the received wisdom. See Chapter 3 of the book, where I discuss criticality.
Q. How do you ensure the AI keeps within the specific themes of the study?
A. This is indeed a problem. The structure of a review should be generated by the question/s being asked, and AI tools will have difficulty developing a structure geared around those questions. I have suggested various ways of intelligently developing structure in Chapter 6 of my book. What you want to avoid is AI imposing its own structure – usually something that is list-like. The key, I think, is to use AI selectively, drawing principally on its search capabilities.
To hear more from Gary Thomas about the use of AI in literature reviews, you can watch the recording of his webinar here and read his previous guest post on this blog.
Find out more about his book ‘How To Do Your Literature Review’, and request an inspection copy.