Popular Paper Examines Ensuring Trustworthiness in Qualitative Analysis
Interviews with this year's winners of Sage's 10-year-impact award, Elo Satu and Helvi Kyngäs.
This article was first published on Social Science Space.
"Trust, but verify,” is a Russian proverb that gained prominence during the Cold War during negotiations centered on nuclear arsenals. That idea informs an influential paper by health researchers Satu Elo and Helvi Kyngäs who sought to measure the trustworthiness of qualitative data gathered in the realm of nursing science.
Their paper, “Qualitative Content Analysis: A Focus on Trustworthiness,” appeared in 2014 in the open access social science megajournal Sage Open, and in the decade since, with more than 2,800 citations, it is among the most-cited papers that appeared in Sage-published journal that year. As part of its mission to best measure the impact of academic work, Elo and Kyngäs have received a 10-Year Impact Award from Sage.
Sage, the parent of Social Science Space, launched awards to move the impact conversation beyond the short termism often associated with commonly used metrics such as Journal Impact Factor. This is an especially important change in outlook for social and behavioral science, where influence can last longer or grow for years as compared to physical sciences.
The authors noted that their work examining trustworthiness in nursing science studies was mostly exploring uncharted territory, past work had described the trustworthiness of qualitative content “using terms such as credibility, dependability, conformability, transferability, and authenticity.” After scouring that limited body of previous studies, adding in their own experiences, and then reviewing methods textbooks, they both “concluded that it is important to scrutinize the trustworthiness of every phase of the analysis process, including the preparation, organization, and reporting of results” while they also compiled a checklist for those concerned about trustworthiness of future content analysis.
Below, Elo answers questions about the paper. She holds a PhD in health sciences and began his research career in 2002 at the University of Oulu. After 16 years, she transitioned closer to nursing education by moving to a university of applied sciences and currently is a principal lecturer in health education at Oulu University of Applied Sciences, while continuing to supervise doctoral dissertations at the University of Oulu. Kyngäs is a professor in the Department of Nursing Science and Health Administration and head of department at the University of Oulu.
In your estimation, what in your research—and obviously the published paper—is it that has inspired others or that they have glommed onto?
Our article focusing on the trustworthiness of content analysis often serves as a companion to another content analysis article I co-authored with Helvi Kyngäs. It seems that both are still considered key sources when researchers are first learning about the process and different approaches to content analysis. When it comes to evaluating trustworthiness, researchers have used this article to assess the overall trustworthiness of content analysis. I believe that what has resonated most is how the article successfully clarifies the often complex concept of trustworthiness by applying general criteria used in qualitative research. This has especially helped novice researchers gain insights into how to evaluate the success of their own analysis.
The idea of being ‘trustworthy’ seems more important today – both in the social science and in society as a whole – than ever, but having criteria to assess it is often lacking. Do you have suggestions for applying the principles of content analysis you developed, especially outside the research sphere or in everyday life?
Our article is quite theoretical in nature, and applying its principles to everyday situations can be challenging. The concept of trustworthiness is also complex and sometimes difficult to grasp. However, I believe that even if one never conducts content analysis themselves, the core principles of evaluating trustworthiness are still useful — for example, when assessing the reliability of other people’s research. It also helps in understanding the potential weaknesses of qualitative research results, especially if the analysis hasn’t been carefully planned and executed from the data collection phase onward.
What, if anything, would you have done differently in the paper (or underlying research) if you were to go back in time and do it again? Are there bibliometric approaches that have arisen in the interim that would have surprised you a decade ago?
One of the starting points for our article was the desire within our research group to share our expertise and understanding with others, especially early-career researchers. Several of us were experienced researchers, and together we were able to concrete the principles of evaluating trustworthiness in both the article and the checklist. Retrospectively, I would include even more concrete examples of how to assess the trustworthiness of content analysis and how to report it in research publications. We still lack good methodological sources on deductive content analysis, and I receive the most questions about this via email. There is a clear need for a solid methodological source that describes how to conduct deductive analysis and how to evaluate its trustworthiness in more detail.
When I began writing methodological articles on content analysis, I couldn’t have dreamed of the number of citations they would receive. It shows there was a real need. I’ve also been surprised that these articles remain among the most cited methodological references.
What direct feedback—as opposed to citations—have you received in the decade since your paper appeared? And have you seen the checklist in use?
At one point, I was receiving weekly emails praising the clarity and comprehensiveness of the article. Often, people also describe their own research and ask for confirmation that they had correctly understood a particular section of the article. I still receive monthly messages related to these two articles. I’ve also had the pleasure of being invited to join some research projects based on the need for consultation and guidance in content analysis. In these cases, I’ve been asked to join research teams, which I deeply appreciate. In fact, I would be happy to do even more of this. One of the best parts of my job has always been mentoring and helping others. It also helps me stay up to date with how content analysis is being applied globally.
How have others built on what you published? (And how have you yourself built on it?)
I’ve noticed that our articles are still being shared in various researcher networks. I also give lectures and courses on content analysis at other Finnish universities. To help spread knowledge about content analysis methods, I share the articles with my own students and during expert lectures. I’ve also created short videos for students where I reference the article and recommend it as a source.
Could you name a paper (or other scholarly work) that has had the most, or at least a large, impact on you and your work?
My interest in research methods began already during my nursing studies, thanks to an inspiring teacher Sirkka-Liisa Halme. When I started my doctoral dissertation, I explored the work of Kim Hesook Suzie and discovered her book Nursing Theories: Conceptual and Philosophical Foundations(opens in a new tab). Her thinking has influenced my own theory development and deepened my engagement with content analysis. While writing methodological articles on content analysis, one of the most impactful articles for me has been the 2004 publication by Ulla Hällgren Graneheim and Berit Lundman. It clearly and concretely describes how to ensure trustworthiness in content analysis, and I still consider it one of the best sources available.
(H. Graneheim & B. Lundman, 2004. Education Today. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness(opens in a new tab).)