Ethics and Evaluation

Abma, T. A., Visse, M., Hanberger, A., Simons, H., & Greene, J. C. (2020). Enriching evaluation practice through care ethics. Evaluation, 26(2), 131–146. https://doi.org/10.1177/1356389019893402

Recently, several authors have called for a critical assessment of the normative dimensions of evaluation practice. This article responds to this call by demonstrating how evaluation practice can be enriched through deliberate engagement with care ethics. Care ethics has a relational and practice view of morality and places caring relationships and responsibilities at the forefront of our being in the world. We will demonstrate how care ethics, in particular Joan Tronto’s moral-political theory of democratic caring, can help evaluators to reshape our way of working by placing caring and relationality at the centre of our evaluative work. Care ethics as a normative orientation for evaluation stretches beyond professional codes of conduct, and rule- or principled-based behaviour. It is part of everything we do or not do, how we interact with others, and what kinds of relationships we forge in our practice. This is illustrated with two examples: a democratic evaluation of a programme for refugee children in Sweden; and a responsive evaluation of a programme for neighbours of people with an intellectual disability in The Netherlands. Both examples show that a caring ethos offers a promising pathway to address the larger political, public issues of our times through the interrogation of un-caring practices. We conclude a caring ethos can help evaluators to strengthen a caring society that builds on people’s deeply felt need to care, to relate, and to connect within and across communities.

Bruton, S. V., & Sacco, D. F. (2018). What’s it to me? Self-interest and evaluations of financial conflicts of interest. Research Ethics, 14(4), 1–17. https://doi.org/10.1177/1747016117739940

Disclosure has become the preferred way of addressing the threat to researcher objectivity arising from financial conflicts of interest (FCOIs). This article argues that the effectiveness of disclosure at protecting science from the corrupting effects of FCOIs—particularly the kind of disclosure mandated by US federal granting agencies—is more limited than is generally acknowledged. Current NIH and NSF regulations require disclosed FCOIs to be reviewed, evaluated, and managed by officials at researchers’ home institutions. However, these reviewers are likely to have institutional and personal interests of their own that may undermine the integrity of their evaluations. This paper presents experimental findings suggesting that such interests affect third-party assessments of FCOIs. Over 200 participants gauged the ethical significance of various hypothetical yet realistic FCOIs in academic research settings. Some of them were led to believe they had a small personal interest in allowing conflicted research to proceed, whereas others’ personal outcomes were unrelated to the conflicted research. The results show that motivated reasoning influences FCOI evaluations, such that those with personal interest in conflicted research provided more lenient evaluations of researcher FCOIs. These findings imply that the capacity of federally mandated FCOI disclosure procedures to enhance bias-free science is quite restricted.

Dean-Coffey, J. (2018). What’s Race Got to Do With It? Equity and Philanthropic Evaluation Practice. American Journal of Evaluation, 39(4), 527–542. https://doi.org/10.1177/1098214018778533

An increasing number of foundations are embracing racial equity/equity as a core value, and it is influencing how they see themselves and operate. However, evaluation has for the most part remained untouched. Knowing how race/racism has influenced both, philanthropy and evaluation, deepens our understanding of how philanthropic evaluation practice may unintentionally reinforce racism. Equitable evaluation shifts the current evaluation paradigm to one that centers equity/racial equity, so that it is more aligned with the values and intentions of current day philanthropic endeavors.

Dupeyron, A. (2021). Capturing the Impacts of Archaeology for Development: Opportunities and Challenges in Evaluating the Sustainable Preservation Initiative in Peru. Progress in Development Studies, 21(2), 161–180. https://doi.org/10.1177/14649934211016700

Archaeology and heritage projects can have profound social, economic, environmental and cultural impacts on the development of communities. Yet, their impacts are rarely articulated or measured in development terms, to the detriment of their accountability, sustainability and replicability.

This article explores the potential for a more systematic evaluation of these impacts through the case study of the Sustainable Preservation Initiative (SPI) and their evaluation strategies in Peru. Informed by an evaluability assessment framework, this study highlights the practical challenges in evaluating small-scale projects in the Global South and the scope for overcoming them, appraising how SPI’s contribution to local development can be measured in practice. Development evaluation methods are measured against the practical concerns expressed by project staff and participants.

Embry, V., Macy, R. J., Moracco, K. E., Scheffey, K., Moore, A., McCort, A., & Taraskiewicz, L. (2021). From “Homegrown” to Research-Ready: Converting an Existing Practitioner-Developed Violence Prevention Intervention Into an Evaluable Intervention. Health Promotion Practice. https://doi.org/10.1177/15248399211031540

There is an increased call for research on promising prevention programs already embedded in communities (“homegrown interventions”). Unfortunately, there is limited guidance to help researchers prepare these types of interventions for rigorous evaluation. To address this need, this article presents our team’s process for revising a promising community-based sexual violence prevention intervention for rigorous research. Our extensive and iterative process of reviewing and revising the intervention was guided by evaluability assessment (EA) approaches, implementation science, and a close collaboration with our community partners. Our EA process allowed us to specify the intervention’s core components and develop a “research ready” standardized curriculum with implementation fidelity assessments. We offer four lessons learned from our process: (1) even with existing materials and an extensive history of community-based delivery, community-developed programs are not necessarily research-ready; (2) close collaboration and a trusting relationship between researchers and community partners throughout the revision process ensures the integrity of core program components are maintained and implementation in diverse community settings is feasible; (3) observations of program implementation are a crucial part of the revision process; and (4) it is important to budget adequate time and resources for such revisions.

Hine, C. (2021). Evaluating the prospects for university-based ethical governance in artificial intelligence and data-driven innovation. Research Ethics, 17(4), 464–479. https://doi.org/10.1177/17470161211022790

There has been considerable debate around the ethical issues raised by data-driven technologies such as artificial intelligence. Ethical principles for the field have focused on the need to ensure that such technologies are used for good rather than harm, that they enshrine principles of social justice and fairness, that they protect privacy, respect human autonomy and are open to scrutiny. While development of such principles is well advanced, there is as yet little consensus on the mechanisms appropriate for ethical governance in this field. This paper examines the prospects for the university ethics committee to undertake effective review of research conducted on data-driven technologies in the university context. Challenges identified include: the relatively narrow focus of university-based ethical review on the human subjects research process and lack of capacity to anticipate downstream impacts; the difficulties of accommodating the complex interplay of academic and commercial interests in the field; and the need to ensure appropriate expertise from both specialists and lay voices. Overall, the challenges identified sharpen appreciation of the need to encourage a joined-up and effective system of ethical governance that fosters an ethical culture rather than replacing ethical reflection with bureaucracy.

Schmidt-Abbey, B., Reynolds, M., & Ison, R. (2020). Towards systemic evaluation in turbulent times – Second-order practice shift. Evaluation, 26(2), 205–226. https://doi.org/10.1177/1356389020909120

The need for, and possibilities of, a second-order shift in evaluation practice are explored. Second-order evaluation practice enables an evaluator to improve practice as a skilled practitioner, acknowledging her embeddedness within an evaluand. The article explores evaluation practice as experienced by professional evaluators, using ideas from developmental evaluation coupled with systemic evaluation in the tradition of systems thinking in practice. Systemic evaluation aims to capture systemic sensibilities – the bigger picture – of complex turbulent situations of change underpinning evaluands. Attributes of second-order practice with systemic evaluation are understood as being aligned with both systemic and systematic modes of evaluation praxis. Personal experiences are provided where this juxtaposing praxis has been found wanting. By example, a systems thinking in practice framework is explored as heuristic support for making a second-order practice shift. The article concludes with a discussion of some implications for developments in professionalising evaluation practice and research.

Waddell-Henowitch, C., Gobeil, J., Tacan, F., Ford, M., Herron, R. V., Allan, J. A., Kruth, M. L., & Spence, S. (2022). A Collaborative Multi-Method Approach to Evaluating Indigenous Land-Based Learning With Men. International Journal of Qualitative Methods. https://doi.org/10.1177/16094069221082359

In Canada, a vast majority of urban Indigenous people face distinct challenges accessing and connecting to Indigenous cultural practices. Research has found that colonial policies and practices continue to disrupt and fracture traditional methods of passing down cultural teachings, including dispossession from traditional lands in which cultural practices are rooted. This disruption continues to affect the availability of educational programming by and with Indigenous people and in Indigenous languages. This research involves a multi-method approach to observe and engage with a land-based traditional drum-making program for Indigenous men in an urban center in Southwestern Manitoba. By participating, watching, and listening to the men within the workshops through unstructured observation, Sharing Circles, individual interviews, and photovoice, we aim to understand the impacts of land-based learning on Indigenous men’s well-being. The study is designed in accordance with University and Tri-Agency ethical guidelines, integrating ownership, control, access and possession (OCAP), as well as the principles of respect, relevance, reciprocity, and responsibility within all phases of the research. The research is co-created by the university researchers, community collaborators, and other relevant stakeholders.