Using systems methods to elicit complex program theories
AbstractContextual interventions pose unique challenges for implementation studies and evaluation research. By contextual, we mean programs that are tightly integrated with the social systems they are seeking to influence or transform. This integration challenges foundational assumptions of intervention science and necessitates the development of innovative approaches to monitoring and evaluation. The W3 ( ‘What Works and Why’) project piloted the use of complex systems theory and methods (CSTM) to develop program theories for peer and community-led (PCL) programs in HIV and viral hepatitis prevention and health promot...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Daniel Reeders,
Graham Brown Tags: ORIGINAL ARTICLE Source Type: research
Introducing systems ‐ and complexity‐informed evaluation
AbstractFor the last several decades and recently amidst the COVID-19 pandemic, many in the global evaluation communities call for shifts from linear, reductionist ways of thinking and working to approaches that embrace systems and complexity. In this introductory chapter, we orient readers to key systems and complexity traditions and terms and how these have been put to use in the evaluation field. Doing so provides a foundation from which to engage with the subsequent chapters. We close this chapter with highlights from the case examples featured in this issue. (Source: New Directions for Evaluation)
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Emily F. Gates,
Mat Walton,
Pablo Vidueira,
Miles McNall Tags: ORIGINAL ARTICLE Source Type: research
“Leveling up” for change: Application of a multilevel system capacity change model to evaluate U.S. Department of Defense global health engagement activities
This article will illustrate two case examples where a boundary-driven systems framework was utilized to integrate systems thinking into U.S. DoD capacity building programs and associated evaluations. The findings from the first case led to developing a theory of change that was later tested and refined in the second case to establish the multilevel system (MLS) concept model. Based on these findings, the four distinct system boundaries and subcomponents of the MLS concept model were refined to include changes within the organizational system. The development of the MLS model allowed for the explicit framing of efforts, me...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Eva Reed,
James Harwood,
Jessica L. A. Jackson,
Tsega Gebreyesus Tags: ORIGINAL ARTICLE Source Type: research
Enhancing cross ‐cultural evaluation practice through kaupapa Māori evaluation and boundary critique: Insights from Aotearoa New Zealand
AbstractIn Aotearoa New Zealand, concern about the impact of colonisation and experience of institutional racism has led to calls for evaluative practice to be firmly grounded in a M āori worldview to reflect indigenous values and avoid deficit framings. With this in mind, our evaluation projects have been informed by a blend of kaupapa Māori evaluation and boundary critique to ensure that our systemic inquiries were responsive to hapū aspirations. We focus on the role that b oundary critique played in supporting our cross-cultural evaluation practice. Applying boundary critique enabled the expansion of the evaluand to ...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Maria Hepi,
Jeff Foote,
Annabel Ahuriri ‐Driscoll,
Marara Rogers‐Koroheke,
Hone Taimona,
Andrea Clark Tags: ORIGINAL ARTICLE Source Type: research
Applying a systems and complexity lens to building evaluation capacity: Learning from a multicountry donor's experience
AbstractAlthough the evaluation field has increasingly integrated systems thinking and complexity science (STCS) into its work, little has been published on how capacity is built to do STCS-informed evaluation. The chapter helps address this gap by exploring paired concepts drawn from the STCS and evaluation capacity building (ECB) literature: adaptive action and culture; boundaries and systems and structures; interrelationships and leadership; and perspectives and communication. The ideas are grounded with illustrative examples from a reflective case narrative of an international donor organization and the system surround...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: KaYing Vang,
Marah Moore,
Claire Nicklin Tags: ORIGINAL ARTICLE Source Type: research
System change evaluation: Insights from The Rippel Foundation and its ReThink Health initiative
AbstractThis chapter explores evaluation about and for system change within a case study of The Rippel Foundation's ReThink Health initiative. Their work involves field building to understand and shift what constrains current systems influencing health and well-being and place-based efforts to change systems carried out with regionally focused stewards. Guiding questions addressed how ReThink Health envisions, supports, and evaluates system change. Data included document review and group and individual interviews with staff members followed by iterative cycles of descriptive pattern coding and analysis. Seven, interrelated...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Emily F. Gates,
Francisca Fils ‐Aime Tags: ORIGINAL ARTICLE Source Type: research
Supporting systems transformation through design ‐driven evaluation
This article introduces the fundamental tenets of a design-driven evaluation (DDE) approach and illustrates its use in developing frameworks for evaluation that can support innovation and program development. Drawing on the science of systems and their application to evaluation, Developmental and Principles-focused Evaluation, and systems-oriented design theory and practice, a model for innovation development is proposed that will reframe program evaluation as both a service and product to aid system change. (Source: New Directions for Evaluation)
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Cameron D. Norman Tags: ORIGINAL ARTICLE Source Type: research
Leadership ‐evaluation partnership: Infusing systems principles and complexity concepts for a transformational alliance
AbstractThe Global Alliance for the Future of Food provides a case exemplar of incorporating systems thinking, complexity understandings, and developmental evaluation into systems transformation work. This transformational engagement is not just rhetoric but is deeply embedded as the focus of the Global Alliance's ongoing work. It is, importantly, predicated on in-depth and long-term cocreation and collaboration between alliance leadership and an independent external evaluator. This chapter will tell the story of how the Global Alliance incorporated systems thinking, complexity understandings, and principles-focused develo...
Source: New Directions for Evaluation - September 1, 2021 Category: Universities & Medical Training Authors: Ruth A. Richardson,
Michael Q. Patton Tags: ORIGINAL ARTICLE Source Type: research
A view from the outside: Reflections on Relational Systems Evaluation
AbstractThis paper provides a review of the current volume and RSE. The author is a senior program officer at the John Templeton Foundation, which has funded some of the work described in this volume. He brings a unique perspective as he has both observed RSE in practice and experienced it first hand as a participant in strategic planning processes that use RSE. In addition, several of the grantees in his portfolio have implemented RSE as part of their projects. This paper offers reflections on these experiences as well as on the information presented in the volume and provides a critical review of RSE, including its stren...
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: Richard Bollinger Tags: ORIGINAL ARTICLE Source Type: research
Issue Information
(Source: New Directions for Evaluation)
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Tags: ISSUE INFORMATION Source Type: research
Editors ’ notes
(Source: New Directions for Evaluation)
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: Jennifer Brown Urban,
Thomas Archibald,
Monica Hargraves,
Jane Buckley,
Claire Hebbard,
Miriam R. Linver,
William M. Trochim Tags: EDITORIAL Source Type: research
Theoretical foundations and philosophical orientation of Relational Systems Evaluation
AbstractEvolutionary Evaluation, Relational Developmental Systems Theory, and Systems Thinking are three broad foundational theoretical perspectives that characterize the shifting paradigm in contemporary evaluation. They are central to the development of Relational Systems Evaluation (RSE). From Evolutionary Evaluation, we focus on the variation and selection role of evaluation, as well as the notions of ontogeny, phylogeny, symbiosis, and co ‐evolution. From Relational Developmental Systems Theory, we highlight how mutually beneficial bidirectional relations between a program and its environmental context can promote a...
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: William M. Trochim,
Jennifer Brown Urban Tags: ORIGINAL ARTICLE Source Type: research
Applying the Systems Evaluation Protocol in the real world: Six case studies
AbstractIn this paper, we present six case studies that illustrate the application of the Systems Evaluation Protocol (SEP) in different real ‐world implementation conditions. The SEP is a step‐by‐step guide for how to implement Relational Systems Evaluation (RSE), accounting for the complex factors inherent in the larger systems within which a given program is embedded. We discuss the specific SEP steps used and products developed in each case study project, including an emphasis on how decisions were made to include particular SEP elements in different contexts. The six cases differ in terms of the scale of the pro...
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: Jennifer Brown Urban,
Miriam R. Linver,
Lisa M. Chauveron,
Thomas Archibald,
Monica Hargraves,
Jane Buckley Tags: ORIGINAL ARTICLE Source Type: research
The relational nature of Evaluation Capacity Building: Lessons from facilitated evaluation partnerships
AbstractThis paper explores the nature and importance of the relationship between Evaluation Capacity Builders (ECBers) and their program partner(s) in the context of the Partnerships for Advancing Character program Evaluation (PACE) Project. PACE, a 3 ‐year initiative supported by the John Templeton Foundation, offered a unique opportunity to initiate, structure, support, and observe evaluation partnerships between an evaluation professional and a program team over 15 months. In this paper, after reviewing pertinent Evaluation Capacity Building (ECB) and adult education literature, we report on our qualitative inquiry i...
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: Jane Buckley,
Monica Hargraves,
Leslie Moorman Tags: ORIGINAL ARTICLE Source Type: research
Introduction to Relational Systems Evaluation
AbstractIn this paper, we introduce Relational Systems Evaluation (RSE), the focal topic of this volume. RSE is a framework for program planning and evaluation that is theoretically grounded, empirically tested and focused on building evaluation capacity. Theoretically, RSE is rooted in an evolutionary approach to program development and evaluation informed by systems thinking. The application of RSE involves a collaborative partnership approach pairing evaluators or researchers with program practitioners, the use of collaborative modeling and planning tools, and an emphasis on nurturing an Evaluative Thinking mindset. The...
Source: New Directions for Evaluation - April 26, 2021 Category: Universities & Medical Training Authors: Jennifer Brown Urban,
Thomas Archibald,
Monica Hargraves,
Jane Buckley,
Claire Hebbard,
Miriam R. Linver,
William M. Trochim Tags: ORIGINAL ARTICLE Source Type: research
