Setting the stage: A selected oral history of culturally responsive evaluation
AbstractDr. Hood was the first culturally responsive evaluation oral historian. For justice, culture, and equity to be mainstreamed, he understood that these concepts must be conveyed in multiple formats, settings, and arenas. He authentically brought what had been in the minds, hearts, tongues, and work of many before him to the written word —he expanded and authenticated history. We, the volume editors, have taken a cue from Dr. Hood's astute observations about the nature of history. As such, we connect CRE's past, present, and future and set the stage for this volume by sharing a living history and sentiments as told ...
Source: New Directions for Evaluation - February 3, 2024 Category: Universities & Medical Training Authors: Ayesha S. Boyce,
Nicole R. Bowman (Lunaape/Mohican),
Melvin E. Hall,
Leah C. Neubauer Tags: ORIGINAL ARTICLE Source Type: research
Large language model applications for evaluation: Opportunities and ethical implications
AbstractLarge language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text-based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep-learning AI techniques, where language models are trained on extremely large text data from the internet and then re-used for downstream tasks with limited fine-tuning required. They offer exciting opportunities for eval...
Source: New Directions for Evaluation - November 4, 2023 Category: Universities & Medical Training Authors: Cari Beth Head,
Paul Jasper,
Matthew McConnachie,
Linda Raftree,
Grace Higdon Tags: ORIGINAL ARTICLE Source Type: research
Issue Information
(Source: New Directions for Evaluation)
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Tags: ISSUE INFORMATION Source Type: research
Editors ’ notes
(Source: New Directions for Evaluation)
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Sarah Mason,
Bianca Montrosse ‐Moorhead Tags: EDITORIAL Source Type: research
Evaluation criteria for artificial intelligence
AbstractCriteria identify and define the aspects on which what we evaluate is judged and play a central role in evaluation practice. While work on the use of AI in evaluation is burgeoning, at the time of writing, a set of criteria to consider in evaluating the use of AI in evaluation has not been proposed. As a first step in this direction, Teasdale's Criteria Domains Framework was used as the lens through which to critically read articles included in this special issue. This resulted in the identification of eight criteria domains for evaluating the use of AI in evaluation. Three of these criteria domains relate to the c...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Bianca Montrosse ‐Moorhead Tags: ORIGINAL ARTICLE Source Type: research
Artificial intelligence and validity
This article explores the interaction between artificial intelligence (AI) and validity and identifies areas where AI can help build validity arguments, and where AI might not be ready to contribute to our work in establishing validity. The validity of claims made in an evaluation is critical to the field, since it highlights the strengths and limitations of findings and can contribute to the utilization of the evaluation. Within this article, validity will be discussed within two broad categories: quantitative validity and qualitative trustworthiness. Within these categories, there are multiple types of validity, includin...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Tarek Azzam Tags: ORIGINAL ARTICLE Source Type: research
Artificial intelligence and the future of evaluation education: Possibilities and prototypes
This article explores the potential implications of AI for evaluator and evaluation education. Specifically, the article discusses key issues in evaluation education including equitable language access to evaluation education, navigating program, social science, and evaluation theory, understanding evaluation theorists and their philosophies, and case studies and simulations. The paper then considers how chatbots might address these issues, and documents efforts to prototype chatbots for three use cases in evaluation education, including a guidance counselor, teaching assistant, and mentor chatbot for young and emerging ev...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Zach Tilton,
John M. LaVelle,
Tian Ford,
Maria Montenegro Tags: ORIGINAL ARTICLE Source Type: research
Using AI to disrupt business as usual in small evaluation firms
AbstractWhile many knowledge workers may fear that the rise of artificial intelligence (AI) will threaten their jobs, this article argues that small evaluation businesses shouldembrace AI tools to increase their value in the marketplace and remain relevant. In this article, consultants from a research, evaluation, and strategy firm, Intention 2 Impact, Inc., make a case for using AI tools to disrupt business as usual in evaluation from theoretical and practical perspectives. Theoretically, AI may be another example of technology that was initially feared but is now ubiquitous in society. Using concrete examples, the author...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Nina R. Sabarre,
Blake Beckmann,
Sahiti Bhaskara,
Kathleen Doll Tags: ORIGINAL ARTICLE Source Type: research
Finding a safe zone in the highlands: Exploring evaluator competencies in the world of AI
This article is intended to be a starting point for discussions that continue throug hout the remainder of this issue. (Source: New Directions for Evaluation)
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Sarah Mason Tags: ORIGINAL ARTICLE Source Type: research
A special delivery by a fork: Where does artificial intelligence come from?
AbstractIn this article, I discuss the use of artificial intelligence (AI) in evaluation and its relevance to the evolution of the field. I begin with a background on how AI models are developed, including how machine learning makes sense of data and how the algorithms it develops go on to power AI models. I go on to explain how this foundational understanding of machine learning and natural language processing informs where AI might and might not be effectively used. A critical concern is that AI models are only as strong as the data on which they are trained, and evaluators should consider important limitations when usin...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Izzy Thornton Tags: ORIGINAL ARTICLE Source Type: research
Vision for an equitable AI world: The role of evaluation and evaluators to incite change
This article highlights contextual concerns with AI from an ecosystem perspective, placing emphasis on structural and racial/ethnic inequities, bias, and prejudice. The author issues a clarion call for the field of evaluation to act collectively to incite change by being proactive, embracing our professional responsibility and critical voice, and employing evidence-based practice. Evaluators are encouraged to exercise our social and political responsibility through courageous leadership and advocacy to attend to the values of stakeholders and advance an equitable AI world. (Source: New Directions for Evaluation)
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Aileen M. Reid Tags: ORIGINAL ARTICLE Source Type: research
Disrupting evaluation? Emerging technologies and their implications for the evaluation industry
This article surveyed different emerging technologies (ET), in particular artificial intelligence, and their burgeoning application in the evaluation industry. Evidence suggests that evaluators have been relatively slow in adopting ET in their practice. However, more recent data suggest that ET adoption is increasing. This article then analyzed if, and how, ET affect the evaluation industry and evaluation practice. The article finds that program evaluation is one of several competing forms of knowledge production informing decision-making, particularly in the government and not-for-profit sectors. Therefore, evaluation fac...
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Steffen Bohni Nielsen Tags: ORIGINAL ARTICLE Source Type: research
Hacking by the prompt: Innovative ways to utilize ChatGPT for evaluators
This article combines practical ideas with deeper reflection on evaluation. It concludes that ChatGPT can substitute for evaluators when evaluations mostly focus on paperwork and conventional approaches “by the book” (a n unfortunate trend in the sector). ChatGPT cannot substitute engagement with reality and critical thinking. Will ChatGPT then be a stimulus to rediscover the humanity and the reality we lost in processes? (Source: New Directions for Evaluation)
Source: New Directions for Evaluation - November 3, 2023 Category: Universities & Medical Training Authors: Silva Ferretti Tags: ORIGINAL ARTICLE Source Type: research
Creating collaborative spaces to practice praxis
AbstractThis chapter presents a call to action grounded in a graduate student clinical program at the Wisconsin Center for Education Research (WCER) at the University of Wisconsin-Madison. The WCER Clinical Program (wcerclinicalprogram.org) creates a reciprocal space where graduate students and community-based partners develop and practice connections between research, evaluation, practice, and policy through applied projects in the community, in real time, in real contexts, and with real impact. The Evaluation Clinic evolved from two primary needs: systematic and applied learning in Culturally Responsive and Equitable Eva...
Source: New Directions for Evaluation - May 26, 2023 Category: Universities & Medical Training Authors: Claudia Triana,
Alison Bowman,
Nicole Bowman,
Ja'Dell Davis,
Jacquie Forbes,
Regina Fuller,
Annalee Good,
Anthony Hernandez,
Tyler Hook,
Marlo Reeves,
Yasmin Rodriguez ‐Escutia,
Hazel Symonette,
Katelyn A. Westaby,
Troy Williams Tags: ORIGINAL ARTICLE Source Type: research
Issue Information
(Source: New Directions for Evaluation)
Source: New Directions for Evaluation - May 25, 2023 Category: Universities & Medical Training Tags: ISSUE INFORMATION Source Type: research
