The Garden of Evaluation Approaches Visualization
Article Sidebar
Main Article Content
Abstract
Background: The Garden of Evaluation Approaches (Montrosse-Moorhead, et al., 2024a) maps evaluation approaches against eight dimensions of practice and situates them in their philosophical orientations and methodological dispositions.
Purpose: The Garden’s guiding question is: How do evaluation approaches compare in terms of dimensions that facilitate use and application? Primary intended users are evaluation practitioners, and secondary intended users include ecosystem actors (e.g., clients, program participants, funders), educators, researchers, and theoreticians.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Exploratory sequential mixed methods.
Data Collection and Analysis: Data collection and analysis included locating and reading primary sources for each evaluation approach. One team member thematically coded each source for evidence of an approach’s stance on the eight dimensions of practice, its’ philosophical orientation and methodological disposition, and steps for implementing the approach in practice. This qualitative evidence was used to generate ordinal ratings for the eight dimensions using a rating scale developed by the team, and to classify the approaches' philosophical orientation and methodological disposition. Drafts of handouts summarizing qualitative and quantitative data were reviewed by the two other team members, with dialogue and deliberation among all members used to come to consensus.
Findings: The first iteration of the Garden includes seven approaches: fourth generation evaluation, made in Africa, nation-to-nation evaluation, practical participatory evaluation, sistematización de experiencias, theory-driven evaluation, and transformative participatory evaluation. Each approach is represented as a flower within the Garden, with each petal corresponding to the approaches’ stance on one of the eight dimensions of practice. The colors of the flowers represent the underlying philosophical foundation of the approach. The pattern within the central disc of each flower represents the methodological disposition encouraged by the approach. Summaries of the empirical mixed-method analysis are made transparent in supporting handouts. Future work will grow flowers for additional approaches and continue to map out the Garden.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. (2004). Evaluation roots: Tracing theorists' views and influences. Los Angeles, CA: Sage Publications. https://doi.org/10.4135/9781412984157
Alkin, M. C., & Christie, T. C. (2023). Evaluation roots: Theory influencing practice (3rd Ed.). Los Angeles, CA: Sage Publications.
Christie, C. A., Montrosse, B. E., & Klein, B. M. (2005). Emergent design evaluation: A case study. Evaluation and Program Planning, 28, 271-277. http://dx.doi.org/10.1016/j.evalprogplan.2005.04.002 https://doi.org/10.1016/j.evalprogplan.2005.04.002
Coryn, C. L., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199-226. https://doi.org/10.1177/1098214010389321 https://doi.org/10.1177/1098214010389321
Becho, L. W. (2019). An exploratory investigation into the role of evaluation theory in evaluative thinking [Doctoral dissertation, Western Michigan University]. WMU Campus Repository. https://scholarworks.wmich.edu/dissertations/3484
Bledsoe, K. L., & Graham, J. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26(3), 302-319. https://doi.org/10.1177/1098214005278749
Boczar, D. P., Montrosse-Moorhead, B., & Davidesco, I. (Under review, Evaluation and Program Planning). Refining nascent programs through emergent design evaluation.
Dewey, J. D., Montrosse, B. E., Schröter, D. C., Sullins, C. D., & Mattox, J. R. (2008). Evaluator competencies: What's taught versus what's sought. American Journal of Evaluation, 29(3), 268-287. https://doi.org/10.1177/1098214008321152 https://doi.org/10.1177/1098214008321152
Fitzpatrick, J. L., Worthen, B. R., Sanders, J. R., & Wingate, L. A. (2023). Program evaluation: Alternative approaches and practical guidelines (5th ed.). Allyn and Bacon.
Hood, S., Hopson, R., & Kirkhart, K. (2015). Culturally responsive evaluation: Theory, practice, and future implications. In K. Newcomer, H. Hatry, & J. Wholey. (Eds.), Handbook of practical program evaluation (pp. 281-317). Jossey-Bass. https://doi.org/10.1002/9781119171386.ch12
Mark, M. M. (2018). Strengthening links between evaluation theory and practice, and more: Comments inspired by George Grob's 2017 Eleanor Chelimsky Forum Presentation. American Journal of Evaluation, 39(1), 133-139. https://doi.org/10.1177/1098214017734641 https://doi.org/10.1177/1098214017734641
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice. Guilford Publications.
Miller, R. L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27(3), 296-319. https://doi.org/10.1177/109821400602700303 https://doi.org/10.1177/109821400602700303
Montrosse-Moorhead, B. (2017, December). An international primer on evaluation theories and approaches. Training course provided at the joint conference of the Latin-American and Caribbean Monitoring, Evaluation and Systematization Network, the Monitoring and Evaluation Network of Latin America and the Caribbean, and the International Development Evaluation Association, Guanajanto, Mexico.
Montrosse-Moorhead, B., & Griffith, J. (2017). Toward the development of reporting standards for evaluations. American Journal of Evaluation, 38, 577-602. doi: 10.1177/1098214017699275 https://doi.org/10.1177/1098214017699275
Montrosse-Moorhead, B., Bellara, A. P., & Gambino, A. J. (2017). Communicating about evaluation: A conceptual model and case example. Journal of MultiDisciplinary Evaluation, 13(29), 16-30. https://doi.org/10.56645/jmde.v13i29.473 https://doi.org/10.56645/jmde.v13i29.473
Montrosse-Moorhead, B., Gambino, A., Yahn, L., Fan, M., & Vo, A. T. (2022). Evaluator education curriculum: What ought to be taught in master's and doctoral programs? American Journal of Evaluation, 43(2), 26-292. doi:10.1177/10982140211020326 https://doi.org/10.1177/10982140211020326
Montrosse-Moorhead, B., & Schröter, D. C. (2017, June). A primer on evaluation theories and approaches. Training course provided at the American Evaluation Association Summer Evaluation Institute, Atlanta, GA.
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2019, August). A primer on evaluation theories and approaches. Training course provided at the American Evaluation Association Graduate Education Diversity Initiative kick-off meeting, Washington, DC.
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2018a, October). An interactive and case-centered primer on evaluation theories and approaches. Training course provided at the bi-annual European Evaluation Society conference, Thessaloniki, Greece.
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2018b, April). A primer on evaluation theories and approaches. Training course provided at the annual American Educational Research Association, New York, NY.
Montrosse-Moorhead, B., Schröter, D., & Becho, L. (2023, March 16). The garden of evaluation approaches open science framework project. https://doi.org/10.17605/OSF.IO/MKXGC https://doi.org/10.56645/jmde.v20i48.1029
Montrosse-Moorhead, B., Schröter, D., & Becho, L. W. (2024a). The garden of evaluation approaches. American Journal of Evaluation, 0(0). https://doi.org/10.1177/10982140231216667 https://doi.org/10.1177/10982140231216667
Montrosse-Moorhead, B., Schroeter, D., & Becho, L. (2024b, February 19). The garden of evaluation approaches rating rubric. Retrieved from osf.io/24we3 https://doi.org/10.56645/jmde.v20i48.1029
Schröter, D. (2016, June). A primer on evaluation theories and approaches. Training course provided at the American Evaluation Association Summer Evaluation Institute, Atlanta, GA.
Schröter, D. C. (2022, November). "Isms" in teaching evaluation theory through practice. Paper presented at the annual meeting of the American Evaluation Association, New Orleans, LA.
Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2019a, March). Elevating your evaluation practice by applying theories and approaches. Training course provided at the bi-annual African Evaluation Association conference, Abidjan, Cote d'Ivoire.
Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2019b, April). Evaluation theories and approaches: An interactive and case-centered primer. Training course provided at the annual American Educational Research Association, Toronto, Canada.
Schröter, D., Becho, L. W., & Montrosse-Moorhead, B. (2019c, May). Applications in evaluation practice: Evaluation Theories and Approaches. Daylong peer-reviewed workshop presented at the United Nations Evaluation Group, Nairobi, Kenya.
Schröter, D., Montrosse-Moorhead, B., & Becho, L. W. (2023). Locating realist evaluation within the garden of evaluation approaches. Presentation at the 2023 conference of the International Public Policy Association (IPPA), Toronto, June 27-29, 2023.
Schröter, D., & Watts, B. (2017, November). Great evidence, lousy findings: Why methods-based evaluation approaches might hinder effective program development. Poster presented at the 2017 annual conference of the Association for Public Policy Analysis and Management.
Shadish, W. R. (1998). Evaluation theory is who we are. American Journal of Evaluation, 19(1), 1-19. https://doi.org/10.1016/S1098-2140(99)80177-5 https://doi.org/10.1016/S1098-2140(99)80177-5
Shanker, V. (2023, June). Glad to see that the ecosystem idea that I first spoke about publicly in 2017 to students in Jean King's [Thumbnail with link attached] [Post]. LinkedIn. https://www.linkedin.com/posts/vidhyashanker_from-evaluation-theory-tree-to-theory-forest-activity-7074927592692170752-_u0j/?utm_source=share&utm_medium
Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, and applications (2nd Ed.). John Wiley & Sons.
Thomas, V. G., & Campbell, P. B. (2020). Evaluation in today's world: Respecting diversity, improving quality, and promoting usability. SAGE Publications, Incorporated.