Using Dissemination Research Approaches to Understand the Awareness, Adoption, and Use of The Program Evaluation Standards
Article Sidebar
Main Article Content
Abstract
Background: The adoption and use of effective, legally defensible, and ethically sound practices relies on the successful dissemination of evidence-based practices and professional standards. The field of program evaluation has standards, competencies, and principles, yet little is known about how these are utilized by education-focused program evaluators.
Purpose: The purpose of this study is to examine the dissemination and use of the program evaluation standards established by the Joint Committee on Standards for Educational Evaluation, relative to the dissemination and use of the American Evaluation Association’s (AEA’s) guiding principles and AEA’s evaluator competencies.
Setting: The SIGnetwork, a network of evaluators of State Personnel Development Grants (SPDGs) funded by the U.S. Department of Education, Office for Special Education Programs (OSEP).
Intervention: NA
Research Design: Descriptive research.
Data Collection and Analysis: Data collection involved administering an online survey to members designated as evaluators in the SIGnetwork directory. Descriptive statistics were used to summarize the data collected via the online survey.
Findings: Using the formative audience research approach to understanding dissemination, the results of the study support previous findings that awareness of the standards was inconsistent among a sample of AEA members. Respondents self-reported low to moderate levels of familiarity with The Program Evaluation Standards and the other two guidance documents: Guiding Principles for Evaluators and AEA Evaluator Competencies. Using the audience segmentation research approach to understanding dissemination, the results of this study indicate that participants who were AEA members were more likely than those who were not members of AEA to report being familiar with the standards and to have earned an advanced degree related to their role as an evaluator.
Keywords: Joint Committee on Standards for Educational Evaluation, American Evaluation Association, program evaluation standards
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
American Evaluation Association (AEA) (n.d.). Topical Interest Groups. https://www.eval.org/Community/Topical-Interest-Groups
American Evaluation Association (2018a). Guiding Principles for Evaluators. https://www.eval.org/About/Guiding-Principles
American Evaluation Association (2018b). AEA Evaluator Competencies. https://www.eval.org/About/Competencies-Standards/AEA-Evaluator-Competencies
Ayoo, S. (2020). Professionalization of program evaluation: A comparative mixed methods case study of Canada and the United States [Unpublished doctoral dissertation]. University of Minnesota, Minneapolis, MN.
Ayoo, S., Wilcox, Y., La Velle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA Evaluation Competencies in the broader context of professionalism. New Directions for Evaluation, 168, 13–30. DOI: https://doi.org/10.1002/ev.20440
Baer, W. C. (1986). Expertise and professional standards. Work and Occupations, 13(4), 532–552. https://doi.org/10.1177/0730888486013004005 DOI: https://doi.org/10.1177/0730888486013004005
Bunger, A. C., Hanson, R. F., Doogan, N. J., Powell, B. J., Cao, Y., & Dunn, J. (2016). Can learning collaboratives support implementation by rewiring professional networks? Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 79–92. https://doi.org/10.1007/s10488-014-0621-x DOI: https://doi.org/10.1007/s10488-014-0621-x
Christie, C. A., Quiñones, P., & Fierro, L. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 35(2), 274–290. DOI: https://doi.org/10.1177/1098214013503697
Cramer, T., Canto Porto de Moraes, J., McKenna, A., Keays Hagerman, K., & Allen, L. R. (2021). Knowledge dissemination among early childhood staff members: A promising pathway for professional learning. Journal of Early Childhood Teacher Education, 00(00), 1–14. https://doi.org/10.1080/10901027.2021.1954567 DOI: https://doi.org/10.1080/10901027.2021.1954567
Davis, R., & MacKay, K. (2014). Evaluator training content and topic valuation in university evaluation courses. American Journal of Evaluation, 35(3), 419–429. https://doi.org/10.1177/1098214013520066. DOI: https://doi.org/10.1177/1098214013520066
Dewey, J. D., Montrose, B. E., Schröter, D. C., Sullins, C. D., & Mattox, J. R. (2008). Evaluator competencies: What’s taught versus what’s sought. American Journal of Evaluation, 29(3), 268–287. https://doi.org/10.1177/1098214008321152. DOI: https://doi.org/10.1177/1098214008321152
Fournier, D. M. (1994). [Review of the book The program evaluation standards: How to assess evaluations, by The Joint Committee on Standards for Educational Evaluation]. Journal of Educational Measurement, 4(31), 363–368.
Gagliardi, A. R., Brouwers, M. C., Palda, V. A., Lemieux-Charles, L., & Grimshaw, J. M. (2011). How can we improve guideline use? A conceptual framework of implementability. Implementation Science, 6(1), 1–11. https://doi.org/10.1186/1748-5908-6-26 DOI: https://doi.org/10.1186/1748-5908-6-26
Gullickson, A. M., King, J. A., La Velle, J. M., & Clinton, J. M. (2019). The current state of evaluation education: A situation analysis and call to action. Evaluation and Program Planning, 75, 20–30. DOI: https://doi.org/10.1016/j.evalprogplan.2019.02.012
Harnar, M. (2019). Familiarity with JCSEE program evaluation standards [Presentation] Annual Meeting of the Joint Committee on Standards for Educational Evaluation, Alexandria, VA, United States.
Hoe, S. L., & McShane, S. (2010). Structural and informal knowledge acquisition and dissemination in organizational learning: An exploratory analysis. Learning Organization, 17(4), 364–386. https://doi.org/10.1108/09696471011043117 DOI: https://doi.org/10.1108/09696471011043117
House, E. R. (1990). Trends in evaluation. Educational Researcher, 19(3) 24–28. DOI: https://doi.org/10.3102/0013189X019003024
James Irvine Foundation. (2009). The strong field framework: A guide and toolkit for funders and nonprofits committed to large-scale impact: A field assessment guide. The Bridgespan Group. https://irvine-dot-org.s3.amazonaws.com/documents/64/attachments/strongfieldframework.pdf?1412656138
Joint Committee on Standards for Educational Evaluation. (1981). Standards for evaluations of educational programs, projects, and materials. McGraw-Hill.
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Sage.
King, J. A. (2003). Evaluating educational programs and projects in the USA. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook on educational evaluation (pp. 721–732). Kluwer Academic Publishers. DOI: https://doi.org/10.1007/978-94-010-0309-4_41
King, J. A., & Stevahn, L. (2020). Presenting the 2018 AEA evaluator competencies. New Directions for Evaluation, 168, 49–61. DOI: https://doi.org/10.1002/ev.20435
La Velle, J. M. (2020). Educating evaluators 1976–2017: An expanded analysis of university-based evaluation education programs. American Journal of Evaluation, 41(4), 494–509. https://doi.org/10.1177%2F1098214019860914 DOI: https://doi.org/10.1177/1098214019860914
La Velle, J. M., & Donaldson, S. I. (2015). The state of preparing evaluators. New Directions for Evaluation, 145, 39–52. DOI: https://doi.org/10.1002/ev.20110
Madaus, G. F., Scriven, M., & Stufflebeam, D. L. (Eds.). (2012). Evaluation models: Viewpoints on educational and human services evaluation (Vol. 6). Springer Science & Business Media. Kluwet-Nijhoff Publishing.
Mertens, D. M. (1994). Training evaluators: Unique skills and knowledge. New Directions for Evaluation, 62, 17–28. DOI: https://doi.org/10.1002/ev.1673
Michie, S., Pilling, S., Garety, P., Whitty, P., Eccles, M. P., Johnston, M., & Simmons, J. (2007). Difficulties implementing a mental health guideline: An exploratory investigation using psychological theory. Implementation Science, 2(1), 1–8. https://doi.org/10.1186/1748-5908-2-8 DOI: https://doi.org/10.1186/1748-5908-2-8
Moss, T., & Crewe, S. E. (2020). The Black Perspective: A framework for culturally competent health related evaluations for African Americans. New Directions for Evaluation, 166, 77–87. https://doi.org/10.1002/ev.20413 DOI: https://doi.org/10.1002/ev.20413
Nahikian, A., Sistenich, V., Audet, F., & Bruderlein, C. (2013). Mapping the Dissemination of Innovation and Practice through Humanitarian Professional Networks. Grotius International. http://www.grotius.fr/mapping-the-dissemination-of-innovation-and-practice-through-humanitarian-professional-networks/
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7 DOI: https://doi.org/10.1007/s10488-010-0319-7
Purtle, J., Marzalik, J. S., Halfond, R. W., Bufka, L. F., Teachman, B. A., & Aarons, G. A. (2020). Toward the data-driven dissemination of findings from psychological science. American Psychologist, 75(8), 1052–1066. https://doi.org/10.1037/amp0000721 DOI: https://doi.org/10.1037/amp0000721
Robertson, M., Scarbrough, H., & Swan, J. (2003). Knowledge creation in professional service firms: Institutional effects. Organization Studies, 24(6), 831–857. https://doi.org/10.1177/0170840603024006002 DOI: https://doi.org/10.1177/0170840603024006002
Rogers, A., Kelly, L. M., & McCoy, A. (2021). Using social psychology to constructively involve colleagues in internal evaluation. American Journal of Evaluation, 42(4), 541–558. https://doi.org/10.1177/1098214020959465 DOI: https://doi.org/10.1177/1098214020959465
Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change (2nd ed.). Basic Books.
Sage, M., Hitchcock, L. I., Bakk, L., Young, J., Michaeli, D., Jones, A. S., & Smyth, N. J. (2020). Professional collaboration networks as a social work research practice innovation: Preparing DSW students for knowledge dissemination roles in a digital society. Research on Social Work Practice, 31(1) 42–52. https://doi.org/10.1177/1049731520961163 DOI: https://doi.org/10.1177/1049731520961163
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Sage Publications.
Stevahn, L., Berger, D. E., Tucker, S. A., & Rodell, A. (2020). Using the 2018 AEA Evaluator Competencies for effective program evaluation practice. New Directions for Evaluation, 168, 75–97. DOI: https://doi.org/10.1002/ev.20434
Stockmann, R., & Meyer, W. (2016). The future of evaluations: Global trends, new challenges and shared perspectives. Palgrave Macmillan. DOI: https://doi.org/10.1057/9781137376374_1
Symonette, H., Miller, R. L., & Barela, E. (2020). Power, privilege, and competence: Using the 2018 AEA evaluator competencies to shape socially just evaluation practice. New Directions for Evaluation, 168, 117–132. DOI: https://doi.org/10.1002/ev.20433
Tucker, S. A., & King, J. A. (2020). Next steps for AEA and the newly created AEA professionalization and competencies working group. New Directions for Evaluation, 168, 149–162. DOI: https://doi.org/10.1002/ev.20436
Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80–91. https://doi.org/10.1016/j.tate.2007.01.004 DOI: https://doi.org/10.1016/j.tate.2007.01.004
Wenger, E. C. (2010). Communities of practice: A brief introduction. Encyclopedia of Knowledge Management, 1, 1–7. https://doi.org/10.4018/978-1-59904-931-1.ch046 DOI: https://doi.org/10.4018/978-1-59904-931-1.ch046
Worthen, B. R. (1994). Is evaluation a mature profession that warrants the preparation of evaluation professionals? New Directions for Evaluation, 62, 3–15. DOI: https://doi.org/10.1002/ev.1672
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.