The Integration of the Program Evaluation Standards into an Evaluation Toolkit for a Transformative Model of Care for Mental Health Service Delivery
Article Sidebar
Main Article Content
Abstract
Background: Stepped Care 2.0 (SC2.0) is a transformative model of mental health service delivery. This model was created by Stepped Care Solutions (SCS), a not-for-profit consultancy that collaborates with governments, public service organizations, and other institutions that wish to redesign their mental health and addictions systems of care. The SC2.0 model is based on 10 foundational principles and 9 core components that can be flexibly adapted to an organization’s or community’s needs. The model supports groups to reorganize and deliver mental health care in an evidence-informed, person-centric way. SCS partnered with evaluators from the Centre for Health Evaluation and Outcome Sciences (CHÉOS) to create a toolkit that provides evaluation guidance. The toolkit includes a theory of change, guidance on selecting evaluation questions and designs, and an evaluation matrix including suggested process and outcome metrics, all of which can be tailored to each unique implementation of the SC2.0 model. The objective of this resource is to support organizations and communities to conduct high-quality evaluations for the purpose of continuous improvement (a core component of the model of care) and to assess the model’s impact.
Purpose: The purpose of this paper is to discuss the integration of the program evaluation standards (PES) into an evaluation toolkit for SC2.0.
Setting: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation.
Intervention: Not applicable
Research Design: Not applicable
Data Collection and Analysis: Not applicable
Findings: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation.
Keywords: program evaluation standards; evaluation; mental health
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
American Evaluation Association. (2018). The 2018 AEA Evaluator Competencies. https://www.eval.org/Portals/0/Docs/AEA%20Evaluator%20Competencies.pdf
Canadian Evaluation Society. (2018). Competencies for Canadian Evaluators. https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice_2018.pdf
Carey, S., Jaouich, A., Churchill, A., Cornish, P., Impey, D., & Kim, M. (2021). Stepped Care 2.0 revised implementation guide. Mental Health Commision of Canada.
Cornish, P. (2020). Stepped Care 2.0: A Paradigm Shift in Mental Health. Springer. DOI: https://doi.org/10.1007/978-3-030-48055-4
Davidson, E. J. (2014). Evaluative Reasoning (Methodological Briefs Impact Evaluation No. 4; p. 14). United National Children’s Fynd (UNICEF). https://www.betterevaluation.org/en/resources/guide/evaluative_reasoning
Davies, R. (2013). Planning evaluability assessments: A synthesis of the literature with recommendations. (Working Paper 40). UK Department for International Development. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf
Dhillon, L., & Vaca, S. (2018). Refining Theories of Change. Journal of MultiDisciplinary Evaluation, 14(30), 64–87. DOI: https://doi.org/10.56645/jmde.v14i30.496
Fetterman, D., Rodríguez-Campos, L., Wandersman, A., & O’Sullivan, R. G. (2014). Collaborative, Participatory, and Empowerment Evaluation: Building a Strong Conceptual Foundation for Stakeholder Involvement Approaches to Evaluation (A Response to Cousins, Whitmore, and Shulha, 2013). American Journal of Evaluation, 35(1), 144–148. https://doi.org/10.1177/1098214013509875 DOI: https://doi.org/10.1177/1098214013509875
Gullickson, A. M., & Hannum, K. M. (2019). Making values explicit in evaluation practice. Evaluation Journal of Australasia, 19(4), 162–178. https://doi.org/10.1177/1035719X19893892 DOI: https://doi.org/10.1177/1035719X19893892
Gullickson, A. M., King, J. A., LaVelle, J. M., & Clinton, J. M. (2019). The current state of evaluator education: A situation analysis and call to action. Evaluation and Program Planning, 75, 20–30. https://doi.org/10.1016/j.evalprogplan.2019.02.012 DOI: https://doi.org/10.1016/j.evalprogplan.2019.02.012
Leviton, L. C., Collins, C. B., Laird, B. L., & Kratt, P. P. (1998). Teaching Evaluation Using Evaluability Assessment. Evaluation, 4(4), 389–409. https://doi.org/10.1177/13563899822208699 DOI: https://doi.org/10.1177/13563899822208699
Mason, P., & Barnes, M. (2007). Constructing Theories of Change: Methods and Sources. Evaluation, 13(2), 151–170. https://doi.org/10.1177/1356389007075221 DOI: https://doi.org/10.1177/1356389007075221
Mills, A. J., Durepos, G., & Wiebe, E. (2010). Eurocentrism. In Encyclopedia of Case Study Research. Sage Publications. https://methods.sagepub.com/reference/encyc-of-case-study-research/n132.xml DOI: https://doi.org/10.4135/9781412957397
Rousseau, D., & Billingham, J. (2018). A Systematic Framework for Exploring Worldviews and Its Generalization as a Multi-Purpose Inquiry Framework. Systems, 6(3), 27. https://doi.org/10.3390/systems6030027 DOI: https://doi.org/10.3390/systems6030027
Stufflebeam, D. L. (1999). Program Evaluations Metaevaluation Checklist (Based on the Program Evalaution Standards). https://wmich.edu/sites/default/files/attachments/u350/2014/program_metaeval_short.pdf
Yarbrough, D., Shulha, L., Hopson, R., & Caruthers, F. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users. Sage.