What do we know about how the Program Evaluation Standards are used in public health?
Article Sidebar
Main Article Content
Abstract
Background: Released by the Centers for Disease Control and Prevention (CDC), Framework for Program Evaluation in Public Health prominently features the program evaluation standards (1999). The program evaluation standards (PES) include 30 statements in five domains: utility, feasibility, propriety, accuracy, and evaluation accountability. Despite decades of attention to the PES among framework users and others, how public health professionals apply these standards in their work is not well understood.
Purpose: The study sought to identify notable commonalities in how the PES are used in public health.
Setting: Application of the PES in evaluative work in public health and allied fields.
Intervention: Not applicable.
Research Design: The study included a search of subscription and nonsubscription sources to identify documents that included explicit content concerning use of standards in evaluative work in public health. Documents identified were screened using predetermined criteria to include or exclude each item in the study. Items included were reviewed and coded using codes developed before examining all documents. For each code, reviewers discussed data from all documents to identify commonalities and variations in application of standards.
Findings: The literature search returned 405 documents to be screened (179 from subscription and 226 from nonsubscription sources). Thirty-eight items were included in the study based on initial screening (11 from subscription and 27 from nonsubscription sources). The study revealed that authors discussed standards as a regular component of evaluation work, but precisely how standards were used was not always explained in detail. Also, authors did not always discuss standards statements but sometimes solely focused on general domains (e.g., feasibility or accuracy). When authors discussed specific statements, they were more descriptive in how they applied the PES (i.e., compared with articles that focused on general domains). Overall, authors placed far greater emphasis on Accuracy and Utility standards, compared with Propriety, Evaluation Accountability, or Feasibility. In many cases, authors used the PES in combination with other resources (e.g., checklists, guidelines, or other standards). Although program evaluation is crucial to public health practice, the mechanics of how professionals consider, integrate, or use evaluation standards is not fully understood.
Keywords: program evaluation; program evaluation standards; public health
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
American Evaluation Association. (2018). Guiding Principles [Brochure]. https://www.eval.org/Portals/0/Docs/AEA_289398-18_GuidingPrinciples_Brochure_2.pdf
American Public Health Association. (2019). Public health code of ethics. https://www.apha.org/-/media/files/pdf/membergroups/ethics/code_of_ethics.ashx
Anderson, S. B., Braskamp, L. A., Cohen, W. M., Evans, J. W., Gilmore, A., Marvin, K. E., Shipman, V. C., Vanecko, J. J., & Wooldridge, R. J. (1982). Evaluation Research Society standards for program evaluation. New Directions for Evaluation, 1982(15), 7–19. https://doi.org/10.1002/ev.1306 DOI: https://doi.org/10.1002/ev.1306
Beerens, R. J. (2021). Improving disaster response evaluations: Supporting advances in disaster risk management through the enhancement of response evaluation usefulness. [Doctoral dissertation, Lund University]. Research Portal.
Bodenhamer, S. A. (2016). Using participatory methods to create a logic model for Healthiest Babies Possible. [Master’s thesis, Simon Fraser University]. Summit Institutional Repository.
Canadian Institute for Health Information. (2009). The CIHI data quality framework. https://secure.cihi.ca/free_products/dq-data_quality_framework_2009_en.pdf
Castrucci, B. C. (2021). The “10 Essential Public Health Services” is the common framework needed to communicate about public health. American Journal of Public Health, 111, 598–599. https://doi.org/10.2105/AJPH.2021.306189 DOI: https://doi.org/10.2105/AJPH.2021.306189
Centers for Disease Control and Prevention (CDC). (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(RR-11), 1–40. https://www.cdc.gov/mmwr/PDF/rr/rr4811.pdf
Dunet, D. O., Sparling, P. B., Hersey, J., Williams-Piehota, P., Hill, M. D., Hanssen, C., Lawrenz, F., & Reyes, M. (2008). A new evaluation tool to obtain practice-based evidence of worksite health promotion programs. Preventing Chronic Disease, 5(4). https://www.cdc.gov/pcd/issues/2008/oct/07_0173.htm
Fields, J. (2014). A metaevaluation of evaluations of health care programs that employ the chronic care model (Publication No. 241) [Doctoral dissertation, Western Michigan University]. ScholarWorks. https://scholarworks.wmich.edu/dissertations/241
Flay, B. R., Biglan, A., Boruch, R. F., González Castro, F., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6(3), 151–75. https://doi.org/10.1007/s11121-005-5553-y DOI: https://doi.org/10.1007/s11121-005-5553-y
Glasgow, R. E., Vogt, M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health, 89(9), 1322–7. https://ajph.aphapublications.org/doi/epdf/10.2105/AJPH.89.9.1322 DOI: https://doi.org/10.2105/AJPH.89.9.1322
Gugiu, P. C. (2015). Hierarchy of evidence and appraisal of limitations (HEAL) grading system. Evaluation and Program Planning, 48, 149–59. https://doi.org/10.1016/j.evalprogplan.2014.08.003 DOI: https://doi.org/10.1016/j.evalprogplan.2014.08.003
International Association for Public Participation. (2007). IAP2 spectrum of public participation. https://www.iap2canada.ca/Resources/Documents/IAP2%20Spectrum_vertical.pdf
Joint Committee on Standards for Educational Evaluation (JCSEE). (n.d.). Program Evaluation Standards. https://evaluationstandards.org/program/
Joint Committee on Standards for Educational Evaluation. (2018). Checklist of the program evaluation standards statements. https://wmich.edu/evaluation/checklists
Kelly, L. M., & Reid, C. (2021). Baselines and monitoring: More than a means to measure the end. Evaluation Journal of Australasia, 21(1), 40–53. https://doi.org/10.1177/1035719X20977522 DOI: https://doi.org/10.1177/1035719X20977522
Organisation for Economic Co-operation and Development (OECD). (2010). Quality standards for development evaluation. https://doi.org/10.1787/9789264083905-en DOI: https://doi.org/10.1787/9789264083905-en
Public Health National Center for Innovations. (2020). The 10 essential public health services. https://phnci.org/uploads/resource-files/EPHS-English.pdf
Purnell, L. (2002). The Purnell model for cultural competence. Journal of Transcultural Nursing, 13(3), 193–196. https://doi.org/10.1177/10459602013003006 DOI: https://doi.org/10.1177/10459602013003006
Scriven, M. (2007). Key evaluation checklist. https://wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20checklist.pdf
Smith, M. Y., Russell, A., Bahri, P., Mol, P. G., Frise, S., Freeman, E., & Morrato, E. H. (2018). The RIMES statement: A checklist to assess the quality of studies evaluating risk minimization programs for medicinal products. Drug Safety, 41(4), 389–401. https://doi.org/10.1007/s40264-017-0619-x DOI: https://doi.org/10.1007/s40264-017-0619-x
Stufflebeam, D. L. (1999). Program evaluations metaevaluation checklist. https://wmich.edu/sites/default/files/attachments/u350/2014/program_metaeval_long.pdf
Tongchiw, T. (2013). The meta-evaluation of revolving funds evaluation in Thailand. [Doctoral dissertation, National Institute of Development Administration]. NIDA Wisdom Repository.
United Nations Evaluation Group. (2016). Norms and standards for evaluation. http://www.unevaluation.org/2016-Norms-and-Standards
United States Government Accountability Office. (2011, December). GAO-12-331G government auditing standards: 2011 revision. https://www.gao.gov/assets/files.gao.gov/assets/gao-12-331g.pdf
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.