Main Article Content
Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice.
Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study.
Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).
Intervention: Not applicable
Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis.
Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
American Evaluation Association (AEA). (2011). American Evaluation Association statement on cultural competence in evaluation. www.eval.org/About/Competencies-Standards/Cutural-Competence-Statement
American Evaluation Association (AEA). (2018). American Evaluation Association: Guiding principles for evaluators. www.eval.org/About/Guiding-Principles
Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Child Trends.
Annie E. Casey Foundation. (2014). Race equity and inclusion action guide. Embracing racial equity: 7 steps to advance and embed race equity and inclusion within your organization. www.aecf.org/resources/race-equity-and-inclusion-action-guide
Bamberger, M., & Segone, M. (2011). How to design and manage equity-focused evaluations. UNICEF.
Bazeley, P. (2012). Integrative analysis strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/0002764211426330
Better Evaluation Rainbow Framework. (2014). http://betterevaluation.org
Caldwell, L. D., & Bledsoe, K. L. (2019). Can social justice live in a house of structural racism? A question for the field of evaluation. American Journal of Evaluation, 40(1), 6–18. https://doi.org/10.1177/1098214018815772
Carden, F. (2013). Asking questions in the solution space: Methodological issues in evaluating equity. Evaluation and Program Planning, 36(1), 213–217. https://doi.org/10.1016/j.evalprogplan.2012.03.010
Carter, C. E., & Hossain, F. (2020). How to apply a diversity, equity, & inclusion lens to strengthen evidence building in the social sector. Project Evident. https://toolkit.projectevident.org/p/how-to-apply-a-diversity-equity-inclusion-lens-to-strengthen-evidence-building-in-the-social-sector
Center for Evaluation Innovation. (2017). Equitable evaluation project framing paper. https://www.equitableeval.org/_files/ugd/21786c_04631f8cfdd44cb4a10a4097bf85acd5.pdf
Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(RR-11).
Cerna, O., Condliffe, B., & Wilson, A. (2021). Guiding questions for supporting culturally responsive evaluation practices and an equity-based perspective. www.mdrc.org/publication/guiding-questions-supporting-culturally-responsive-evaluation-practices-and-equity-based
Change Elemental. (n.d.). Deep equity. https://changeelemental.org/offerings/deep-equity/
Chicago Beyond. (2018). Why am I always being researched? https://chicagobeyond.org/researchequity/
Christie, C. A. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New Directions for Evaluation, 97, 7–36. https://doi.org/10.1002/ev.72
Christie, C. A., & Lemire, S. T. (2019). Why evaluation theory should be used to inform evaluation policy. American Journal of Evaluation, 40(4), 490–508. https://doi.org/10.1177/1098214018824045
Coleman, S., Byrd, K., Scaccia, J., Stout, S., Schall, M., Callender, S., Anderson, J., Behrman, N., Budnik, A., Smith, D., Brown, L., Douglas, W., Bussey, R., McDermott, E., Munene, E., Mullin, F., Hatchett, L., Pohorelsky, J., VanLanen, T., Pairolero, & B., Mann, Z. (2017). Engaging community members with lived experience. SCALE 1.0 Synthesis Reports. Institute for Healthcare Improvement.
Corbin, J. M., & Strauss, A. L. (2015). Basics of qualitative research: Techniques and procedures for developing grounded theory (4th ed.). SAGE.
Coryn, C. L. S., Wilson, L. N., Westine, C. D., Hobson, K. A., Ozeki, S., Fiekowsky, E. L., Greenman, G. D., & Shroeter, D. C. (2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347.
Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). SAGE.
Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches. SAGE.
Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. Jossey-Bass.
Datta, L.-E. (2000). Seriously seeking fairness: Strategies for crafting non-partisan evaluations in a partisan world. American Journal of Evaluation, 21(1), 1–14. https://doi.org/10.1177/109821400002100101
Dean-Coffey, J. (2017). Equitable Evaluation Framework™️. Equitable Evaluation Initiative. www.equitableeval.org/framework
Donaldson, S. I., & Picciotto, R. (2016). Evaluation for an equitable society. Information Age Publishing.
Equitable Evaluation Initiative (EEI), & Grantmakers for Effective Organizations (GEO). (2021). Shifting the evaluation paradigm: The Equitable Evaluation Framework. https://www.equitableeval.org/_files/ugd/21786c_7db318fe43c342c09003046139c48724.pdf
Farrow, F., & Morrison, S. (2019). Placing equity concerns at the center of knowledge development. Center for the Study of Social Policy. https://cssp.org/wp-content/uploads/2019/05/Putting-Equity-at-the-Center-of-Knowledge-Development.pdf
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs—principles and practices. Health Services Research, 48(6, Pt. 2), 2134–2156. https://doi.org/10.1111/1475-6773.12117
Gates, E. F. (2018). Toward valuing with critical systems heuristics. American Journal of Evaluation. 39(2), 201–220. https://doi.org/10.1177/1098214017703703
Gates, E. F., Williamson, E., Madres, J., Benitez Alvarez, K., & Hall, J.N. (2021). Strengthening capacity and equity in New England evaluation (SCENE) Full Report. Boston College. https://www.researchgate.net/publication/353804145_Strengthening_Capacity_and_Equity_in_New_England_Evaluation_SCENE_Full_Report
Gates, E. F., Madres, J., Hall, J. N., & Alvarez, K. B. (2022). It takes an ecosystem: Socioecological factors influencing equity-oriented evaluation in New England, U.S., 2021. Evaluation and Program Planning, 92, 102068. https://doi.org/10.1016/j.evalprogplan.2022.102068
Giacomini, M., & Hurley, J. (2008). Issues in evaluating equity. In Health promotion evaluation practices in the Americas (pp. 285–298). Springer.
Greene, J. (2014). Values-engaged evaluation. In M. Bamberger & M. Segone (Eds.), Evaluation for equitable development results (pp. 196–206). UNICEF.
Greene, J. C., DeStefano, L., Burgon, H., & Hall, J. (2006). An educative, values-engaged approach to evaluating STEM educational programs. New Directions for Evaluation, 2006(109), 53–71. https://doi.org/10.1002/ev.178
Hall, J. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485
Hall, M. E. (2020). Blest be the tie that binds. New Directions for Evaluation, 166, 13–22.
Hawn Nelson, A., Jenkins, D., Zanti, S., Katz, M., & Berkowitz, E. (2020). A toolkit for centering racial equity throughout data integration. Actionable Intelligence for Social Policy. University of Pennsylvania. www.aisp.upenn.edu/centering-equity/
Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally responsive evaluation. In K. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical program evaluation (pp. 281–318). Jossey-Bass.
Hopson, R.K., Kirkhart, K.E., & Bledsoe, K. (2011). Decolonizing Evaluation in a Developing World: Implications and Cautions for Equity-Focused Evaluation. In M. Segone (Ed.), Evaluation for equitable development results (pp. 59-82).
House, E., & Howe, K. R. (1999). Values in evaluation and social research. SAGE.
Julnes, G. (2012). Developing policies to support valuing in the public interest. New Directions for Evaluation, 133, 109–129. https://doi.org/10.1002/ev.20012
LaFrance, J., & Nichols, R. (2009). Indigenous evaluation framework: Telling our story in our place and time. American Indian Higher Education Consortium.
Lemire, S., Nielsen, S. B., & Christie, C. A. (2018). Toward understanding the evaluation market and its industry—advancing a research agenda. New Directions for Evaluation, 160, 145–163. https://doi.org/10.1002/ev.20339
Lo, F., & Espiritu, R. (2021). Evaluation is so White: Systemic wrongs reinforced by common practices and how to start righting them. Funder & Evaluator Affinity Network.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition: A comprehensive guide. Guilford Publications.
Miles, M. B., Huberman, A. M, & Saldaña, J. (2014). Qualitative data analysis: a methods sourcebook (3rd ed.). SAGE.
Nielsen, S. B., Lemire, S., & Christie, C. A. (2018). The evaluation marketplace and its industry. New Directions for Evaluation, 160, 13–28.
Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field Methods, 22(4), 295–318. https://doi.org/10.1177/1525822X10379795
Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). SAGE.
Peterson, C. H., Peterson, N. A., & Powell, K. G. (2017). Cognitive interviewing for item development: Validity evidence based on content and response processes. Measurement and Evaluation in Counseling and Development, 50(4), 217–223. https://dx.doi.org
Petty, S., & Leach, M. (2020). Systems change & deep equity: Pathways toward sustainable impact, beyond “Eureka!,” unawareness & unwitting harm. Change Elemental. https://changeelemental.org/resources/systems-change-and-deep-equity-monograph/
Public Policy Associates, Inc. (2015). Considerations for conducting evaluation using a culturally responsive and racial equity lens. http://publicpolicy.com/wp-content
Reid, A. M., Boyce, A. S., Adetogun, A., Moller, J. R., & Avent, C. (2020). If not us, then who? Evaluators of color and social change. New Directions for Evaluation, 166, 23–36. https://doi.org/10.1002/ev.20407
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). SAGE.
Schwandt, T. A., & Gates, E. F. (2016). What can evaluation do? An agenda for evaluation in service of an equitable society. In Donaldson, S. and Picciotto, R. (Eds.), Evaluation for an Equitable Society (pp. 67-81). Charlotte, NC: Information Age Publishing, Inc.
Schwandt, T.A. & Gates, E.F. (2021). Evaluating and valuing in social research. Guilford Press.
Shanker, V. (2021). Statement from the AEA Board of Directors regarding racism and inequality in our society. American Evaluation Association. www.eval.org/Full-Article/statement-from-the-aea-board-of-directors-regarding-racism-and-inequality-in-our-society
Smagorinsky, P. (2008). The method section as conceptual epicenter in constructing social science research reports. Written Communication, 25(3), 389–411.
SPEC Associates. (2018). Survey of small sellers of evaluation services: Print version [Unpublished survey instrument].
Symonette, H., Miller, R. L., & Barela, E. (2020). Power, privilege, and competence: Using the 2018 AEA Evaluator Competencies to shape socially just evaluation practice. New Directions for Evaluation, 168, 117–132.
Teasdale, R. M. (2021). Evaluative criteria: An integrated model of domains and sources. American Journal of Evaluation, 42(3), 354–376. https://doi.org/10.1177
Thomas, V. G., Madison, A., Rockcliffe, F., DeLaine, K., & Lowe, S. M. (2018). Racism, social programming, and evaluation: Where do we go from here? American Journal of Evaluation, 39(4), 514–526. https://doi.org/10.1177/1098214018772910
Westaby, K. A., Williams, T. M., Robinson, N. N., & Connors, E. (2019). Being responsive: The first assessment of culturally responsive evaluation in Wisconsin: Findings from the 2017 survey. ¡Milwaukee Evaluation!, Inc.
Wiggins, M. E., & Sileo, A. (2020). What’s the role of equity in evaluation policy? The Forum
for Youth Investment.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users. SAGE.