Equity “On the Sideline” A Mixed Methods Study of New England Evaluation Practice in 2020

Main Article Content

Emily F Gates
Eric Williamson
Joseph Madres
Kayla Benitez Alvarez
Jori N Hall

Abstract

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice.


Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study.


Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).  


Intervention: Not applicable


Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis.


Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.


 

Downloads

Download data is not yet available.

Article Details

How to Cite
Gates, E. F., Williamson, E., Madres, J., Benitez Alvarez, K., & Hall, J. N. (2023). Equity “On the Sideline”: A Mixed Methods Study of New England Evaluation Practice in 2020. Journal of MultiDisciplinary Evaluation, 18(42). https://doi.org/10.56645/jmde.v18i42.715 (Original work published December 31, 2022)
Section
Research Articles

References

American Evaluation Association (AEA). (2011). American Evaluation Association statement on cultural competence in evaluation. www.eval.org/About/Competencies-Standards/Cutural-Competence-Statement

American Evaluation Association (AEA). (2018). American Evaluation Association: Guiding principles for evaluators. www.eval.org/About/Guiding-Principles

Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Child Trends.

Annie E. Casey Foundation. (2014). Race equity and inclusion action guide. Embracing racial equity: 7 steps to advance and embed race equity and inclusion within your organization. www.aecf.org/resources/race-equity-and-inclusion-action-guide

Bamberger, M., & Segone, M. (2011). How to design and manage equity-focused evaluations. UNICEF.

Bazeley, P. (2012). Integrative analysis strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/0002764211426330

Better Evaluation Rainbow Framework. (2014). http://betterevaluation.org

Caldwell, L. D., & Bledsoe, K. L. (2019). Can social justice live in a house of structural racism? A question for the field of evaluation. American Journal of Evaluation, 40(1), 6–18. https://doi.org/10.1177/1098214018815772

Carden, F. (2013). Asking questions in the solution space: Methodological issues in evaluating equity. Evaluation and Program Planning, 36(1), 213–217. https://doi.org/10.1016/j.evalprogplan.2012.03.010

Carter, C. E., & Hossain, F. (2020). How to apply a diversity, equity, & inclusion lens to strengthen evidence building in the social sector. Project Evident. https://toolkit.projectevident.org/p/how-to-apply-a-diversity-equity-inclusion-lens-to-strengthen-evidence-building-in-the-social-sector

Center for Evaluation Innovation. (2017). Equitable evaluation project framing paper. https://www.equitableeval.org/_files/ugd/21786c_04631f8cfdd44cb4a10a4097bf85acd5.pdf

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(RR-11).

Cerna, O., Condliffe, B., & Wilson, A. (2021). Guiding questions for supporting culturally responsive evaluation practices and an equity-based perspective. www.mdrc.org/publication/guiding-questions-supporting-culturally-responsive-evaluation-practices-and-equity-based

Change Elemental. (n.d.). Deep equity. https://changeelemental.org/offerings/deep-equity/

Chicago Beyond. (2018). Why am I always being researched? https://chicagobeyond.org/researchequity/

Christie, C. A. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New Directions for Evaluation, 97, 7–36. https://doi.org/10.1002/ev.72

Christie, C. A., & Lemire, S. T. (2019). Why evaluation theory should be used to inform evaluation policy. American Journal of Evaluation, 40(4), 490–508. https://doi.org/10.1177/1098214018824045

Coleman, S., Byrd, K., Scaccia, J., Stout, S., Schall, M., Callender, S., Anderson, J., Behrman, N., Budnik, A., Smith, D., Brown, L., Douglas, W., Bussey, R., McDermott, E., Munene, E., Mullin, F., Hatchett, L., Pohorelsky, J., VanLanen, T., Pairolero, & B., Mann, Z. (2017). Engaging community members with lived experience. SCALE 1.0 Synthesis Reports. Institute for Healthcare Improvement.

Corbin, J. M., & Strauss, A. L. (2015). Basics of qualitative research: Techniques and procedures for developing grounded theory (4th ed.). SAGE.

Coryn, C. L. S., Wilson, L. N., Westine, C. D., Hobson, K. A., Ozeki, S., Fiekowsky, E. L., Greenman, G. D., & Shroeter, D. C. (2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347.

Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). SAGE.

Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches. SAGE.

Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. Jossey-Bass.

Datta, L.-E. (2000). Seriously seeking fairness: Strategies for crafting non-partisan evaluations in a partisan world. American Journal of Evaluation, 21(1), 1–14. https://doi.org/10.1177/109821400002100101

Dean-Coffey, J. (2017). Equitable Evaluation Framework™️. Equitable Evaluation Initiative. www.equitableeval.org/framework

Donaldson, S. I., & Picciotto, R. (2016). Evaluation for an equitable society. Information Age Publishing.

Equitable Evaluation Initiative (EEI), & Grantmakers for Effective Organizations (GEO). (2021). Shifting the evaluation paradigm: The Equitable Evaluation Framework. https://www.equitableeval.org/_files/ugd/21786c_7db318fe43c342c09003046139c48724.pdf

Farrow, F., & Morrison, S. (2019). Placing equity concerns at the center of knowledge development. Center for the Study of Social Policy. https://cssp.org/wp-content/uploads/2019/05/Putting-Equity-at-the-Center-of-Knowledge-Development.pdf

Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs—principles and practices. Health Services Research, 48(6, Pt. 2), 2134–2156. https://doi.org/10.1111/1475-6773.12117

Gates, E. F. (2018). Toward valuing with critical systems heuristics. American Journal of Evaluation. 39(2), 201–220. https://doi.org/10.1177/1098214017703703

Gates, E. F., Williamson, E., Madres, J., Benitez Alvarez, K., & Hall, J.N. (2021). Strengthening capacity and equity in New England evaluation (SCENE) Full Report. Boston College. https://www.researchgate.net/publication/353804145_Strengthening_Capacity_and_Equity_in_New_England_Evaluation_SCENE_Full_Report

Gates, E. F., Madres, J., Hall, J. N., & Alvarez, K. B. (2022). It takes an ecosystem: Socioecological factors influencing equity-oriented evaluation in New England, U.S., 2021. Evaluation and Program Planning, 92, 102068. https://doi.org/10.1016/j.evalprogplan.2022.102068

Giacomini, M., & Hurley, J. (2008). Issues in evaluating equity. In Health promotion evaluation practices in the Americas (pp. 285–298). Springer.

Greene, J. (2014). Values-engaged evaluation. In M. Bamberger & M. Segone (Eds.), Evaluation for equitable development results (pp. 196–206). UNICEF.

Greene, J. C., DeStefano, L., Burgon, H., & Hall, J. (2006). An educative, values-engaged approach to evaluating STEM educational programs. New Directions for Evaluation, 2006(109), 53–71. https://doi.org/10.1002/ev.178

Hall, J. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485

Hall, M. E. (2020). Blest be the tie that binds. New Directions for Evaluation, 166, 13–22.

Hawn Nelson, A., Jenkins, D., Zanti, S., Katz, M., & Berkowitz, E. (2020). A toolkit for centering racial equity throughout data integration. Actionable Intelligence for Social Policy. University of Pennsylvania. www.aisp.upenn.edu/centering-equity/

Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally responsive evaluation. In K. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical program evaluation (pp. 281–318). Jossey-Bass.

Hopson, R.K., Kirkhart, K.E., & Bledsoe, K. (2011). Decolonizing Evaluation in a Developing World: Implications and Cautions for Equity-Focused Evaluation. In M. Segone (Ed.), Evaluation for equitable development results (pp. 59-82).

House, E., & Howe, K. R. (1999). Values in evaluation and social research. SAGE.

Julnes, G. (2012). Developing policies to support valuing in the public interest. New Directions for Evaluation, 133, 109–129. https://doi.org/10.1002/ev.20012

LaFrance, J., & Nichols, R. (2009). Indigenous evaluation framework: Telling our story in our place and time. American Indian Higher Education Consortium.

Lemire, S., Nielsen, S. B., & Christie, C. A. (2018). Toward understanding the evaluation market and its industry—advancing a research agenda. New Directions for Evaluation, 160, 145–163. https://doi.org/10.1002/ev.20339

Lo, F., & Espiritu, R. (2021). Evaluation is so White: Systemic wrongs reinforced by common practices and how to start righting them. Funder & Evaluator Affinity Network.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition: A comprehensive guide. Guilford Publications.

Miles, M. B., Huberman, A. M, & Saldaña, J. (2014). Qualitative data analysis: a methods sourcebook (3rd ed.). SAGE.

Nielsen, S. B., Lemire, S., & Christie, C. A. (2018). The evaluation marketplace and its industry. New Directions for Evaluation, 160, 13–28.

Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field Methods, 22(4), 295–318. https://doi.org/10.1177/1525822X10379795

Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). SAGE.

Peterson, C. H., Peterson, N. A., & Powell, K. G. (2017). Cognitive interviewing for item development: Validity evidence based on content and response processes. Measurement and Evaluation in Counseling and Development, 50(4), 217–223. https://dx.doi.org

/10.1080/07481756.2017.1339564

Petty, S., & Leach, M. (2020). Systems change & deep equity: Pathways toward sustainable impact, beyond “Eureka!,” unawareness & unwitting harm. Change Elemental. https://changeelemental.org/resources/systems-change-and-deep-equity-monograph/

Public Policy Associates, Inc. (2015). Considerations for conducting evaluation using a culturally responsive and racial equity lens. http://publicpolicy.com/wp-content

/uploads/2017/04/PPA-Culturally-Responsive-Lens.pdf

Reid, A. M., Boyce, A. S., Adetogun, A., Moller, J. R., & Avent, C. (2020). If not us, then who? Evaluators of color and social change. New Directions for Evaluation, 166, 23–36. https://doi.org/10.1002/ev.20407

Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). SAGE.

Schwandt, T. A., & Gates, E. F. (2016). What can evaluation do? An agenda for evaluation in service of an equitable society. In Donaldson, S. and Picciotto, R. (Eds.), Evaluation for an Equitable Society (pp. 67-81). Charlotte, NC: Information Age Publishing, Inc.

Schwandt, T.A. & Gates, E.F. (2021). Evaluating and valuing in social research. Guilford Press.

Shanker, V. (2021). Statement from the AEA Board of Directors regarding racism and inequality in our society. American Evaluation Association. www.eval.org/Full-Article/statement-from-the-aea-board-of-directors-regarding-racism-and-inequality-in-our-society

Smagorinsky, P. (2008). The method section as conceptual epicenter in constructing social science research reports. Written Communication, 25(3), 389–411.

SPEC Associates. (2018). Survey of small sellers of evaluation services: Print version [Unpublished survey instrument].

Symonette, H., Miller, R. L., & Barela, E. (2020). Power, privilege, and competence: Using the 2018 AEA Evaluator Competencies to shape socially just evaluation practice. New Directions for Evaluation, 168, 117–132.

Teasdale, R. M. (2021). Evaluative criteria: An integrated model of domains and sources. American Journal of Evaluation, 42(3), 354–376. https://doi.org/10.1177

/1098214020955226

Thomas, V. G., Madison, A., Rockcliffe, F., DeLaine, K., & Lowe, S. M. (2018). Racism, social programming, and evaluation: Where do we go from here? American Journal of Evaluation, 39(4), 514–526. https://doi.org/10.1177/1098214018772910

Westaby, K. A., Williams, T. M., Robinson, N. N., & Connors, E. (2019). Being responsive: The first assessment of culturally responsive evaluation in Wisconsin: Findings from the 2017 survey. ¡Milwaukee Evaluation!, Inc.

Wiggins, M. E., & Sileo, A. (2020). What’s the role of equity in evaluation policy? The Forum

for Youth Investment.

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users. SAGE.