We Can’t Hear You – You’re on Mute: Findings From a Review of Evaluation Capacity Building (ECB) Practice Online

Main Article Content

Ann Marie Castleman
Minji Cho
Isabelle Bourgeois
Leslie Fierro
Sebastian Lemire


Background: In her presidential address to the American Evaluation Association (AEA) in 2007, Hallie Preskill (2008) highlighted the potential role of technology to promote learning from evaluation, noting the increased use of computers, the internet, and social media as untapped ways to facilitate evaluation. More than ten years later in the context of the COVID-19 pandemic, evaluators and evaluation capacity building (ECB) practitioners found themselves needing to shift to online modalities to conduct evaluation and build capacity. The COVID-19 pandemic, technological advancements, and the rapid shift to remote work have changed our way of working (Gratton, 2021; Kane et al., 2021). Building evaluation capacity is no exception to this trend.

Purpose: This study aimed to examine ways that practitioners have built evaluation capacity online or have used technology to do so, to capture lessons learned that can be applied in a COVID and post-normal context.

Setting: Findings from this study can be applied in online contexts for developing evaluation capacity.

Intervention: Not applicable.

Research Design: The study design consisted of a rapid review of the ECB literature published from 2000 to 2019 in eight academic journals focused on evaluation research and practice.

Data Collection and Analysis: Twenty-nine case applications of ECB practice that: 1) mentioned use of technology as a strategy for building evaluation capacity or 2) noted that at least one component of the ECB intervention was carried out online or virtually were reviewed for this study. Quantitative data were analyzed via descriptive statistics. Qualitative data were coded in MAXQDA using conventional content analysis (Hsieh & Shannon, 2005).

Findings: More diverse online interventions have increased over time. Less than half (45%) of ECB interventions made use of both asynchronous and synchronous strategies for building capacity while more than one-third (38%) made use of asynchronous only strategies. Key barriers to implementing ECB strategies online included lack of social connections to other participants during the capacity building activity, technical malfunctions, lack of access to or familiarity with the technology in use, and limited resources for carrying out evaluation activities. Key facilitators for enhancing implementation included facilitating participant interaction and relationship-building both on and off-line, tailoring ECB activities to participant work contexts, and providing tutorials for accessing and using the technology in play.


Download data is not yet available.

Article Details

How to Cite
Castleman, A. M., Cho, M., Bourgeois, I., Fierro, L., & Lemire, S. (2023). We Can’t Hear You – You’re on Mute: Findings From a Review of Evaluation Capacity Building (ECB) Practice Online. Journal of MultiDisciplinary Evaluation, 19(45), 34–49. https://doi.org/10.56645/jmde.v19i45.739


Anderson, C., Chase, M., Johnson, J., Mekiana, D., McIntyre, D., Ruerup, A., & Kerr, S. (2012). It is only new because it has been missing for so long: Indigenous evaluation capacity building. American Journal of Evaluation, 33(4), 566–582. https://doi.org/10.1177/1098214012449686

Beere, D. (2005). Evaluation capacity-building: A tale of value-adding. Evaluation Journal of Australasia, 5(2), 41–47. https://doi.org/10.1177/1035719X0500500207

Bennington, T. L., Gay, G., & Jones, M. L. W. (1999) Using multimedia records to support mixed-method evaluation. New Directions for Evaluation, 84, 59–72. https://doi.org/10.1002/ev.1153

Bourgeois, I., Lemire, S. T., Fierro, L. A., Castleman, A. M., & Cho, M. (2023). Laying a solid foundation for the next generation of evaluation capacity building: Findings from an integrative review. American Journal of Evaluation, 44(1), 29–49. https://doi.org/10.1177/10982140221106991

Brandon, P. R., & Higa, T. A. F. (2004). An empirical study of building the evaluation capacity of K-12 site-managed project personnel. Canadian Journal of Program Evaluation, 19(1), 125–142.

Campbell, R., Townsend, S. M., Shaw, J., Karim, N., & Markowitz, J. (2015). Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use. Evaluation and Program Planning, 52, 107–117. https://doi.org/10.1016/j.evalprogplan.2015.04.005

Cohen, C. (2006). Evaluation learning circles: A sole proprietor’s evaluation capacity-building strategy. New Directions for Evaluation, 111, 85–93. https://doi.org/10.1002/ev.200

Compton, D. W., MacDonald, G., Schooley, M., Zhang, L., & Baizerman, M. (2008). Using evaluation capacity building (ECB) to interpret evaluation strategy and practice in the United States National Tobacco Control Program (NTCP): A preliminary study. Canadian Journal of Program Evaluation, 23(3 Spec. Issue), 199–224.

Dette, R., Steets, J., & Sagmeister, E. (2016). Technologies for monitoring in insecure environments. Secure Access in Volatile Environments (SAVE). https://www.gppi.net/media/SAVE__2016__Toolkit_on_Technologies_for_Monitoring_in_Insecure_Environments.pdf

Duncan, D. F., White, J. B., & Nicholson, T. (2003). Using internet-based surveys to reach hidden populations: Case of nonabusive illicit drug users. American Journal of Health Behavior, 27(3), 208–218. https://doi.org/10.5993/AJHB.27.3.2

Fayard, A.-L., Weeks, J., & Khan, M. (2021). Designing the hybrid office. Harvard Business Review, 99(2), 114–114.

Fetterman, D. M. (2002). Web surveys to digital movies: Technological tools of the trade. Educational Researcher, 31(6), 29–37.

Fleming, M. L., & Easton, J. (2010). Building environmental educators’ evaluation capacity through distance education. Evaluation and Program Planning, 33(2), 172–177. https://doi.org/10.1016/j.evalprogplan.2009.07.007

Galen, M., & Grodzicki, D. (2011). Utilizing emerging technology in program evaluation. New Directions for Evaluation, 131, 123–128.

Gay, G., & Bennington, T. L. (Eds.). (1999). Information technologies in evaluation: Social, moral, epistemological, and practical implications [Special issue]. New Directions for Evaluation, 84.

Gibson, R., & Robichaud, S. (2020). Evaluating Dancing with Parkinson’s: Reflections from the perspective of a community organization. Evaluation and Program Planning, 80. https://doi.org/10.1016/j.evalprogplan.2017.05.010

Goodyear, L. K. (2011). Building a community of evaluation practice within a multisite program. New Directions for Evaluation, 129, 97–105. https://doi.org/10.1002/ev.358

Gratton, L. (2021). How to do hybrid right. Harvard Business Review, 99(3), 66–66.

Hilton, L., & Libretto, S. (2017). Evaluation capacity building in the context of military psychological health: Utilizing Preskill and Boyle’s multidisciplinary model. American Journal of Evaluation, 38(3), 393–404. https://doi.org/10.1177/1098214016664584

Hong, L. (2008). Blending online components into traditional instruction in pre-service teacher education: The good, the bad, and the ugly. International Journal for the Scholarship of Teaching and Learning, 2(1). https://doi.org/10.20429/ijsotl.2008.020114

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–88. https://doi.org/10.1177/1049732305276687

Jamieson, V., & Azzam, T. (2012). The use of technology in evaluation practice. Journal of MultiDisciplinary Evaluation, 8(18), 1–15. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/340

Kane, G. C., Nanda, R., Phillips, A., & Copulsky, J. (2021). Redesigning the post-pandemic workplace. MIT Sloan Management Review, 62(3), 12–13.

Kentnor, H. E. (2015). Distance education and the evolution of online learning in the United States. Curriculum and Teaching Dialogue, 17(1/2), 21–34. https://digitalcommons.du.edu/law_facpub/24/

Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: The evolution of a rapid review approach. Systematic Reviews, 1(1), 1–9. https://doi.org/10.1186/2046-4053-1-10

Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338. https://doi.org/10.1177/1098214011434608

Lachance, L., Watson, C., Blais, D., Ungar, M., Healey, G., Salaffie, M., Sundar, P., Kelly, L., & Lagace, M. C. (2019). Strengthening child and youth programs: A look at inter-organizational mentoring strategies. Evaluation and Program Planning, 76. https://doi.org/10.1016/j.evalprogplan.2019.101679

Ma, L., & Lee, C. S. (2021). Evaluating the effectiveness of blended learning using the arcs model. Journal of Computer Assisted Learning, 37(5), 1397–1408. https://doi.org/10.1111/jcal.12579

Mackay, K. (2002). The World Bank’s ECB experience. New Directions for Evaluation, 93, 81–100. https://doi.org/10.1002/ev.43

Mulvey, K. P., Atkinson, D. D., Avula, D., & Luckey, J. W. (2005). Using the internet to measure program performance. American Journal of Evaluation, 26(4), 587–597. https://doi.org/10.1177/1098214005281320

Mayberry, R. M., Daniels, P., Yancey, E. M., Akintobi, T. H., Berry, J., Clark, N., & Dawaghreh, A. (2009). Enhancing community-based organizations’ capacity for HIV/AIDS education and prevention. Evaluation and Program Planning, 32(3), 213–220. https://doi.org/10.1016/j.evalprogplan.2009.01.002

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. https://eric.ed.gov/?id=ED505824

Naccarella, L., Pirkis, J., Kohn, F., Morley, B., Burgess, P., & Blashki, G. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30(3), 231–236. https://doi.org/10.1016/j.evalprogplan.2007.05.001

Nelson, M., & Eddy, R. (2008). Evaluative thinking and action in the classroom. New Directions for Evaluation, 117, 37–46. https://doi.org/10.1002/ev.250

Palvia, S., Aeron, P., Gupta, P., Mahapatra, D., Parida, R., Rosner, R., & Sindhi, S. (2018). Online education: Worldwide status, challenges, trends and implications. Journal of Global Information Technology Management, 21(4), 233–241. https://doi.org/10.1080/1097198X.2018.1542262

Preskill, H. (2008). Evaluation’s second act: A spotlight on learning. American Journal of Evaluation, 29(2), 127–138. https://doi.org/10.1177/1098214008316896

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182

Rolfe, A., & Cheek, B. (2012). Learning styles. InnovAiT, 5(3), 176–181. https://doi.org/10.1093/innovait/inr239

Rorrer, A. S. (2016). An evaluation capacity building toolkit for principal investigators of undergraduate research experiences: A demonstration of transforming theory into practice. Evaluation and Program Planning, 55, 103–111. https://doi.org/10.1016/j.evalprogplan.2015.12.006

Rosenstein, B., & Englert, P. E. (2008). The road to evaluation capacity building: A case study from Israel. Canadian Journal of Program Evaluation, 23(3), 83–102.

Satterlund, T. D., Treiber, J., Kipke, R., Kwon, N., & Cassady, D. (2013). Accommodating diverse clients’ needs in evaluation capacity building: A case study of the tobacco control evaluation center. Evaluation and Program Planning, 36(1), 49–55. https://doi.org/10.1016/j.evalprogplan.2012.05.004

Scharbatke-Church, C. & Patel, A. G. (2016). Technology for evaluation in fragile and conflict affected states: An introduction for the digital immigrant evaluator. The Fletcher School, Tufts University and Besa. https://sites.tufts.edu/ihs/files/2018/02/Technology-and-Evaluation-Hitachi-Paper.pdf

Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 93, 7–26. https://doi.org/10.1002/ev.39

Sundar, P., Kasprzak, S., Halsall, T., & Woltman, H. (2010). Using web-based technologies to increase evaluation capacity in organizations providing child and youth mental health services. Canadian Journal of Program Evaluation, 25(1), 91–112.

Tang, H., Cowling, D. W., Koumjian, K., Roesler, A., Lloyd, J., & Rogers, T. (2002). Building local program evaluation capacity toward a comprehensive evaluation. New Directions for Evaluation, 95, 39–56. https://doi.org/10.1002/ev.57

Taut, S. (2007). Studying self-evaluation capacity building in a large international development organization. American Journal of Evaluation, 28(1), 45–59. https://doi.org/10.1177/1098214006296430

Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015). A scoping review of rapid review methods. BMC Medicine, 13, 224. https://doi.org/10.1186/s12916-015-0465-6

Watt, J. H. (1999). Internet systems for evaluation research. New Directions for Evaluation, 84, 23–43. https://doi.org/10.1002/ev.1151

Zhao, K., Sridharan, S., Ingabire, M.-G., Yu, M., Nakaima, A., Li, X., Xiao, Y., & Chen, E. (2017). An experiment on building evaluation capacity to address health inequities in China. New Directions for Evaluation, 154, 17–28. https://doi.org/10.1002/ev.20239