Integrating Evaluation and Agent-Based Modeling: Rationale and an Example for Adopting Evidence-Based Practices
Main Article Content
Abstract
Background: While there is a great deal of discussion about complex adaptive systems in the field of evaluation, little has been written about the expression of complex adaptive systems in terms of agent-based models, the execution of those models as computer simulations, or the tight integration of agent-based modeling with traditional evaluation methods. These topics need to be explored if evaluation is to move beyond using complex adaptive systems in an exclusively heuristic fashion.
Purpose: This paper advocates for the integration of agent based modeling into traditional evaluation activities. We advance this position in order to spark a movement toward incorporating the formal application of complex adaptive systems into the theory and methods of evaluation. To make our case, we provide a hypothetical example of how interactions between an agent-based model and evaluation can provide unique and powerful understanding about the adoption of evidence-based practices in the field of addiction treatment. We advance our argument by addressing four questions: Why combine evaluation methods with simulation methods? Why use a complex adaptive systems approach? Why add agent-based modeling to the mix of evaluation tools? What is the relationship between agent-based modeling and the theories and methods of evaluation that are being developed to deal with surprise, unintended consequences, and program evolution?
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Bennett, D. S. (2008). Governments, civilians, and the evolution of insurgency: Modeling the early dynamics of insurgencies. Journal of Artificial Societies and Social Simulation, 11(4). Retrieved July 15, 2010, from http://jasss.soc.surrey.ac.uk/11/4/7.html
Boren, S. A., & Balas, E. A. (1999). Evidence-based quality measurement. Journal of Ambulatory Care Management, 22(3), 17-23. https://doi.org/10.1097/00004479-199907000-00005
Cabrera, D., Colosi, L., & Lobdell, C. (2008). Systems thinking. Evaluation and Program Planning, 31(3), 299- 310. https://doi.org/10.1016/j.evalprogplan.2007.12.001
Capoccia, V. A., Cotter, F., Gustafson, D. H., Cassidy, E. F., Ford, J. H., Madden, L., Owens, B. H., et al. (2007). Making "stone soup": Improvements in clinic access and retention in addiction treatment. Joint Commission Journal on Quality and Patient Safety, 33, 95- 103. https://doi.org/10.1016/S1553-7250(07)33011-0
Chen, S. L., Clark, S., Pierce, L. J., Hayes, D. F., Helvie, M. A., Greeno, P. L., Newman, L. A., et al. (2004). An academic health center cost analysis of screening mammography. Cancer, 101(5), 1043-1050. https://doi.org/10.1002/cncr.20476
Condon, T. P., Miner, L. L., Balmer, C. W., & Pintello, D. (2008). Blending addiction research and practice: Strategies for technology transfer. Journal of Substance Abuse Treatment, 35(2), 156-160. https://doi.org/10.1016/j.jsat.2007.09.004
Deming, W. E. (1986). Out of the Crisis. Cambridge, MA: MIT Press.
Diez Roux, A. V. (2008). Next steps in understanding the multilevel determinants of health. Journal of Epidemiology & Community Health, 62(11), 957-959. https://doi.org/10.1136/jech.2007.064311
Eddy, D. M. (2005). Evidence-based medicine: A unified approach. Health Affairs, 24(1), 9-17. https://doi.org/10.1377/hlthaff.24.1.9
Fals-Stewart, W., O'Farrell, T. J., & Birchler, G. R. (2004). Behavioral couples therapy for substance abuse: Rationale, methods, and findings. NIDA Science & Practice Perspectives, 2(2), 30-41. https://doi.org/10.1151/spp042230
Ferlie, E. B., & Shortell, S. M. (2001). Improving the quality of health care in the United Kingdom and the United States: A framework for change. The Milbank Quarterly, 79(2), 281-315. https://doi.org/10.1111/1468-0009.00206
Fountain, D. L. (1992). Avoiding the quality assurance boondoggle in drug treatment programs through total quality management. Journal of Substance Abuse Treatment, 9(4), 355-364. https://doi.org/10.1016/0740-5472(92)90031-I
Friedman, P. D., Taxman, F. S., & Henderson, C. E. (2007). Evidence- based treatment practices for drug- involved adults in the criminal justice system. Journal of Substance Abuse Treatment, 32(3), 267-277. https://doi.org/10.1016/j.jsat.2006.12.020
Galea, S., Riddle, M., & Kaplan, G. A. (2009). Causal thinking and complex system approaches in epidemiology. International Journal of Epidemiology, 39(1), 97-106. https://doi.org/10.1093/ije/dyp296
Garner, B. R. (2009). Research on the diffusion of evidence-based treatments within substance abuse treatment: A systematic review. Journal of Substance Abuse Treatment, 36(4), 376-399. https://doi.org/10.1016/j.jsat.2008.08.004
Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x
Guydish, J., Turcotte Manser, S., Jessup, M. A., Tajima, B., & Sorensen, J. L. (2007). Implementation of evidence based practices: Learning from clinical trials. Paper presented at the Co-Occurring Center for Excellence (COCE/COSIG), National Drug Research Institute (NDRI).
Hammond, R. A. (2009). Complex systems modeling for obesity research. Preventing Chronic Disease, 6(3). Retrieved from http://www.cdc.gov/pcd/issues/2009/jul/09_0017.htm
Haug, N. A., Shopshire, M., Tajima, B., Gruber, V., & Guydish, J. (2008). Adoption of evidence-based practices among substance abuse treatment providers. Journal of Drug Education, 38(2), 181-192. https://doi.org/10.2190/DE.38.2.f
Hawe, P., Shiell, A., & Riley, T. (2009). Theorising Interventions as Events in Systems. American Journal of Community Psychology, 43(3-4), 267- 276. https://doi.org/10.1007/s10464-009-9229-9
Henderson, C. E., Taxman, F. S., & Young, D. W. (2008). A Rasch model analysis of evidence-based treatment practices used in the criminal justice system. Drug and Alcohol Dependence, 93(1- 2), 163-175. https://doi.org/10.1016/j.drugalcdep.2007.09.010
Henderson, C. E., Young, D. W., Farrell, J., & Taxman, F. S. (2009). Associations among state and local organizational contexts: Use of evidence-based practices in the criminal justice system. Drug and Alcohol Dependence, 103 (Supplement 1), S23-S32. https://doi.org/10.1016/j.drugalcdep.2008.12.006
Hirsch, G. B., Levine, R., & Miller, R. L. (2007). Using system dynamics modeling to understand the impact of social change initiatives. American Journal of Community Psychology, 39(3-4), 239-253. https://doi.org/10.1007/s10464-007-9114-3
Hoffman, K. A., Ford II, J. H., Choi, D., Gustafson, D. H., & McCarty, D. (2008). Replication and sustainability of improved access and retention within the network for the improvement of addiction treatment. Drug and Alcohol Dependence, 98(1- 2), 63-69. https://doi.org/10.1016/j.drugalcdep.2008.04.016
Huang, T. T., Drewnowski, A., Kumanyika, S. K., & Glass, T. A. (2009). A systems-oriented multilevel framework for addressing obesity in the 21st century. Preventing Chronic Disease, 6(3). Retrieved July 15, 2010, from http://www.cdc.gov/pcd/issues/2009 / jul/09_0013.htm.
Institute of Medicine. (1998). Bridging the Gap between Practice and Research: Forging Partnerships with Community based Drug and Alcohol Treatment. Washington, DC: National Academy Press.
Jackson, J., Forest, B., & Sengupta, R. (2008). Agent-based simulation of urban residential dynamics and land rent change in a gentrifying area of Boston. Transactions in GIS, 12(4), 475-491. https://doi.org/10.1111/j.1467-9671.2008.01109.x
Jiang, W., Bin, H., Yu, Z., Spence, C., Hall, S. B., & Carley, K. M. (2009). An agent-based simulation study for exploring organizational adaptation. Simulation, 85(6), 397-413. https://doi.org/10.1177/0037549709105267
Knudsen, H. K., & Roman, P. M. (2004). Modeling the use of innovations in private treatment organizations: The role of absorptive capacity. Journal of Substance Abuse Treatment, 26(1), 51- 59. https://doi.org/10.1016/S0740-5472(03)00158-2
Magura,S.(2000).Introduction: Program quality in substance dependency treatment. Substance Use & Misuse, 35(12), 1617 - 1627. https://doi.org/10.3109/10826080009148234
McCarty, D., Gustafson, D. H., Wisdom, J. P., Ford, J., Choi, D., Molfenter, T., Capoccia, V., et al. (2006). The network for the improvement of addiction treatment (NIATx): Enhancing access and retention. Drug and Alcohol Dependence, 88(2-3), 138-145. https://doi.org/10.1016/j.drugalcdep.2006.10.009
Miller, J. H., & Page, S. E. (2007). Complex Adaptive Systems: An Introduction to Computational Models of Social Life. Princeton, NJ: Princeton University Press.
Miller, W. R., Sorensen, J. L., Selzer, J. A., & Brigham, G. S. (2006). Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment, 31(1), 25-39. https://doi.org/10.1016/j.jsat.2006.03.005
Mills, P. D., & Weeks, W. B. (2004). Characteristics of successful quality improvement teams: Lessons from five collaborative projects in the VHA. Joint Commission Journal on Quality and Patient Safety, 30, 152-162. https://doi.org/10.1016/S1549-3741(04)30017-1
Mitchell, M. (2009). Complexity: A guided Tour. New York: Oxford University Press. https://doi.org/10.1093/oso/9780195124415.001.0001
Morell, J. A. (2005). Why are there unintended consequences of program action, and what are the implications for doing evaluation? American Journal of Evaluation, 26(4), 444- 463. https://doi.org/10.1177/1098214005281354
Morell, J. A. (2010). Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable. New York: Guilford.
Mosby's Medical Dictionary. (2005). (7th ed.). St. Louis, MO.
Nolan, K., Schall, M. W., Erb, F., & Nolan, T. (2005). Using a framework for spread: The case of patient access in the Veterans Health Administration. Joint Commission Journal on Quality and Patient Safety, 31, 339-347. https://doi.org/10.1016/S1553-7250(05)31045-2
Paley, J. (2007). Complex adaptive systems and nursing. Nursing Inquiry, 14(3), 233 - 242. https://doi.org/10.1111/j.1440-1800.2007.00359.x
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford.
Perez, L., & Dragicevic, S. (2009). An agent-based approach for modeling dynamics of contagious disease spread. International Journal of Health Geographics, 8(1), 50-50. https://doi.org/10.1186/1476-072X-8-50
Petrakis, I. L., Leslie, D., & Rosenheck, R. (2003). Use of Naltrexone in the treatment of alcoholism nationally in the Department of Veterans Affairs. Alcoholism: Clinical and Experimental Research, 27(11), 1780- 1784. https://doi.org/10.1097/01.ALC.0000095861.43232.19
Pronovost, P. J., Morlock, L., Davis, R. O., Cunningham, T., Paine, L., & Scheulen, J. (2000). Using online and offline change models to improve ICU access and revenues. Joint Commission Journal on Quality and Patient Safety, 26, 5-17. https://doi.org/10.1016/S1070-3241(00)26001-X
Rogers, E. M. (2003). Diffusion of innovation (5th ed.). New York: Free Press.
Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (2000). Evidence-based Medicine: How to Practice and Teach EBM (2nd ed.). London, UK: Churchill Livingstone.
Schwarz, N., & Ernst, A. (2009). Agent- based modeling of the diffusion of environmental innovations-An empirical approach. Technological Forecasting & Social Change, 76, 497- 511. https://doi.org/10.1016/j.techfore.2008.03.024
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi- experimental designs for generalized casual inference. New York: Houghton-Mifflin.
Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. BMJ, 336, 1281- 1283. https://doi.org/10.1136/bmj.39569.510521.AD
Sloboda, Z., & Schildhaus, S. (2002). A discussion of the concept of technology transfer of research-based drug ''abuse'' prevention and treatment interventions. Substance Use & Misuse, 37(8-10), 1079-1087. https://doi.org/10.1081/JA-120004167
Smith, K. P., & Christakis, N. (2008). Social networks and health. Annual Reviews of Sociology, 34, 405-429. https://doi.org/10.1146/annurev.soc.34.040507.134601
Solberg, L. I. (2000). Failure of a continuous quality improvement intervention to increase the delivery of preventive services: A randomized trial. Effective Clinical Practice, 30(3), 105-115.
Sterman, J. (2000). Business dynamics: Systems thinking and modeling for a complex world. Irwin: McGraw-Hill.
Tawfik, A. Y., & Farag, R. R. (2008). Modeling the spread of preventable diseases: Social culture and epidemiology. In IFIP International Federation for Information Processing (Vol. 276, pp. 277-286). Presented at the Artificial Intelligence and Practice II, Boston: Springer. https://doi.org/10.1007/978-0-387-09695-7_27
Trickett, E. J. (2009). Multilevel community-based culturally situated interventions and community impact: An ecological perspective. American Journal of Community Psychology, 43(3-4), 257-266. https://doi.org/10.1007/s10464-009-9227-y
Wang, A., Wolf, M., Carlyle, R., Wilkerson, J., Porterfield, D., & Reaves, J. (2004). The North Carolina experience with the Diabetes Health Disparities Collaboratives. Joint Commission Journal on Quality and Patient Safety, 30, 396-404. https://doi.org/10.1016/S1549-3741(04)30045-6
Wasserman, D. L. (2010). Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations. Evaluation and Program Planning. Retrieved July 15, 2010, from http://www.sciencedirect.com/science/article/B6V7V-4WJHB6F-1/2/66d9e5ff8e744048fc8aeae9b1cf5cfa
Williams, B. (2005). Systems thinking: Using systems concepts. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 406 - 412). Thousand Oaks, CA: Sage.
Williams, B., & Imam, I. (2007). Systems concepts in evaluation: An expert anthology. Point Reyes, CA: Edge Press.