Sampling and the Curse of the Case Study
Keywords:
Preventing Violent Extremism, PVE, Countering Violent Extremism, CVE, P/CVE, Design, Evaluation, SamplingAbstract
Regardless of how the outcomes of a given P/CVE program are measured or evaluated, a fundamental, implicit (if not explicit) research question is: to what extent can obtained results apply to others within a given population. In short, to what extent can the results apply to others, in general; what is the so-called generalizability of the findings? In other words, the outcomes of a given P/CVE program are relatively useless unless they can be replicated, and the likelihood of replication is synonymous with generalizability. Therefore, it is virtually impossible to overstate the importance of generalizability with respect to P/CVE research and evaluation, and generalizability is fundamentally a function of how well sampling is performed. Therefore, it is also virtually impossible to overstate the importance of sampling with respect to an evidence-based approach to P/CVE. This research methods brief describes fundamental issues (including potential pitfalls and means to avoid them) with respect to sampling in the context of P/CVE program design and evaluation: including issues related to sampling online “Big Data,” and “nested” (multi-level/hierarchical) program/research designs.
References
Baldwin, S. A., Bauer, D. J., Stice, E., & Rohde, P. (2011). Evaluating models for partially clustered designs. In Psychological Methods (Vol. 16, Issue 2, pp. 149–165). American Psychological Association. https://doi.org/10.1037/a0023464
Cialdini, R. B. (2013). Influence: Science and practice (5th ed.). Pearson.
Global Counter Terrorism Forum. (2013). Ankara memorandum on good practices for a multisectoral approach to countering violent extremism. https://www.thegctf.org/documents/10162/72352/13Sep19_Ankara+Memorandum.pdf
Groves, R. M., Fowler Jr, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology. John Wiley & Sons.
Kaplan, R. M., Chambers, D. A., & Glasgow, R. E. (2014). Big data and large sample size: A cautionary note on the potential for bias. Clinical and Translational Science, 7(4), 342–346. https://doi.org/10.1111/cts.12178
Kim, Y. J., Oh, Y., Park, S., Cho, S., & Park, H. (2013). Stratified sampling design based on data mining. Healthcare Informatics Research, 19(3), 186–195. https://doi.org/10.4258/hir.2013.19.3.186
Leetaru, K. (2019). The Big Data revolution will be sampled: How “Big Data” has come to mean “small sampled data.” Forbes. https://www.forbes.com/sites/kalevleetaru/2019/02/17/the-big-data-revolution-will-be-sampled-how-big-data-has-come-to-mean-small-sampled-data/?sh=1ffc0051199e
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y
Pratkanis, A. R. (Ed.). (2011). The science of social influence: Advances and future progress. Psychology Press.
Ruggiero, G. (2020). How Big Data killed sampling. Altavia. https://medium.com/altavia/how-big-data-killed-sampling-72205a8e1b6a
Sagan, C. (2011). The demon-haunted world: Science as a candle in the dark. Ballantine Books.
Sample. (n.d.). Better Evaluation. Retrieved June 15, 2019, from https://www.betterevaluation.org/en/rainbow_framework/describe/sample
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer (Vol. 1). Sage.
West, S. G., Biesanz, J. C., & Kwok, O. M. (2004). Within-subject and longitudinal experiments: Design and analysis issues. In C. Sansone, C. C. Morf, & A. T. Panter (Eds.), The SAGE Handbook of Methods in Social Psychology (pp. 287–312). https://doi.org/10.4135/9781412976190.n13
Whittemore, A. S., & Halpern, J. (1997). Multi-stage sampling in genetic epidemiology. Statistics in Medicine, 16(1–3), 153–167. https://doi.org/10.1002/(sici)1097-0258(19970130)16:2<153::aid-sim477>3.0.co;2-7
Williams, M. J. (2022). Incentivizing Participants P/CVE Research, Evaluation, & Program Participants. Journal for Deradicalization, 31, 164–174.
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Michael J. Williams
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The JD Journal for Deradicalization uses a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND) Licence. You are free to share - copy and redistribute the material in any medium or format under the following conditions:
Attribution — You must give appropriate credit, provide a link to the license, andindicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
