Sampling and the Curse of the Case Study

Michael J. Williams

Authors

Michael J. Williams The Science of P/CVE https://orcid.org/0000-0001-5630-9814

Keywords:

Preventing Violent Extremism, PVE, Countering Violent Extremism, CVE, P/CVE, Design, Evaluation, Sampling

Abstract

Regardless of how the outcomes of a given P/CVE program are measured or evaluated, a fundamental, implicit (if not explicit) research question is: to what extent can obtained results apply to others within a given population. In short, to what extent can the results apply to others, in general; what is the so-called generalizability of the findings? In other words, the outcomes of a given P/CVE program are relatively useless unless they can be replicated, and the likelihood of replication is synonymous with generalizability. Therefore, it is virtually impossible to overstate the importance of generalizability with respect to P/CVE research and evaluation, and generalizability is fundamentally a function of how well sampling is performed. Therefore, it is also virtually impossible to overstate the importance of sampling with respect to an evidence-based approach to P/CVE. This research methods brief describes fundamental issues (including potential pitfalls and means to avoid them) with respect to sampling in the context of P/CVE program design and evaluation: including issues related to sampling online “Big Data,” and “nested” (multi-level/hierarchical) program/research designs.

References

Baldwin, S. A., Bauer, D. J., Stice, E., & Rohde, P. (2011). Evaluating models for partially clustered designs. In Psychological Methods (Vol. 16, Issue 2, pp. 149–165). American Psychological Association. https://doi.org/10.1037/a0023464

Cialdini, R. B. (2013). Influence: Science and practice (5th ed.). Pearson.

Global Counter Terrorism Forum. (2013). Ankara memorandum on good practices for a multisectoral approach to countering violent extremism. https://www.thegctf.org/documents/10162/72352/13Sep19_Ankara+Memorandum.pdf

Groves, R. M., Fowler Jr, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology. John Wiley & Sons.

Kaplan, R. M., Chambers, D. A., & Glasgow, R. E. (2014). Big data and large sample size: A cautionary note on the potential for bias. Clinical and Translational Science, 7(4), 342–346. https://doi.org/10.1111/cts.12178

Kim, Y. J., Oh, Y., Park, S., Cho, S., & Park, H. (2013). Stratified sampling design based on data mining. Healthcare Informatics Research, 19(3), 186–195. https://doi.org/10.4258/hir.2013.19.3.186

Leetaru, K. (2019). The Big Data revolution will be sampled: How “Big Data” has come to mean “small sampled data.” Forbes. https://www.forbes.com/sites/kalevleetaru/2019/02/17/the-big-data-revolution-will-be-sampled-how-big-data-has-come-to-mean-small-sampled-data/?sh=1ffc0051199e

Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

Pratkanis, A. R. (Ed.). (2011). The science of social influence: Advances and future progress. Psychology Press.

Ruggiero, G. (2020). How Big Data killed sampling. Altavia. https://medium.com/altavia/how-big-data-killed-sampling-72205a8e1b6a

Sagan, C. (2011). The demon-haunted world: Science as a candle in the dark. Ballantine Books.

Sample. (n.d.). Better Evaluation. Retrieved June 15, 2019, from https://www.betterevaluation.org/en/rainbow_framework/describe/sample

Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer (Vol. 1). Sage.

West, S. G., Biesanz, J. C., & Kwok, O. M. (2004). Within-subject and longitudinal experiments: Design and analysis issues. In C. Sansone, C. C. Morf, & A. T. Panter (Eds.), The SAGE Handbook of Methods in Social Psychology (pp. 287–312). https://doi.org/10.4135/9781412976190.n13

Whittemore, A. S., & Halpern, J. (1997). Multi-stage sampling in genetic epidemiology. Statistics in Medicine, 16(1–3), 153–167. https://doi.org/10.1002/(sici)1097-0258(19970130)16:2<153::aid-sim477>3.0.co;2-7

Williams, M. J. (2022). Incentivizing Participants P/CVE Research, Evaluation, & Program Participants. Journal for Deradicalization, 31, 164–174.