Insufficient Effort Responding in Surveys Assessing Self-Regulated Learning: Nuisance or Fatal Flaw?

Main Article Content

Ryan Iaconelli
Christopher A. Wolters


Despite concerns about their validity, self-report surveys remain the primary data collection method in the research of self-regulated learning (SRL). To address some of these concerns, we took a data set comprised of college students’ self-reported beliefs and behaviours related to SRL, assessed across three surveys, and examined it for instance of a specific threat to validity, insufficient effort responding (IER; Huang, Curran, Keeny, Poposki, & DeShon, 2012). Using four validated indicators of IER, we found the rate of IER to vary between 12-16%. Critically, while we found that students characterised as inattentive and attentive differed in some basic descriptive statistics, the inclusion of inattentive students within the data set did not alter more substantial inferences or conclusions drawn from the data. This study provides the first direct examination of the impact of respondents’ attention on the validity of SRL data generated from self-report surveys.

Article Details

How to Cite
Iaconelli, R., & Wolters, C. A. (2020). Insufficient Effort Responding in Surveys Assessing Self-Regulated Learning: Nuisance or Fatal Flaw?. Frontline Learning Research, 8(3), 104–125.


Azevedo, R., Taub, M., & Mudrick, N.V. (2018). Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies. In D. Schunk & Greene, J.A (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 254-270). New York, NY: Routledge.

Bandura, A. (2006). Guide for constructing self-efficacy scales. In Self-efficacy beliefs of adolescents (pp. 307–337).

Beach, D. A. (1989). Identifying the random responder. Journal of Psychology: Interdisciplinary and Applied, 123(1), 101–103.

Biswas, G., Baker, R,, & Paquette, L. (2018). Data mining methods for assessing self-regulated learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 388 - 403). New York: Routledge.

Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218-229.

Butler, D. L. (2002). Qualitative approaches to self regulated learning: Contributions and challenges. Educational Psychologist, 37, 59–63.

Carver, R.P. (1992). Reading rate: Theory, practice, and practical mplications. Journal of Reading, 36(2), 84-95.

Choi, J. N., & Moran, S. V. (2010). Why not procrastinate? Development and validation of a new active procrastination scale why not procrastinate. The Journal of Social Psychology, 149, 37–41.

Cleary, T. J, & Callan, G. (2018). Assessing self-regulated learning using microanalytic methods. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.

Cleary, T. J., & Zimmerman, B. J. (2004). Self-regulation empowerment program: A school-based program to enhance self-regulated and self-motivated cycles of student learning. Psychology in the Schools, 41(5), 537–550.

Costa, P. T., & McCrae, R. R. (2008). The revised NEO personality inventory (NEO-PI-R). In The SAGE Handbook of Personality Theory and Assessment: Volume 2 - Personality Measurement and Testing (pp. 179–198). London: SAGE Publications Ltd.

Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.

DeSimone, J. A., & Harms, P. D. (2017). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 1–19.

Diedenhofen, B., & Musch, J. (2016). cocron : A web interface and R package for the statistical comparison of Cronbach ’ s alpha coefficients. International Journal of Internet Science, 11(1), 51–60.

Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231–264.

Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the short grit scale ( Grit – S ). Journal of Personality Assessment, 3891, 166–174.

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251.

Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2016). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 1–17.

Dweck, C. (2012). Mindset: How You Can Fulfil Your Potential. London: Robinson.

Feldt, L. S., Woodruff, D. J., & Salih, F. a. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93–103.

Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114.

Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845.

Hulleman, C. S., Durik, A. M., Schweigert, S. A., & Harackiewicz, J. M. (2008). Task values, achievement goals, and interest: An integrative analysis. Journal of Educational Psychology, 100(2), 398–416.

Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

Karabenick, S. A., & Zusho, A. (2011). Examining approaches to research on self-regulated learning: conceptual and methodological considerations. Metacognition and Learning, 10(1), 151–163.

Macan, T. H. (1994). Time management: Test of a process model. Journal of Applied Psychology, 79(3), 381–391.

Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83.

McCardle, L., & Hadwin, A. F. (2015). Using multiple, contextualized data sources to measure learners’ perceptions of their self-regulated learning. Metacognition and Learning, 10(1), 43–75.

McKibben, W. B., & Silvia, P. J. (2017). Evaluating the distorting effects of inattentive responding and social desirability on self-report scales in creativity and the arts. Journal of Creative Behavior, 51(1), 57–69.

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.

Nichols, D. S., Greene, R. L., & Schmolck, P. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: Rationale, development, and empirical trials. Journal of Clinical Psychology, 45(2), 239–250.<239::AID-JCLP2270450210>3.0.CO;2-1

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8(APR), 1–28.

Pajares, F. (2007). Motivational role of self-efficacy beliefs in self-regulated learning. In B. J. Zimmerman, & D. H. Schunk (Eds.), Motivation and Self-Regulated Learning: Theory, Research, and Applications (pp. 111-140). New York: Erlbaum.

Perry, N. E., Hutchinson, L. R., Yee, N., & Maatta, E. (2018). Advances in understanding young children’s self-regulation of learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.

Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning suestionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.

Pintrich, P. R., & Zusho, A. (2007). Students’ motivation and self-regulated learning in the college classroom. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence based perspective (pp. 731–810). New York: Springer.

Schellings, G., & van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6(2), 83–90.

Schunk, D. H., & Zimmerman, B. J. (1998). Self-regulated learning: From teaching to self-reflective practice. Psychological Science. Guilford Press.

Schwarz, N., & Oyserman, D. (2001). Asking questions about behaviour. American Journal of Evaluation, 22(2), 127–160.

Shukla, K., & Konold, T. (2018). A two-step latent profile method for identifying invalid respondents in self-reported survey data. Journal of Experimental Education, pp. 1–16.

Tuckman, B. W. (1991). The development and concurrent validity of the procrastination scale. Educational and Psychological Measurement, 51(2), 473–480.

Usher, E., & Schunk, D. H. (2018). Social cognitive thoretical perspective of self-regulation. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.

Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Erlbaum Associates.

Winne, P. H., & Jamieson-Noel, D. (2003). Self-regulating studying by objectives for learning: Students’ reports compared to a model. Contemporary Educational Psychology, 28(3), 259–276.

Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of Self-Regulation (pp. 531–566). Elsevier.

Wolters, C. A., & Benzon, M. B. (2013). Assessing and predicting college students use of strategies for the self-regulation of motivation. Journal of Experimental Education, 81(2), 199–221.

Wolters, C. A., & Won, S. (2018). Validity and the use of self-report questionnaires to assess self-regulated learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.

Zimmerman, B. J. (2008). Investigating self-Regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183.