A mixed method approach to studying self-regulated learning in MOOCs combining trace data with interviews

Main Article Content

Renee Jansen
Anouschka van Leeuwen
Jeroen Janssen
Liesbeth Kester


To be successful in online education, learners should be able to self-regulate their learning due to the autonomy offered to them. Accurate measurement of learners’ self-regulated learning (SRL) in online education is necessary to determine which learners are in need of support and how to best offer support. Trace data is gathered automatically and unobtrusively during online education, and is therefore considered a valuable source to measure learners’ SRL. However, measuring SRL with trace data is challenging for two main reasons. First, without information on the how and why of learner behaviour it is difficult to interpret trace data correctly. Second, SRL activities outside of the online learning environment are not captured in trace data. To address these two challenges, we propose a mixed method approach with a sequential design. Such an approach is novel for the measurement of SRL. We present a pilot study in which we combined trace data with interview data to analyse learners’ SRL in online courses. In the interview, cued retrospective reporting was conducted by presenting learners with visualizations of their trace data. In the second part of the interview, learners’ activities outside of the online course environment were discussed. The results show that the mixed-method approach is indeed a promising approach to address the two described challenges. Suggestions for future research are provided, and include methodological considerations such as how to best visualize trace data for cued retrospective recall.  

Article Details

How to Cite
Jansen, R., Leeuwen, A. van, Janssen, J., & Kester, L. (2020). A mixed method approach to studying self-regulated learning in MOOCs: combining trace data with interviews. Frontline Learning Research, 8(2), 35 - 64. https://doi.org/10.14786/flr.v8i2.539


Allen, I. E., & Seaman, J. (2014). Grade change: Tracking Online Education in the United States.

Allen, I. E., & Seaman, J. (2016). Online report card: Tracking online education in the United States.

Alshenqeeti, H. (2014). Interviewing as a Data Collection Method: A Critical Review. English Linguistics Research, 3(1). https://doi.org/10.5430/elr.v3n1p39

Azevedo, R., & Aleven, V. (2013). Metacognition and Learning Technologies: An Overview of Current Interdisciplinary Research. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (Vol. 28, pp. 1–16). New York, NY: Springer New York. Retrieved from http://link.springer.com/10.1007/978-1-4419-5546-3_1

Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96, 523–535. https://doi.org/10.1037/0022- 0663.96.3.523

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S.-L. (2009). Measuring self-regulation in online and blended learning environments. The Internet and Higher Education, 12(1), 1–6. https://doi.org/10.1016/j.iheduc.2008.10.005

Beishuizen, J., & Steffens, K. (2011). A conceptual framework for research on self-regulated learning. In R. Carneiro, P. Lefrere, K. Steffens, & J. Underwood (Eds.), Self-Regulated Learning in Technology Enhanced Learning Environments (pp. 3–19). Rotterdam, The Netherlands: Sense Publishers.

Ben-Eliyahu, A., & Linnenbrink-Garcia, L. (2015). Integrating the regulation of affect, behaviour, and cognition into self-regulated learning paradigms among secondary and post-secondary students. Metacognition and Learning, 10, 15–42. https://doi.org/10.1007/s11409-014-9129-8

Broadbent, J. (2017). Comparing online and blended learner’s self-regulated learning strategies and academic performance. The Internet and Higher Education, 33, 24–32. https://doi.org/10.1016/j.iheduc.2017.01.004

Cicchinelli, A., Veas, E., Pardo, A., Pammer-Schindler, V., Fessl, A., Barreiros, C., & Lindstädt, S. (2018). Finding traces of self-regulated learning in activity streams (pp. 191–200). ACM Press. https://doi.org/10.1145/3170358.3170381

Creswell, J. W. (2008). Mixed method designs. In Educational research: Planning, conducting, and evaluating quantitative and qualitative research (pp. 551–595). New Jersey, NJ: Pearson Education International.

DiCicco-Bloom, B., & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314–321. https://doi.org/10.1111/j.1365-2929.2006.02418.x

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Cambridge, MA: MIT press.

Fincham, O. E., Gasevic, D. V., Jovanovic, J. M., & Pardo, A. (2018). From Study Tactics to Learning Strategies: An Analytical Method for Extracting Interpretable Representations. IEEE Transactions on Learning Technologies, 1–1. https://doi.org/10.1109/TLT.2018.2823317

Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2, 107–124. https://doi.org/10.1007/s11409-007-9016-7

Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45–58. https://doi.org/10.1016/j.edurev.2014.05.001

Howard-Rose, D., & Winne, P. H. (1993). Measuring component and sets of cognitive processes in self- regulated learning. Journal of Educational Psychology, 85(4), 591–604. https://doi.org/10.1037/0022-0663.85.4.591

Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29, 6–27. https://doi.org/10.1007/s12528-016-9125-x

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/0013189X033007014

Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85. https://doi.org/10.1016/j.iheduc.2017.02.001

Karabenick, S. A., & Zusho, A. (2015). Examining approaches to research on self-regulated learning: conceptual and methodological considerations. Metacognition and Learning, 10(1), 151–163. https://doi.org/10.1007/s11409-015-9137-3

Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behaviour and goal attainment in Massive Open Online Courses. Computers & Education, 104, 18–33. https://doi.org/10.1016/j.compedu.2016.10.001

Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self- regulated learning in MOOCs. The Internet and Higher Education, 29, 40–48. https://doi.org/10.1016/j.iheduc.2015.12.003

Maldonado-Mahauad, J., Pérez-Sanagustín, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from big data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behaviour, 80, 179–196. https://doi.org/10.1016/j.chb.2017.11.011

McGrath, J. E., Martin, J., & Kulka, R. A. (1981). Some Quasi-Rules for Making Judgment Calls in Research. American Behavioural Scientist, 25(2), 211–224. https://doi.org/10.1177/000276428102500206

McIntosh, M. J., & Morse, J. M. (2015). Situating and Constructing Diversity in Semi-Structured Interviews. Global Qualitative Nursing Research, 2, 233339361559767. https://doi.org/10.1177/2333393615597674

Min, L., & Jingyan, L. (2017). Assessing the effectiveness of self-regulated learning in MOOCs using macro-level behavioural sequence data. In Proceedings of EMOOCs 2017 (pp. 1–9). Madrid, Spain.

Morse, J. M. (2010). Sampling in grounded theory. In A. Bryant & K. Charmaz (Eds.), The SAGE handbook of grounded theory (pp. 229–244). SAGE publications.

Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new findings. In Psychology of Learning and Motivation (Vol. 26, pp. 125–173). Academic Press.

Panadero, E. (2017). A Review of Self-regulated Learning: Six Models and Four Directions for Research. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00422

Phillips, R., Maor, D., Cumming-Potvin, W., Roberts, P., Herrington, J., Preston, G., … Perry, L. (2011). Learning analytics and study behaviour: A pilot study. Presented at the Australasian Society for Computers in Learning in Tertiary Education, Hobart, Tasmania, Australia.

Puustinen, M., & Pulkkinen, L. (2001). Models of self-regulated learning: A review. Scandinavian Journal of Educational Research, 45, 269–286. https://doi.org/10.1080/00313830120074206

Reimann, P., Markauskaite, L., & Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute?: e-Research and learning theory. British Journal of Educational Technology, 45(3), 528–540. https://doi.org/10.1111/bjet.12146

Robinson, O. C. (2014). Sampling in Interview-Based Qualitative Research: A Theoretical and Practical Guide. Qualitative Research in Psychology, 11(1), 25–41. https://doi.org/10.1080/14780887.2013.801543

Rovers, S. F. E., Clarebout, G., Savelberg, H. H. C. M., de Bruin, A. B. H., & van Merriënboer, J. J. G. (2019). Granularity matters: comparing different ways of measuring self-regulated learning. Metacognition and Learning. https://doi.org/10.1007/s11409-019-09188-6

Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & Cognition, 17(6), 759–769. https://doi.org/10.3758/BF03202637

Salmeron, L., Gil, L., & Bråten, I. (2018). Using eye-tracking to assess sourcing during multiple document

reading: A critical analysis. Frontline Learning Research, 6(3), 105-122. https://doi.org/10.14786/flr.v6i3.368

Schraw, G. (2010). Measuring Self-Regulation in Computer-Based Learning Environments. Educational Psychologist, 45(4), 258–266. https://doi.org/10.1080/00461520.2010.515936

Strauss, A., & Corbin, J. (1994). Grounded theory methodology: an overview. In N. K. Denzin & Y. S. Lincoln (Eds.) (Vol. 17, pp. 273–285). Thousand Oaks, CA, US: SAGE publications.

Trevors, G., Feyzi-Behnagh, R., Azevedo, R., & Bouchet, F. (2016). Self-regulated learning processes vary

as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and

concurrent and retrospective reports. Learning and Instruction, 42, 31-46. https://doi.org/10.1016/j.learninstruc.2015.11.003

van Gog, T., Paas, F., van Merriënboer, J. J. G., & Witte, P. (2005). Uncovering the Problem-Solving Process: Cued Retrospective Reporting Versus Concurrent and Retrospective Reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237

Van Laer, S., & Elen, J. (2018). Towards a methodological framework for sequence analysis in the field of self-regulated learning. Frontline Learning Research, 6(3), 228–249.

Veletsianos, G., Reich, J., & Pasquini, L. A. (2016). The Life Between Big Data Log Events: Learners Strategies to Overcome Challenges in MOOCs. AERA Open, 2(3). https://doi.org/10.1177/2332858416657002

Wang, C.-H., Shannon, D. M., & Ross, M. E. (2013). Students’ characteristics, self-regulated learning, technology self-efficacy, and course outcomes in online learning. Distance Education, 34, 302–323. https://doi.org/10.1080/01587919.2013.835779

Winne, P. H. (2010). Improving Measurements of Self-Regulated Learning. Educational Psychologist, 45(4), 267–276. https://doi.org/10.1080/00461520.2010.517150

Winne, P. H. (2014). Issues in researching self-regulated learning as patterns of events. Metacognition and Learning, 9(2), 229–237. https://doi.org/10.1007/s11409-014-9113-3

Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572. https://doi.org/10.1016/S0361-476X(02)00006-1

Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology, 11(4), 307–313. https://doi.org/10.1016/0361- 476X(86)90027-5