A mixed method approach to studying self-regulated learning in MOOCs combining trace data with interviews
Main Article Content
Abstract
To be successful in online education, learners should be able to self-regulate their learning due to the autonomy offered to them. Accurate measurement of learners’ self-regulated learning (SRL) in online education is necessary to determine which learners are in need of support and how to best offer support. Trace data is gathered automatically and unobtrusively during online education, and is therefore considered a valuable source to measure learners’ SRL. However, measuring SRL with trace data is challenging for two main reasons. First, without information on the how and why of learner behaviour it is difficult to interpret trace data correctly. Second, SRL activities outside of the online learning environment are not captured in trace data. To address these two challenges, we propose a mixed method approach with a sequential design. Such an approach is novel for the measurement of SRL. We present a pilot study in which we combined trace data with interview data to analyse learners’ SRL in online courses. In the interview, cued retrospective reporting was conducted by presenting learners with visualizations of their trace data. In the second part of the interview, learners’ activities outside of the online course environment were discussed. The results show that the mixed-method approach is indeed a promising approach to address the two described challenges. Suggestions for future research are provided, and include methodological considerations such as how to best visualize trace data for cued retrospective recall.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Allen, I. E., & Seaman, J. (2014). Grade change: Tracking Online Education in the United States.
Allen, I. E., & Seaman, J. (2016). Online report card: Tracking online education in the United States.
Alshenqeeti, H. (2014). Interviewing as a Data Collection Method: A Critical Review. English Linguistics Research, 3(1). https://doi.org/10.5430/elr.v3n1p39
Azevedo, R., & Aleven, V. (2013). Metacognition and Learning Technologies: An Overview of Current Interdisciplinary Research. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (Vol. 28, pp. 1–16). New York, NY: Springer New York. Retrieved from http://link.springer.com/10.1007/978-1-4419-5546-3_1
Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96, 523–535. https://doi.org/10.1037/0022- 0663.96.3.523
Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S.-L. (2009). Measuring self-regulation in online and blended learning environments. The Internet and Higher Education, 12(1), 1–6. https://doi.org/10.1016/j.iheduc.2008.10.005
Beishuizen, J., & Steffens, K. (2011). A conceptual framework for research on self-regulated learning. In R. Carneiro, P. Lefrere, K. Steffens, & J. Underwood (Eds.), Self-Regulated Learning in Technology Enhanced Learning Environments (pp. 3–19). Rotterdam, The Netherlands: Sense Publishers.
Ben-Eliyahu, A., & Linnenbrink-Garcia, L. (2015). Integrating the regulation of affect, behaviour, and cognition into self-regulated learning paradigms among secondary and post-secondary students. Metacognition and Learning, 10, 15–42. https://doi.org/10.1007/s11409-014-9129-8
Broadbent, J. (2017). Comparing online and blended learner’s self-regulated learning strategies and academic performance. The Internet and Higher Education, 33, 24–32. https://doi.org/10.1016/j.iheduc.2017.01.004
Cicchinelli, A., Veas, E., Pardo, A., Pammer-Schindler, V., Fessl, A., Barreiros, C., & Lindstädt, S. (2018). Finding traces of self-regulated learning in activity streams (pp. 191–200). ACM Press. https://doi.org/10.1145/3170358.3170381
Creswell, J. W. (2008). Mixed method designs. In Educational research: Planning, conducting, and evaluating quantitative and qualitative research (pp. 551–595). New Jersey, NJ: Pearson Education International.
DiCicco-Bloom, B., & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314–321. https://doi.org/10.1111/j.1365-2929.2006.02418.x
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Cambridge, MA: MIT press.
Fincham, O. E., Gasevic, D. V., Jovanovic, J. M., & Pardo, A. (2018). From Study Tactics to Learning Strategies: An Analytical Method for Extracting Interpretable Representations. IEEE Transactions on Learning Technologies, 1–1. https://doi.org/10.1109/TLT.2018.2823317
Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2, 107–124. https://doi.org/10.1007/s11409-007-9016-7
Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45–58. https://doi.org/10.1016/j.edurev.2014.05.001
Howard-Rose, D., & Winne, P. H. (1993). Measuring component and sets of cognitive processes in self- regulated learning. Journal of Educational Psychology, 85(4), 591–604. https://doi.org/10.1037/0022-0663.85.4.591
Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29, 6–27. https://doi.org/10.1007/s12528-016-9125-x
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14–26. https://doi.org/10.3102/0013189X033007014
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85. https://doi.org/10.1016/j.iheduc.2017.02.001
Karabenick, S. A., & Zusho, A. (2015). Examining approaches to research on self-regulated learning: conceptual and methodological considerations. Metacognition and Learning, 10(1), 151–163. https://doi.org/10.1007/s11409-015-9137-3
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behaviour and goal attainment in Massive Open Online Courses. Computers & Education, 104, 18–33. https://doi.org/10.1016/j.compedu.2016.10.001
Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self- regulated learning in MOOCs. The Internet and Higher Education, 29, 40–48. https://doi.org/10.1016/j.iheduc.2015.12.003
Maldonado-Mahauad, J., Pérez-Sanagustín, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from big data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behaviour, 80, 179–196. https://doi.org/10.1016/j.chb.2017.11.011
McGrath, J. E., Martin, J., & Kulka, R. A. (1981). Some Quasi-Rules for Making Judgment Calls in Research. American Behavioural Scientist, 25(2), 211–224. https://doi.org/10.1177/000276428102500206
McIntosh, M. J., & Morse, J. M. (2015). Situating and Constructing Diversity in Semi-Structured Interviews. Global Qualitative Nursing Research, 2, 233339361559767. https://doi.org/10.1177/2333393615597674
Min, L., & Jingyan, L. (2017). Assessing the effectiveness of self-regulated learning in MOOCs using macro-level behavioural sequence data. In Proceedings of EMOOCs 2017 (pp. 1–9). Madrid, Spain.
Morse, J. M. (2010). Sampling in grounded theory. In A. Bryant & K. Charmaz (Eds.), The SAGE handbook of grounded theory (pp. 229–244). SAGE publications.
Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new findings. In Psychology of Learning and Motivation (Vol. 26, pp. 125–173). Academic Press.
Panadero, E. (2017). A Review of Self-regulated Learning: Six Models and Four Directions for Research. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00422
Phillips, R., Maor, D., Cumming-Potvin, W., Roberts, P., Herrington, J., Preston, G., … Perry, L. (2011). Learning analytics and study behaviour: A pilot study. Presented at the Australasian Society for Computers in Learning in Tertiary Education, Hobart, Tasmania, Australia.
Puustinen, M., & Pulkkinen, L. (2001). Models of self-regulated learning: A review. Scandinavian Journal of Educational Research, 45, 269–286. https://doi.org/10.1080/00313830120074206
Reimann, P., Markauskaite, L., & Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute?: e-Research and learning theory. British Journal of Educational Technology, 45(3), 528–540. https://doi.org/10.1111/bjet.12146
Robinson, O. C. (2014). Sampling in Interview-Based Qualitative Research: A Theoretical and Practical Guide. Qualitative Research in Psychology, 11(1), 25–41. https://doi.org/10.1080/14780887.2013.801543
Rovers, S. F. E., Clarebout, G., Savelberg, H. H. C. M., de Bruin, A. B. H., & van Merriënboer, J. J. G. (2019). Granularity matters: comparing different ways of measuring self-regulated learning. Metacognition and Learning. https://doi.org/10.1007/s11409-019-09188-6
Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & Cognition, 17(6), 759–769. https://doi.org/10.3758/BF03202637
Salmeron, L., Gil, L., & Bråten, I. (2018). Using eye-tracking to assess sourcing during multiple document
reading: A critical analysis. Frontline Learning Research, 6(3), 105-122. https://doi.org/10.14786/flr.v6i3.368
Schraw, G. (2010). Measuring Self-Regulation in Computer-Based Learning Environments. Educational Psychologist, 45(4), 258–266. https://doi.org/10.1080/00461520.2010.515936
Strauss, A., & Corbin, J. (1994). Grounded theory methodology: an overview. In N. K. Denzin & Y. S. Lincoln (Eds.) (Vol. 17, pp. 273–285). Thousand Oaks, CA, US: SAGE publications.
Trevors, G., Feyzi-Behnagh, R., Azevedo, R., & Bouchet, F. (2016). Self-regulated learning processes vary
as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and
concurrent and retrospective reports. Learning and Instruction, 42, 31-46. https://doi.org/10.1016/j.learninstruc.2015.11.003
van Gog, T., Paas, F., van Merriënboer, J. J. G., & Witte, P. (2005). Uncovering the Problem-Solving Process: Cued Retrospective Reporting Versus Concurrent and Retrospective Reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237
Van Laer, S., & Elen, J. (2018). Towards a methodological framework for sequence analysis in the field of self-regulated learning. Frontline Learning Research, 6(3), 228–249.
Veletsianos, G., Reich, J., & Pasquini, L. A. (2016). The Life Between Big Data Log Events: Learners Strategies to Overcome Challenges in MOOCs. AERA Open, 2(3). https://doi.org/10.1177/2332858416657002
Wang, C.-H., Shannon, D. M., & Ross, M. E. (2013). Students’ characteristics, self-regulated learning, technology self-efficacy, and course outcomes in online learning. Distance Education, 34, 302–323. https://doi.org/10.1080/01587919.2013.835779
Winne, P. H. (2010). Improving Measurements of Self-Regulated Learning. Educational Psychologist, 45(4), 267–276. https://doi.org/10.1080/00461520.2010.517150
Winne, P. H. (2014). Issues in researching self-regulated learning as patterns of events. Metacognition and Learning, 9(2), 229–237. https://doi.org/10.1007/s11409-014-9113-3
Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572. https://doi.org/10.1016/S0361-476X(02)00006-1
Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology, 11(4), 307–313. https://doi.org/10.1016/0361- 476X(86)90027-5