Tracking Patterns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data

Main Article Content

Nicolette van Halem
Chris van Klaveren
Hendrik Drachsler
Marcel Schmitz
Ilja Cornelisz

Abstract

For decades, self-report instruments – which rely heavily on students’ perceptions and beliefs – have been the dominant way of measuring motivation and strategy use. An event-based measure based on online trace data arguably has the potential to remove analytical restrictions of self-report measures. The purpose of this study is therefore to triangulate constructs suggested in theory and measured using self-reported data with revealed online traces of learning behaviour. The results show that online trace data of learning behaviour are complementary to self-reports, as they explained a unique proportion of variance in student academic performance and reveal that self-reports explain more variance in online learning behaviour of prior weeks than variance in learning behaviour in succeeding weeks. Student motivation is, however, to a lesser extent captured with online trace data, likely because of its covert nature. In that respect, it is of importance to recognize the crucial role of self-reports in capturing student learning holistically. This manuscript is ‘frontline’ in the sense that event-based measurement methodologies using online trace data are relatively unexplored. The comparison with self-report data made in this manuscript sheds new light on the added value of innovative and traditional methods of measuring motivation and strategy use.

Article Details

How to Cite
van Halem, N., Klaveren, C. van, Drachsler, H., Schmitz, M., & Cornelisz, I. (2020). Tracking Patterns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data. Frontline Learning Research, 8(3), 140–163. https://doi.org/10.14786/flr.v8i3.497
Section
Articles

References

Bera, S., & Liu, M. (2006). Cognitive tools, individual differences, and group processing as mediating factors in a hypermedia environment. Computers in Human Behavior, 22, 295-319. http://doi.org/10.1016/j.chb.2004.05.001

Boekaerts, M., & Corno, L. (2005). Self‐regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54, 199-231. http://doi.org/10.1016/j.chb.2004.05.001

Boekaerts, M., Pintrich, P. R., & Zeidner, M. (2000). Self-regulation: An introductory overview. In Handbook of self-regulation (pp. 1-9). Academic Press.

Charleer, S., Moere, A. V., Klerkx, J., Verbert, K., & De Laet, T. (2017). Learning analytics dashboards to support adviser-student dialogue. IEEE Transactions on Learning Technologies, 11, 389-399. http://doi.org/10.1109/TLT.2017.2720670

Cho, M. H., & Heron, M. L. (2015). Self-regulated learning: the role of motivation, emotion, and use of learning strategies in students’ learning experiences in a self-paced online mathematics course. Distance Education, 36, 80-99. http://doi.org/10.1080/01587919.2015.1019963

Cho, M. H., & Yoo, J. S. (2017). Exploring online students’ self-regulated learning with self-reported surveys and log files: a data mining approach. Interactive Learning Environments, 25, 970-982. http://doi.org/10.1080/10494820.2016.1232278

Coogan, J., Dancey, C. P., & Attree, E. A. (2005). WebCT: a useful support tool for psychology undergraduates–a Q methodological study. Psychology Learning and Teaching, 5, 61-66. http://doi.org/10.2304/plat.2005.5.1.61

Cornelisz, I., & Van Klaveren, C. (2018). Student engagement with computerized practising: Ability, task value, and difficulty perceptions. Journal of Computer Assisted Learning, 34(6), 828-842. http://doi.org/10.1111/jcal.12292

DeNeui, D. L., & Dodge, T. L. (2006). Asynchronous learning networks and student outcomes: The utility of online learning components in hybrid courses. Journal of Instructional Psychology, 33, 256-260.

Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391-409. http://doi.org/10.1007/s10648-008-9083-6

Greller, W. & Drachsler, H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics. Journal of Educational Technology & Society, 15, 42-57. Retrieved from http://www.jstor.org/stable/jeductechsoci.15.3.42

Chauliac, M., Catrysse, L., Gijbels, D., & Donche, V. (2020). It is all in the surv-eye: Can eye tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies? Frontline Learning Research, 8(3), 26–39. http://doi.org/10.14786/flr.v8i3.489

Duncan, T. G., & McKeachie, W. J. (2005). The making of the motivated strategies for learning questionnaire. Educational psychologist, 40, 117-128. http://doi.org/10.1207/s15326985ep4002_6

Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46, 6-25. http://doi.org/10.1080/00461520.2011.538645

Ellis, R. A., Marcus, G., & Taylor, R. (2005). Learning through inquiry: student difficulties with online course‐based Material. Journal of Computer Assisted Learning, 21, 239-252. http://doi.org/10.1111/j.1365-2729.2005.00131.x.

Entwistle, N., & McCune, V. (2004). The conceptual bases of study strategy inventories. Educational Psychology Review, 16, 325-345. http://doi.org/10.1007/s10648-004-0003-0

Fryer, L. K. (2017). Building bridges: Seeking structure and direction for higher education motivated learning strategy models. Educational Psychology Review, 29, 325-344. http://doi.org/10.1007/s10648-017-9405-7

Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68-84. http://doi.org/10.1016/j.iheduc.2015.10.002.

Gašević, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. The International Review of Research in Open and Distributed Learning, 15, 134-176. http://doi.org/10.19173/irrodl.v15i5.1954.

Guerra, J., Hosseini, R., Somyurek, S., & Brusilovsky, P. (2016). An intelligent interface for learning content: Combining an open learner model and social comparison to support self-regulated learning and engagement. In Proceedings of the 21st international conference on intelligent user interfaces (pp. 152-163). ACM.

Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., & Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2-3), 107-124. http://doi.org/10.1007/s11409-007-9016-7

Han, J., Pei, J., & Kamber, M. (2011). Data mining: concepts and techniques. Elsevier.

Heffner, M., & Cohen, S. H. (2005). Evaluating student use of web-based course material. Journal of Instructional Psychology, 32, 74-82.

Hilpert, J. C., Stempien, J., van der Hoeven Kraft, K. J., & Husman, J. (2013). Evidence for the latent factor structure of the MSLQ: A new conceptualization of an established questionnaire. SAGE Open, 3, 2158244013510305.

Hoskins, S. L., & Van Hooff, J. C. (2005). Motivation and ability: which students use online learning and what influence does it have on their achievement? British Journal of Educational Technology, 36, 177-192. http://doi.org/10.1111/j.1467-8535.2005.00451.x.

Howard-Rose, D., & Winne, P. H. (1993). Measuring component and sets of cognitive processes in self-regulated learning. Journal of Educational Psychology, 85, 591. http://doi.org/10.1111/j.1464-0597.2005.00205.x

Huon, G., Spehar, B., Adam, P., & Rifkin, W. (2007). Resource use and academic performance among first year psychology students. Higher Education, 53, 1-27. http://doi.org/10.1007/s10734-005-1727-6

Iaconelli, R., & Wolters, C. A. (2020). Insufficient effort responding in surveys assessing self-regulated learning: Nuisance or fatal flaw? Frontline Learning Research, 8(3), 105–127. http://doi.org/10.14786/flr.v8i3.521

Itmazi, J. A., & Megías, M. G. (2005). Survey: Comparison and evaluation studies of learning content management systems. Unpublished manuscript.

Jamieson-Noel, D., & Winne, P. H. (2003). Comparing Self-Reports to Traces of Studying Behavior as Representations of Students' Studying and Achievement. Zeitschrift für Pädagogische Psychologie/German Journal of Educational Psychology. http://doi.org/10.1024//1010-0652.17.34.159

Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practice. In É. L., H. D., K. V., J. B., & M. P-S. (Eds.), Data Driven Approaches in Digital Education: 12th European Conference on Technology Enhanced Learning, EC-TEL 2017, Tallinn, Estonia, September 12–15, 2017, Proceedings (Lecture Notes in Computer Science (LNCS); Vol. 10474). Cham: Springer International Publishing AG. http://doi.org/10.1007/978-3-319-66610-5_7

Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate: preparing learning analytics dashboards for educational practice. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 31-40). ACM. http://doi.org/10.1145/3170358.3170421

Jiang, L., Elen, J., & Clarebout, G. (2009). The relationships between learner variables, tool-usage behaviour and performance. Computers in Human Behavior, 25, 501-509. http://doi.org/10.1016/j.chb.2008.12.010

Jupp, V. (2006). The Sage dictionary of social research methods. Sage.

Kim, D., Yoon, M., Jo, I. H., & Branch, R. M. (2018). Learning analytics to support self-regulated learning in asynchronous online courses: A case study at a women's university in South Korea. Computers & Education, 127, 233-251. http://doi.org/10.1016/j.compedu.2018.08.023.

Knight, J. (2010). Distinguishing the learning approaches adopted by undergraduates in their use of online resources. Active Learning in Higher Education, 11, 67–76. http://doi.org/10.1177/1469787409355873

Lust, G., Collazo, N. A. J., Elen, J., & Clarebout, G. (2012). Content management systems: enriched learning opportunities for all? Computers in Human Behavior, 28, 795-808. http://doi.org/10.1016/j.chb.2011.12.009

Lust, G., Elen, J., & Clarebout, G. (2013). Regulation of tool-use within a blended course: Student differences and performance effects. Computers & Education, 60, 385-395. http://doi.org/10.1016/j.compedu.2012.09.001

McKeachie, W. J., Pintrich, P. R., & Lin, Y. G. (1985). Teaching learning strategies. Educational Psychologist, 20, 153-160. http://doi.org/10.1207/s15326985ep2003_5

Molenda, M. (2008). Historical foundations. In M. J. Spector, M. D. Merrill, J. van Merrienboer, & M. P. Driscoll (Eds.). Handbook of research for educational communications and technology (pp. 5–20). Routledge.

Moeller, J., Dietrich, J., Viljaranta, J., & Kracke, B. (2020). Disentangling objective characteristics of learning situations from subjective perceptions thereof, using an experience sampling method design. Frontline Learning Research, 8(3), 63–85. http://doi.org/10.14786/flr.v8i3.529

Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77, 177–195. http://doi.org/10.1348/000709905X90876

Munoz, KD, & Van Duzer, J. (2005). BlackBoard vs. Moodle: A Comparison of Satisfaction with Online Teaching and Learning Tools. Humboldt State University.

Panadero, E. (2017). A review of self-regulated learning: six models and four directions for research. Frontiers in Psychology, 8, 422. http://doi.org/10.3389/fpsyg.2017.00422

Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60, 723-735. http://doi.org/10.1080/00313831.2015.1066436

Pintrich, P. R. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ).

Pintrich, P. R. (2000). Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology, 92, 544. http://doi.org/10.1037/0022-0663.92.3.544

Pintrich, P. R., McKeachie, W. J., & Lin, Y. G. (1987). Teaching a course in learning to learn. Teaching of Psychology, 14, 81-86. http://doi.org/10.1207/s15328023top1402_3

Rogiers, A.; Merchie, E., & van Keer, H. (2020). Opening the black box of students’ text-learning processes: A process mining perspective. Frontline Learning Research, 8(3), 40–62. http://doi.org/10.14786/flr.v8i3.527

Schmitz, M., Scheffel, M., van Limbeek, E., van Halem, N., Cornelisz, I., van Klaveren, C., ... & Drachsler, H. (2018). Investigating the Relationships Between Online Activity, Learning Strategies and Grades to Create Learning Analytics-Supported Learning Designs. In European Conference on Technology Enhanced Learning (pp. 311-325). Springer, Cham.

Taylor, S. E. & Brown, J. D. (1988). Illusion and well-being: A social psychological perspective on mental health. Psychological Bulletin, 103, 193–210. http://doi.org/10.1037/0033-2909.103.

Tock, J. L., & Moxley, J. H. (2017). A comprehensive reanalysis of the metacognitive self-regulation scale from the MSLQ. Metacognition and Learning, 12, 79-111. http://doi.org/10.1007/s11409-016-9161-y

Veenman, M. (2011). Learning to self-monitor and self-regulate. In R. Mayer & P. Alexander (Eds.), Handbook of research on learning and instruction (pp. 197–218). New York: Routledge.

Vriesema, C. C., & McCaslin, M. (2020) Experience and meaning in small-group contexts: Fusing observational and self-report data to capture self and other dynamics. Frontline Learning Research, 8(3), 128–141. http://doi.org/10.14786/flr.v8i3.493

Wang, A. Y., & Newlin, M. H. (2000). Characteristics of students who enroll and succeed in psychology web-based classes. Journal of Educational Psychology, 92, 137. http://doi.org/10.1037/0022-0663.92.1.137.

Winne, P. H., & Baker, R. S. (2013). The potentials of educational data mining for researching metacognition, motivation and self-regulated learning. Journal of Educational Data Mining, 5, 1-8. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/28

Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation. Academic Press.

Zhou, M., & Winne, P. H. (2012). Modelling academic performance by self-reported versus traced goal orientation. Learning and Instruction, 22, 413-419. http://doi.org//j.learninstruc.2012.03.004

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45, 166–183. http://doi.org/10.3102/0002831207312909