Disentangling objective characteristics of learning situations from subjective perceptions thereof, using an experience sampling method design

Main Article Content

Julia Moeller
Jaana Viljaranta
Bärbel Kracke
Julia Dietrich


This article proposes a study design developed to disentangle the objective characteristics of a learning situation from individuals’ subjective perceptions of that situation. The term ‘objective characteristics’ refers to the agreement across students, whereas ‘subjective perceptions’ refers to inter-individual heterogeneity. We describe a novel strategy for assessing and disentangling objective situation characteristics and subjective perceptions thereof, propose methods for analyzing the resulting data, and illustrate the procedure with an example of a first study using this design to examine situational interest in 155 university students. Situational interest was assessed nine times per weekly lecture with three measurement time points per person and a rotated multi-group schedule. Assessments took place over the course of an entire semester of ten weeks.

One of the advantages of the proposed design is that ‘objective’ group agreements can be disentangled from subjective deviations from the group’s average at each of the nine measurement time points per weekly lecture. Furthermore, the proposed design makes it possible to study the development of both subjective and objective parameters across the time span of one weekly lecture and an entire semester, while the burden for each person is kept relatively low with three beeps per lecture.

Article Details

How to Cite
Moeller, J., Viljaranta, J., Kracke, B., & Dietrich, J. (2020). Disentangling objective characteristics of learning situations from subjective perceptions thereof, using an experience sampling method design. Frontline Learning Research, 8(3), 63 - 84. https://doi.org/10.14786/flr.v8i3.529


Altrabsheh, N., Gaber, M. M., & Cocea, M. (2013). SA-E: Sentiment analysis for education. In: R. Neves-Silva, J. Watada, G. Philipps-Wren, L. C. Jain, & R. J. Howlett (Eds.), Intelligent Decision Technologies (pp. 353 - 362), Amsterdam: IOS Press. doi: 10.3233/978-1-61499-264-6-353

Asparouhov, T., & Muthén, B. (2012). Multiple group multilevel analysis. Mplus Web Notes: No. 16. Retrieved March 5, 2020 from https://www.statmodel.com/examples/webnotes/webnote16.pdf

Asparouhov, T. & Muthén, B. (2019). Comparison of models for the analysis of intensive longitudinal data, Structural Equation Modeling: A Multidisciplinary Journal, 00: 1–23. https://doi.org/10.1080/10705511.2019.1626733

Banister, S., Reinhart, R., & Ross, C. (2014). Using digital resources to support personalized learning experiences in K-12 classrooms: The evolution of mobile devices as innovations in schools in Northwest Ohio. In M. Searson & M. Ochoa (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2014 (pp. 2715-2721). Chesapeake, VA: Association for the Advancement of Computing in Education. Retrieved March 5, 2020 from https://www.learntechlib.org/primary/p/131202/.

Bastiaansen, J. A., Kunkels, Y. K., Blaauw, F., Boker, S. M., Ceulemans, E., Chen, M., … Bringmann, L. F. (2019, March 21). Time to get personal? The impact of researchers’ choices on the selection of treatment targets using the experience sampling methodology. Preprint retrieved on August 24, 2019 from https://doi.org/10.31234/osf.io/c8vp7

Bingham, A. J., Pane, J. F., Steiner, E. D., & Hamilton, L. S. (2018). Ahead of the curve: Implementation challenges in personalised learning school models. Educational Policy, 32(3), 454 – 489. https://doi.org/10.1177/0895904816637688

Barrett, L. F. (2018). How emotions are made. The secret life of the brain. Mariner Books: New York.

Battle, A., & Wigfield, A. (2003). College women’s value orientations toward family, career, and graduate school. Journal of Vocational Behavior, 62, 56–75. https://doi.org/10.1016/S0001-8791(02)00037-4

Beretvas, S. N. (2010). Cross-classified and multiple membership models. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge.

Bieg, M., Goetz, T., Wolter, I., & Hall, N. C. (2015). Gender stereotype endorsement differentially predicts girls' and boys' trait-state discrepancy in math anxiety. Frontiers in Psychology, 6, 1404. https://doi.org/10.3389/fpsyg.2015.01404

Carroll, E. A., Czerwinski, M., Roseway, A., Kapoor, A., Johns, P., Rowan, K., & Schraefel, M. C. (2013). Food and mood: Just-in-time support for emotional eating. 2013 Humaine Association Conference of Affective Computing and Intelligent Interaction. Geneva, Switzerland.

Chauliac, M; Catrysse, L. ; Gijbels, D. & Donche V. (2020). It is all in the surv-eye: can eye tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies? Frontline Learning Research. 8 (3), 26 – 39. https://doi.org/10.14786/flr.v8i3.489

Chung, H., Kim, J., Park, R., & Jean, H. (2018). The impact of sample size in cross-classified multiple membership multilevel models. Journal of Modern Applied Statistical Methods, 17 (1), Article 26. https://doi.org/10.22237/jmasm/1542209860

Cole, J. S., Bergin, D. A., & Whittaker, T. A. (2008). Predicting student achievement for low stakes tests with effort and task value. Contemporary Educational Psychology, 33, 609–624. https://doi.org/10.1016/j.cedpsych.2007.10.002

Corno, L. (2008). On teaching adaptively. Educational Psychologist, 43(3), 161–173. https://doi.org/10.1080/00461520802178466

Creswell, J. W. & Guetterman, T. C. (2019). Educational research: Planning, conducting, and evaluating quantitative and qualitative research, 6th edition, Pearson.

Dietrich, J., Viljaranta, J., Moeller, J., & Kracke, B. (2017). Situational expectancies and task values: Associations with students' effort. Learning and Instruction, 47, 53–64. https://doi.org/10.1016/j.learninstruc.2016.10.009

Dietrich, J., Moeller, J., Guo, J., Viljaranta, J., & Kracke, B. (2019a). In-the-moment profiles of expectancies, task values, and costs. Frontiers in Psychology, 10:1662. https://doi.org/10.3389/fpsyg.2019.01662

Durik, A. M. & Jenkins J. S. (2020). Variability in Certainty of Self-Reported Interest: Implications for Theory and Research. Frontline Learning Research. 8 (3) 85-103. https://doi.org/10.14786/flr.v8i3.491

Douglas, H., (2011). Facts, values, and objectivity. In: I. Jarvie & J. Zamora Bonilla (eds.), The SAGE Handbook of Philosophy of Social Science, 513–529, London: SAGE Publications.

Durik, A. M., Vida, M., & Eccles, J. S. (2006). Task values and ability beliefs as predictors of high school literacy choices: A developmental analysis. Journal of Educational Psychology, 98, 382–393. https://doi.org/10.1037/0022-0663.98.2.382

Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives (pp.74–146). San Francisco, CA: Freeman.

Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153

Efron, B. and Tibshirani, R. (1993) An Introduction to the Bootstrap. Chapman and Hall, New York, London.

Eisner, E. (1992). Objectivity in educational research. Curriculum Inquiry, 22(1), 9-15. https://doi.org/10.1080/03626784.1992.11075389

Enders, C. K. (2010). Applied missing data analysis. New York, NY: The Guilford Press.

Fahrenberg, J. (1996). Ambulatory assessment: Issues and perspectives. In: Fahrenberg, J. & Myrtek, M. (Eds.). (1996). Ambulatory Assessment: Computer-assisted Psychological and Psychophysiological Methods in Monitoring and Field Studies (pp. 3 – 20). Seattle, WA: Hogrefe and Huber. University of Freiburg i. Br., Germany

Fink, B. (1991). Interest development as structural change in person-object relationships. In: Oppenheimer L., Valsiner J. (eds) The Origins of Action. Springer, New York, NY. https://doi.org/10.1007

Firstbeat (2012). Heart beat based recovery analysis for athletic training. Firstbeat Whitepapers. Retrieved from: http://www.firstbeat.fi/physiology/white-papers

Fisher W. P. Jr. (2000). Objectivity in psychosocial measurement: what, why, how. Journal of Outcome Measurement, 4(2), 527-563.

Fryer, L. K. & Nakao K. (2020). The Future of Survey Self-report: An experiment contrasting Likert, VAS, Slide, and Swipe touch interfaces. Frontline Learning Research, 8 (3),10-25. https://doi.org/10.14786/flr.v8i3.501

Glanzberg, M. (2018). Truth. In: Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy (Fall 2018 Edition), Retrieved from https://plato.stanford.edu/archives/fall2018/entries/truth/

Göllner, R., Wagner, W., Eccles, J. S., & Trautwein, U. (2018). Students’ idiosyncratic perceptions of teaching quality in mathematics: A result of rater tendency alone or an expression of dyadic effects between students and teachers? Journal of Educational Psychology, 110(5), 709–725. https://doi.org/10.1037/edu0000236

Goetz, T., Bieg, M., Lüdtke, O., Pekrun, R., & Hall, N. C. (2013). Do girls really experience more anxiety in mathematics? Psychological Science, 24(10), 2079-2087. https://doi.org/10.1177/0956797613486989

Green, A. S., Rafaeli, E., Bolger, N., Shrout, P. E., & Reis, H. T. (2006). Paper or plastic? Data equivalence in paper and electronic diaries. Psychological Methods, 11, 87–105. https://doi.org/10.1037/1082-989X.11.1.87

Hartigan, J. A., & Hartigan, P. M. (1985) The dip test of unimodality. Annals of Statistics, 13, 70–84.

Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method. Measuring the quality of everyday life. Thousand Oaks, CA, US: Sage Publications.

Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41, 111-127. https://doi.org/10.1207/s15326985ep4102_4

Ketonen, E., Dietrich, J., Moeller, J., Salmela-Aro, K., & Lonka, K. (2018). The influence of autonomous and controlled daily goals on positive and negative emotional states: An experience sampling approach. Learning and Instruction, 53, 10-20. https://doi.org/10.1016/j.learninstruc.2017.07.003

Krapp, A. (1998). Entwicklung und Förderung von Interessen im Unterricht [Development and promotion of interest in instruction]. Psychologie in Erziehung und Unterricht, 45, 186-203.

Krapp, A. (2002). Structural and dynamic aspects of interest development: theoretical considerations from an ontogenetic perspective. Learning and Instruction, 12(4), 383-409. https://doi.org/10.1016/S0959-4752(01)00011-1

Krapp, A., & Fink, B. (1992). The development and function of interests during the critical transition from home to preschool. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 397–429). Hillsdale, NJ: Lawrence Erlbaum Associates.

Krapp, A., Hidi, S., & Renninger, K. A. (1992). Interest, learning and development. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 3–25). Hillsdale, NJ: Lawrence Erlbaum Associates.

Krothapalli, K. S. & Koolagudi, S. G. (2013). Emotion recognition using speech features. London: Springer

Lüdtke, O., Robitzsch, A., Trautwein, U., Kunter. M. (2009). Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology 34, 120–131. https://doi.org/10.1016/j.cedpsych.2008.12.001

Maechler, M. (2016). Package ‘diptest’. Hartigan's dip test statistic for unimodality – Corrected. R package. Retrieved March 5, 2020 from https://cran.r-project.org/web/packages/diptest/diptest.pdf

Moeller, J., Dietrich, J., Viljaranta, J., & Kracke, B. (2019). Data, R and Mplus codes for disentangling objective characteristics of learning situations from subjective perceptions thereof, using an experience sampling method design. Rerieved from https://osf.io/yszvm/. https://doi.org/10.17605/OSF.IO/YSZVM

Moeller, J., Ivcevic, Z., White, A., & Brackett, M. A. (2018). Mixed emotions: network analyses of intra-individual co-occurrences within and across situations. Emotion,18(8), 1106-1121. https://doi.org/10.1037/emo0000419

Popper, K. R. (1934 [2002]), Logik der Forschung [The Logic of Scientific Discovery], Berlin: Akademie Verlag.

Prenzel, M., Krapp, A. & Schiefele, H. (1986). Grundzüge einer pädagogischen Interessentheorie [Outline of an educational interest theory]. Zeitschrift für Pädagogik, 32(2), 163-173.

Reitzle, M. & Dietrich, J. (2019). From between-person statstics to within-person dynamics. Diskurs Kindheits- und Jugendforschung, 3-2019, 319-339. https://doi.org/10.3224/diskurs.v14i3.06

Schmitz, B. & Perels, F. (2011). Self-monitoring of self-regulation during math homework behaviour using standardized diaries. Metacognition & Learning, 6, 255-273. https://doi.org/10.1007/s11409-011-9076-6

Schönbrodt, F. D. & Perugini, M. (2013). At what sampe size do correlations stabilize? Journal of Research on Personality, 47, 609-612. https://doi.org/10.1016/j.jrp.2013.05.009

Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1-32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415

Silge, J. & Robinson, D. (2017). Text mining with R: A tidy approach. Sebastopol, CA: O’Reilly

Takarangi, M. K. T., Garry, M., & Loftus, E. F. (2006). Dear diary, is plastic better than paper? I can’t remember: Comment on Green, Rafaeli, Bolger, Shrout, and Reis (2006). Psychological Methods, 11, 119 –122. https://doi.org/10.1037/1082-989X.11.1.119

Tibshirani, R. & Leisch, F. (2019). bootstrap: Functions for the Book "An Introduction to the Bootstrap. R package. https://cran.r-project.org/web/packages/bootstrap/index.html

Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., Wilke, C., & Woo, K. (n.d.). Jittered points. Retrieved from: https://ggplot2.tidyverse.org/reference/geom_jitter.html

Wickham, H. (2016). Ggplot2: Elegant graphics for data analysis. Springer-Verlag New York.