Prospects and pitfalls in combining eye tracking data and verbal reports
Main Article Content
Abstract
It is intuitively appealing to try to combine eye-tracking data and verbal reports when investigating medical image interpretation. However, before collecting such data, important decisions have to be made including exactly when and how to collect the verbal reports. The purpose of this methodological article is to reflect upon the pros and cons of different solutions and to offer some guidelines to investigators. We start by exploring the ontology of vision and speech production and the epistemology of eye movements to grasp what fixations and verbal reports actually reflect. We are also interested in the major constraints of the two systems. Second, we elaborate on two dominant investigational approaches to verbal accounts, namely concurrent think-aloud and Chi’s explanations, and move on to other approaches. Third, we present and critically evaluate studies from the literature on medical image interpretation that have sought to contrast or integrate eye movement data and verbal reports. Fourth, we conclude with some practical guidelines and suggestions for further research.
Article Details
How to Cite
Helle, L. (2017). Prospects and pitfalls in combining eye tracking data and verbal reports. Frontline Learning Research, 5(3), 81–93. https://doi.org/10.14786/flr.v5i3.254
Section
Articles
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Al-Moteri, M. O., Symmons, M., Plummer, V., & Cooper, S. (2017). Eye tracking to investigate cue processing in medical decision-making: A scoping review. Computers in Human Behavior, 66, 52-66.
Azevedo, R., Faremo, S., & Lajoie, S. P. (2007, January). Expert-novice differences in mammogram interpretation. In Proceedings of the Cognitive Science Society (Vol. 29, No. 29).
Balslev, T., Jarodzka, H., Holmqvist, K., de Grave, W., Muijtjens, A. M., Eika, B., ... & Scherpbier, A. J. (2012). Visual expertise in paediatric neurology. European journal of paediatric neurology, 16(2), 161-166.
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Caldwell, R. T., & Krupinski, E. A. (2000). Role of faulty decision making in the satisfaction of search effect in chest radiography. Academic Radiology, 7(12), 1098-1106.
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Miller, E. M., Caldwell, R. T., Kuehn, D. M., & Berbaum, M. L. (1998). Role of faulty visual search in the satisfaction of search effect in chest radiography. Academic radiology, 5(1), 9-19.
Bertram, R., Helle, L., Kaakinen, J. K., & Svedström, E. (2013). The effect of expertise on eye movement behaviour in medical image perception. PloS one, 8(6), e66169.
Blondon, K., Wipfli, R., & Lovis, C. (2015). Use of eye-tracking technology in clinical reasoning: a systematic review. Studies in Health Technolology and Informatics, 210, 90-94.
Chi, M. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), 271-315.
Cowan, N. (2010). The magical mystery four how is working memory capacity limited, and why?. Current directions in psychological science, 19(1), 51-57.
Ericsson, K. A. (2006). Protocol analysis and expert thought: concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich & R. R. Hoffman (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 223-241). Cambridge University Press.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological review, 87(3), 215.
Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis. Verbal Reports as Data. MIT press.
Gegenfurtner, A., Kok, E., Geel, K., Bruin, A., Jarodzka, H., Szulewski, A., & Merriënboer, J. J. (2017). The challenges of studying visual expertise in medical image diagnosis. Medical Education, 51(1), 97-104.
Gegenfurtner, A., Siewiorek, A., Lehtinen, E., & Säljö, R. (2013). Assessing the quality of expertise differences in the comprehension of medical visualizations. Vocations and Learning, 6(1), 37-54.
Goodale, M. A. (1998). Visuomotor control: Where does vision end and action begin?. Current Biology, 8(14), R489-R491.
Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in neurosciences, 15(1), 20-25.
Hertzum, M., Hansen, K. D., & Andersen, H. H. (2009). Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?. Behaviour & Information Technology, 28(2), 165-181.
Jaarsma, T., Jarodzka, H., Nap, M., Merrienboer, J. J., & Boshuizen, H. (2014). Expertise under the microscope: Processing histopathological slides. Medical education, 48(3), 292-300.
Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the beholder: How experts and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146-154.
Kundel, H. L., Nodine, C. F., & Carmody, D. (1978). Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investigative radiology, 13(3), 175-181.
Kirchner, H., & Thorpe, S. J. (2006). Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited. Vision research, 46(11), 1762-1776.
Krupinski, E. A., Tillack, A. A., Richter, L., Henderson, J. T., Bhattacharyya, A. K., Scott, K. M., Graham, A.R., Descour, M. R., Davis, J. R., & Weinstein, R. S. (2006). Eye-movement study and human performance using telepathology virtual slides. Implications for medical education and differences with experience. Human pathology, 37(12), 1543-1556.
Lesgold, A. M., Feltovich, P. J., Glaser, R., & Wang, Y. (1981). The Acquisition of Perceptual Diagnostic Skill in Radiology (No. LRDC-81/PDS-1). PITTSBURGH UNIV PA LEARNING RESEARCH AND DEVELOPMENT CENTER.
Levelt, W. J. M. (1989). Speaking. From intention to articulation. Cambridge MA: MIT press.
Levelt, W. J., Roelofs, A., & Meyer, A. S. (1999). A theory of lexical access in speech production. Behavioral and brain sciences, 22(01), 1-38.
Li, R., Pelz, J., Shi, P., Alm, C. O., & Haake, A. R. (2012, March). Learning eye movement patterns for characterization of perceptual expertise. In Proceedings of the Symposium on Eye Tracking Research and Applications (pp. 393-396). ACM.
Littlefair, S., Brennan, P., Reed, W., Williams, M., & Pietrzyk, M. W. (2012, February). Does the thinking aloud condition affect the search for pulmonary nodules? In SPIE Medical Imaging (pp. 83181A-83181A). International Society for Optics and Photonics.
Morita, J., Miwa, K., Kitasaka, T., Mori, K., Suenaga, Y., Iwano, S., ... & Ishigaki, T. (2008). Interactions of perceptual and conceptual processing: Expertise in medical image diagnosis. International journal of human-computer studies, 66(5), 370-390.
Nodine, C. F., & Kundel, H. L. (1987). Using eye movements to study visual search and to improve tumor detection. Radiographics, 7(6), 1241-1250.
Patrick, J., & James, N. (2004). Process tracing of complex cognitive work tasks. Journal of occupational and organizational psychology, 77(2), 259-280.
Rubin, G. D., Roos, J. E., Tall, M., Harrawood, B., Bag, S., Ly, D. L., ... & Roy Choudhury, K. (2014). Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking. Radiology, 274(1), 276-286.
Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & cognition, 17(6), 759-769.
Salthouse, T. A., & Ellis, C. L. (1980). Determinants of eye-fixation duration. The American journal of psychology, 207-234.
van der Gijp, A., Ravesloot, C. J., Jarodzka, H., van der Schaaf, M. F., van der Schaaf, I. C., van Schaik, J. P. J., & ten Cate, T. J. (2016). How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education, 1-23.
van der Gijp, A., Ravesloot, C. J., van der Schaaf, M. F., van der Schaaf, I. C., Huige, J. C., Vincken, K. L., ... & van Schaik, J. P. (2015). Volumetric and two-dimensional image interpretation show different cognitive processes in learners. Academic radiology, 22(5), 632-639.
van der Gijp, A., van der Schaaf, M. F., Van der Schaaf, I. C., Huige, J. C. B. M., Ravesloot, C. J., van Schaik, J. P. J., & ten Cate, T. J. (2014). Interpretation of radiological images: towards a framework of knowledge and skills. Advances in Health Sciences Education, 19(4), 565-580.
Van Gog, T., Paas, F., Van Merriënboer, J. J., & Witte, P. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237.
Vanni, S. (2004). Näkötiedon käsittely aivokuoressa [Processing of visual data in the cerebral cortex]. Duodecim, 120, 2653-2662.
Vanni, S., & Heikkinen, H. (2015). Onko aivoissamme käyttämätöntä kapasiteettia? [Is there unused capacity in our brain?] Duodecim, 131, 1644-1649.
Wolfe, J. M., Evans, K. K., Drew, T., Aizenman, A., & Josephs, E. (2015). How do radiologists use the human search engine?. Radiation protection dosimetry, ncv501.
Wolfe, J. M., Võ, M. L. H., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends in cognitive sciences, 15(2), 77-84.
Azevedo, R., Faremo, S., & Lajoie, S. P. (2007, January). Expert-novice differences in mammogram interpretation. In Proceedings of the Cognitive Science Society (Vol. 29, No. 29).
Balslev, T., Jarodzka, H., Holmqvist, K., de Grave, W., Muijtjens, A. M., Eika, B., ... & Scherpbier, A. J. (2012). Visual expertise in paediatric neurology. European journal of paediatric neurology, 16(2), 161-166.
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Caldwell, R. T., & Krupinski, E. A. (2000). Role of faulty decision making in the satisfaction of search effect in chest radiography. Academic Radiology, 7(12), 1098-1106.
Berbaum, K. S., Franken, E. A., Dorfman, D. D., Miller, E. M., Caldwell, R. T., Kuehn, D. M., & Berbaum, M. L. (1998). Role of faulty visual search in the satisfaction of search effect in chest radiography. Academic radiology, 5(1), 9-19.
Bertram, R., Helle, L., Kaakinen, J. K., & Svedström, E. (2013). The effect of expertise on eye movement behaviour in medical image perception. PloS one, 8(6), e66169.
Blondon, K., Wipfli, R., & Lovis, C. (2015). Use of eye-tracking technology in clinical reasoning: a systematic review. Studies in Health Technolology and Informatics, 210, 90-94.
Chi, M. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), 271-315.
Cowan, N. (2010). The magical mystery four how is working memory capacity limited, and why?. Current directions in psychological science, 19(1), 51-57.
Ericsson, K. A. (2006). Protocol analysis and expert thought: concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich & R. R. Hoffman (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 223-241). Cambridge University Press.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological review, 87(3), 215.
Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis. Verbal Reports as Data. MIT press.
Gegenfurtner, A., Kok, E., Geel, K., Bruin, A., Jarodzka, H., Szulewski, A., & Merriënboer, J. J. (2017). The challenges of studying visual expertise in medical image diagnosis. Medical Education, 51(1), 97-104.
Gegenfurtner, A., Siewiorek, A., Lehtinen, E., & Säljö, R. (2013). Assessing the quality of expertise differences in the comprehension of medical visualizations. Vocations and Learning, 6(1), 37-54.
Goodale, M. A. (1998). Visuomotor control: Where does vision end and action begin?. Current Biology, 8(14), R489-R491.
Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in neurosciences, 15(1), 20-25.
Hertzum, M., Hansen, K. D., & Andersen, H. H. (2009). Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?. Behaviour & Information Technology, 28(2), 165-181.
Jaarsma, T., Jarodzka, H., Nap, M., Merrienboer, J. J., & Boshuizen, H. (2014). Expertise under the microscope: Processing histopathological slides. Medical education, 48(3), 292-300.
Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the beholder: How experts and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146-154.
Kundel, H. L., Nodine, C. F., & Carmody, D. (1978). Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investigative radiology, 13(3), 175-181.
Kirchner, H., & Thorpe, S. J. (2006). Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited. Vision research, 46(11), 1762-1776.
Krupinski, E. A., Tillack, A. A., Richter, L., Henderson, J. T., Bhattacharyya, A. K., Scott, K. M., Graham, A.R., Descour, M. R., Davis, J. R., & Weinstein, R. S. (2006). Eye-movement study and human performance using telepathology virtual slides. Implications for medical education and differences with experience. Human pathology, 37(12), 1543-1556.
Lesgold, A. M., Feltovich, P. J., Glaser, R., & Wang, Y. (1981). The Acquisition of Perceptual Diagnostic Skill in Radiology (No. LRDC-81/PDS-1). PITTSBURGH UNIV PA LEARNING RESEARCH AND DEVELOPMENT CENTER.
Levelt, W. J. M. (1989). Speaking. From intention to articulation. Cambridge MA: MIT press.
Levelt, W. J., Roelofs, A., & Meyer, A. S. (1999). A theory of lexical access in speech production. Behavioral and brain sciences, 22(01), 1-38.
Li, R., Pelz, J., Shi, P., Alm, C. O., & Haake, A. R. (2012, March). Learning eye movement patterns for characterization of perceptual expertise. In Proceedings of the Symposium on Eye Tracking Research and Applications (pp. 393-396). ACM.
Littlefair, S., Brennan, P., Reed, W., Williams, M., & Pietrzyk, M. W. (2012, February). Does the thinking aloud condition affect the search for pulmonary nodules? In SPIE Medical Imaging (pp. 83181A-83181A). International Society for Optics and Photonics.
Morita, J., Miwa, K., Kitasaka, T., Mori, K., Suenaga, Y., Iwano, S., ... & Ishigaki, T. (2008). Interactions of perceptual and conceptual processing: Expertise in medical image diagnosis. International journal of human-computer studies, 66(5), 370-390.
Nodine, C. F., & Kundel, H. L. (1987). Using eye movements to study visual search and to improve tumor detection. Radiographics, 7(6), 1241-1250.
Patrick, J., & James, N. (2004). Process tracing of complex cognitive work tasks. Journal of occupational and organizational psychology, 77(2), 259-280.
Rubin, G. D., Roos, J. E., Tall, M., Harrawood, B., Bag, S., Ly, D. L., ... & Roy Choudhury, K. (2014). Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking. Radiology, 274(1), 276-286.
Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & cognition, 17(6), 759-769.
Salthouse, T. A., & Ellis, C. L. (1980). Determinants of eye-fixation duration. The American journal of psychology, 207-234.
van der Gijp, A., Ravesloot, C. J., Jarodzka, H., van der Schaaf, M. F., van der Schaaf, I. C., van Schaik, J. P. J., & ten Cate, T. J. (2016). How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education, 1-23.
van der Gijp, A., Ravesloot, C. J., van der Schaaf, M. F., van der Schaaf, I. C., Huige, J. C., Vincken, K. L., ... & van Schaik, J. P. (2015). Volumetric and two-dimensional image interpretation show different cognitive processes in learners. Academic radiology, 22(5), 632-639.
van der Gijp, A., van der Schaaf, M. F., Van der Schaaf, I. C., Huige, J. C. B. M., Ravesloot, C. J., van Schaik, J. P. J., & ten Cate, T. J. (2014). Interpretation of radiological images: towards a framework of knowledge and skills. Advances in Health Sciences Education, 19(4), 565-580.
Van Gog, T., Paas, F., Van Merriënboer, J. J., & Witte, P. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237.
Vanni, S. (2004). Näkötiedon käsittely aivokuoressa [Processing of visual data in the cerebral cortex]. Duodecim, 120, 2653-2662.
Vanni, S., & Heikkinen, H. (2015). Onko aivoissamme käyttämätöntä kapasiteettia? [Is there unused capacity in our brain?] Duodecim, 131, 1644-1649.
Wolfe, J. M., Evans, K. K., Drew, T., Aizenman, A., & Josephs, E. (2015). How do radiologists use the human search engine?. Radiation protection dosimetry, ncv501.
Wolfe, J. M., Võ, M. L. H., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and nonselective pathways. Trends in cognitive sciences, 15(2), 77-84.