Frontline Learning Research Vol.8 No. 1 (2020) 76
- 95
ISSN 2295-3159
aBehavioural Science Institute,
Radboud University, The Netherlands
bInstructional Science, Twente University, The
Netherlands
Article received 9 April 2019/ revised 30 January / accepted 18 February / available online 26 February
This study investigated macro-level navigation patterns in children’s hypermedia learning, and how they related to task structure and learning outcomes. For this purpose, 5th and 6th grade learners performed a hypermedia assignment in which a high (n=57) versus a low (n=54) level of structure was provided. By means of qualitative analyses of their navigation activities, 6 macro-level navigation patterns were distinguished: linear reading, selective reading, video viewing, massed writing, late onset writing, and unpredictable reading. Results showed that the linear reading pattern was more frequent in the high structure environment, and that both the high structure environment and the linear reading pattern were associated with the highest quality of the children’s written assignments. Navigation patterns and task structure did not clearly predict children’s declarative knowledge gains or knowledge transfer. These findings show that there are multiple ways to navigate through a hypermedia environment, but that these are not all equally successful for learning. Moreover, the provided task structure in the environment may affect the occurrence of successful navigation patterns.
Keywords: Navigation activities; task structure; hypermedia; primary education
In order to be able to participate in today’s society, primary school children are taught multiple skills that are deemed crucial in the 21st century (Rotherham & Willingham, 2010). One of these skills involves the adequate use of online materials that are found on the Internet (Ananiadou & Claro, 2009). Children sometimes struggle to learn on the Internet, however, and learners may learn and comprehend less in digital environments (Delgado, Vargas, Ackerman, & Salmeron, 2018; Kong, Seo, & Zhai, 2018). On top of that, hypermedia environments (such as the Internet), which combine hypertext with multimedia, may result in disorientation and distraction (Scheiter & Gerjets, 2007). Providing external support within the environment, by structuring the learning material, can improve learning outcomes, depending on the needs of the learner (Goldman, 2009). These findings show that individual variation exists in hypermedia learning outcomes, and is affected by task structure. In addition, these findings show there also exists individual variation in the learning process (i.e., whatever happens during learning). As such, investigating children’s learning processes may help to explain individual differences in learning outcomes and their relation with task structure. This, in turn, can be used to develop adequate support for weaker learners. However, when studying the learning process, for example, by studying how learners navigate through a website, interpreting individual events is problematic (see e.g., Salmeron, Naumann, García, & Fajardo, 2017). Consequently, a macro-level view of their learning process, which looks at overall patterns of events, may aid interpretation. Research that takes a macro-level view is largely lacking, however. Hence, it is unclear which macro-level patterns can be distinguished in children. The current study therefore investigated which macro-level navigation patterns could be distinguished in children, and how these patterns were related to task structure and learning outcomes.
Hypermedia environments provide learners with the flexibility to navigate through their learning environment in many different ways. An advantage of this, is that learners can adapt their learning process to their own needs (Gorissen, Kester, Brand-Gruwel, & Martens, 2015; Scheiter & Gerjets, 2007). However, not all leaners succeed in learning in a hypermedia environment. The hypertext structure may create high cognitive load (DeStefano & LeFevre, 2007), result in disorientation (McDonald & Stevenson, 1996), or result in gaps in comprehension (Bezdan, Kester, & Kirschner, 2013). Moreover, because of the inherent flexibility of hypermedia environments, learners need to regulate their own learning (Azevedo & Cromley, 2004).
The field of self-regulated learning (SRL) focusses on how children achieve learning goals, by using cognitive activities to study the learning material, metacognitive activities to monitor and control their progress, and motivation to keep engaged with the material (Winne & Hadwin, 1998; Winne & Nesbit, 2010). The information processing view of SRL (Winne & Hadwin, 1998) describes SRL as a cyclical process in which the learner moves through four phases, namely task definition, goal setting and planning, enacting study tactics and strategies, and metacognitively adapting studying. The learner can move through these phases in a non-linear fashion, and may go through them multiple times within a learning assignment (Azevedo, 2009; Molenaar & Järvelä, 2014). SRL, and metacognitive activities in particular, have been related to better learning outcomes (e.g., Eilam & Aharon, 2003; van der Stel & Veenman, 2008). Meanwhile, however, children often do not spontaneously regulate their learning (de Jong & van Joolingen, 1998), which may partly explain their difficulties in hypermedia learning.
One way to support the regulation of learning in hypermedia environments is by providing external structure or support. Empirically, research has investigated the effect of different levels of task structure in the learning environment on both the learning process and learning outcomes. Such research included comparisons between linear and non-linear text (Blom, Segers, Knoors, Hermans, & Verhoeven, 2018; Klois, Segers, & Verhoeven, 2013; McDonald & Stevenson, 1996), the effect of advance organizers (Urakami & Krems, 2012), the presence of link suggestions (Ignacio Madrid, Van Oostendorp, & Puerta Melguizo, 2009), the specificity of problem solving goals (Künsting, Wirth, & Paas, 2011), and the specificity of questions (Rouet, 2003). These studies showed that providing more structure does not universally lead to better learning outcomes. Instead, the provided structure needs to fit the regulatory needs of the learner (Goldman, 2009; Kalyuga, Ayres, Chandler, & Sweller, 2003). Furthermore, while it has been suggested that learners adapt to the requirements of a task, this adaptation does not always lead to better performance (Pieschl, Stahl, Murray, & Bromme, 2012).
Also on a theoretical level, SRL research has emphasized the importance of taking into account the learning context when studying the effect of SRL on learning outcomes (Boekaerts, 1999; Greene & Azevedo, 2010; Zimmerman, 1989). In the information processing model of SRL, the learning context is incorporated in the notion of task conditions (Winne & Hadwin, 1998), which refer to task characteristics such as provided instructional cues, or the available time. These task conditions affect which cognitive and metacognitive activities are performed during learning. As such, both empirically and theoretically, differences in task structure have been suggested to result in differences in the learning process (e.g., Bezdan et al., 2013; Ignacio Madrid et al., 2009; Mobrand & Spyridakis, 2007). Therefore, investigating the learning process may help to understand how task structure affects learning outcomes.
As described above, one way to illuminate why children differ in their hypermedia learning outcomes, and how task structure affects these outcomes, is by looking at their learning process (see e.g., Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Malmberg, Järvenoja, & Järvelä, 2013; Salmeron et al., 2017). One aspect of the learning process is how children navigate through the hypermedia environment. Log-files provide a record of children’s navigation activities while they occur (Greene & Azevedo, 2010). Such online measurement provides a more in-depth way of mapping the learning process than off-line measures, such as questionnaires, without having to rely on the memory of, or introspection by, the learner. Log-files have the advantage over other online measures that they are relatively easy to collect at a large scale, are not intrusive, and do not disrupt the learning process (Veenman, 2015; Winne, Hadwin, & Gress, 2010). They may also be more suitable than, for example, think aloud protocols when studying children, because they do not rely on the learner’s ability to verbalize what they are thinking (see also Veenman, 2011).
While navigation activities may be useful for gaining insight into children’s learning processes, interpreting individual navigation events is not straightforward. For example, a reading time of a web-page may indicate that a child scanned the entire page, or that he or she read part of it more thoroughly (Salmeron et al., 2017). In addition, the effect of individual navigation events on learning outcomes may depend on how the learner followed up on the event. For example, if a learner read only half of one page, the effect on learning outcomes may depend on whether or not the learner already knew the content of the page, and on whether he or she reread the page at a later stage. Consequently, a more holistic approach (see also Reimann, 2009) to investigating navigation activities may help to interpret individual navigation events. As such, looking at macro-level navigation patterns, rather than individual navigation events, may help with the interpretation of log-file data. We define macro-level navigation patterns as the overall approach a learner takes to navigate through a hypermedia assignment.
At the macro-level, several studies have shown that different learning processes can be observed in (groups of) learners who learn under different task conditions (Bannert, Reimann, & Sonnenberg, 2014; Sobocinski, Malmberg, & Järvelä, 2017; Sonnenberg & Bannert, 2015), or who have different learning outcomes (Schoor & Bannert, 2012). As learners gain experience in a task, they may approach the task differently (see e.g., expertise reversal effect; Kalyuga et al., 2003). In addition, their approach may change depending on task conditions (Pieschl et al., 2012). Hence, instead of being stable across time, navigation patterns are likely to be inherently dynamic and variable, even within one individual. Together, these findings suggest that children may move through a hypermedia assignment in multiple ways. It is still unclear, however, which macro-level navigation patterns can be distinguished in children.
While research is scarce, some studies have investigated macro-level navigation patterns, by means of cluster analysis. MacGregor (1999) found three types of navigation patterns in their study on 7th and 11th grade students who used an instructional hypermedia system. First, sequential studiers accessed information in a sequential manner. They moved slowly and methodically through the pages, with an emphasis on the textual components. Second, video viewers primarily watched videos. Last, concept connectors access the information more selectively. This selection appeared to be based on prior knowledge. In addition, they alternated between skimming text and reading thoroughly.
Similarly, Lawless and Kulikowich (1996) also found three macro-level navigation patterns in their study on hypertext learning in undergraduate students who learned about Lyme disease, which they referred to as feature explorers, who invested more time in understanding the environment than learning information; knowledge seekers, who had high scores on the outcome measures; and apathetic hypertext users, who spent little time in the environment, used few special features, and inspected few pages. However, Barab, Bowdish, and Lawless (1997) found four patterns in undergraduates who performed a hypermedia information retrieval task. First, model users chose the simpler information retrieval task and provided the correct solution without deviating to irrelevant pages. Second, disenchanted volunteers had low retrieval scores and explored very little of the environment. Third, feature explorers used the help screens, watched many videos, and had low retrieval scores. Finally, cyber cartographers spent a lot of time in the environment and explored many of the pages. While these studies show some similarities in the clusters they found, there is no complete overlap. One potential reason for this difference is that the measures that were used for extracting the clusters differed between the studies. In fact, many studies that use navigation activities to understand the learning process, do not state reasons for their choice concerning which variables they extract from the log-files. As a consequence, it is unclear which navigation measures reflect the most dominant differences between groups of learners. One solution to this problem is to take a qualitative approach, in which macro-level navigation patterns are clustered based on their visual appearance. This way, the most dominant features of a pattern become salient. This is the approach we took in this study.
Summarizing, investigating children’s navigation activities may
help to explain how task structure affects hypermedia learning
outcomes. Because the meaning of individual navigation events is
not self-evident and depends on the context in which they occur,
the interpretation of children’s navigation activities may benefit
from analysing them in terms of macro-level navigation patterns.
Furthermore, although various clusters have been suggested for
older students, it is unclear which patterns children use during
hypermedia assignments. The present study therefore investigated
the effect of task structure on macro-level navigation patterns
and hypermedia learning outcomes in children. For this purpose,
three research questions were examined:
RQ1: Which macro-level navigation patterns can be distinguished in
children learning in a hypermedia assignment?
RQ2: Does the prevalence of these navigation patterns depend on
task structure?
RQ3: How do navigation patterns and task structure relate to
learning outcomes?
Participants were recruited via a letter to the school administrators. Active parental consent was obtained. In total, 111 5th and 6th grade children from five different primary schools in the Netherlands participated. Of these, 54 were boys, and 56 were girls (one child did not fill in his/her gender). Age ranged from 9 to 13 years (M = 11 years, 8.5 months, SD = 9.7 months). Participants in each classroom were randomly assigned to one of two conditions of the learning environment (see Task Structure Variation). Of the participants, 57 learned in a high structure environment, whereas 54 learned in a low structure environment. Participants in the two conditions did not differ in terms of their gender or age, p’s > .10.
2.2.1 The learning environment
In the present study, children learnt about the heart and living a healthy lifestyle by means of a WebQuest (Segers & Verhoeven, 2009). They were instructed to write a 300-word text that covered four topics: (1) the anatomy of the heart; (2) an explanation of the circulatory system; (3) the components of blood; and (4) how to keep the heart healthy. The assignment was done in a closed hypermedia environment. Nine preselected content pages, with links to four short videos were provided as resources. A schematic overview of such a content page is shown in Figure 1. Participants could not search the Internet for more information. On the left of the screen, a navigation menu was displayed, by means of which the children could access information pages about the assignment, a content overview from which they could access the resources, and a link to the worksheet in which they wrote their assignment. Participants could switch between reading the resources and writing. Whenever they saved their worksheet, they were informed of how many words they had written, and could return to the last page they accessed prior to the worksheet, without having to navigate through the content overview.
Figure 1. Schematic overview of a content page in the WebQuest environment
2.2.2 Task structure variation
Two versions of the hypermedia environment were created. Both versions contained exactly the same assignment and resources, but one provided high structure, whereas the other provided low structure. The two versions differed in the following three ways.
First, the information pages about the assignment differed. In the high structure condition, participants were provided with an introduction page; an assignment page in which the assignment was introduced; a roadmap page that broke the assignment down into smaller steps; a review page in which children could review their work; and a conclusion page. In the low structure condition, the road map page was not provided. Instead, their assignment page introduced the assignment, and contained three additional sentences about the topics that had to be included, and on how to navigate the website.
Second, the layout of the worksheet differed for the two conditions (see Figure 2). The high structure condition provided four text blocks with a header, one for each assignment topic. In the low structure condition, it showed only one large text block.
Finally, the layout of the content overview differed for the two conditions. In the high structure condition, the nine resources were grouped under four headers that roughly corresponded to the four assignment topics. In the low structure condition, these four headers were missing. The order of the resources was the same in both conditions, however.
Figure 2. Screenshot of the worksheet in the structured (left) and unstructured (right) learning environment.
2.2.3 Navigation activities
In order to measure children’s navigation activities, log-files were collected that registered every hyperlink selection with a time stamp with a one-second precision. In the current study, only information concerning their page views for resource pages and the worksheet was taken into account. Their use of the assignment pages was not investigated for two reasons. Frist, viewing frequencies of these pages were relatively low. On average, the assignment page was viewed 7.54 times (SD = 8.89; total viewing duration: M = 136.69 seconds; SD = 111.28); the roadmap page (for the high structure condition only) was viewed an average of 5.65 times (SD = 6.77; total viewing duration: M = 78.31 seconds; SD = 54.70); and finally, the review pages were viewed an average of 3.55 (SD = 4.16; total viewing duration: M = 54.29 seconds; SD = 94.49). Second, a previous study with a similar set-up showed that the mapping of assignment related pages to metacognitive activities is not straightforward, especially for learners with low learning gain (see also: Paans, Molenaar, Segers, & Verhoeven, 2019). Moreover, a few participants repeatedly accessed the worksheet via the assignment page or roadmap page, rather than the navigation menu, which further complicated the interpretation of page views for these assignment-related pages.
2.2.4 Assignment quality
The quality of the assignment, which participants wrote in the worksheet, was assessed. For this purpose, their assignment was rated by means of a predefined list of 50 terms, and explanations of terms or mechanisms. The number of used terms and explanations constituted their final score. Of the assignments, 24 (22%) were scored by a second rater. Interrater reliability was high (terms: intraclass r = .93; explanations: intraclass r = .93; total scale: intraclass r = .98).
2.2.5 Declarative knowledge
To measure declarative knowledge, participants had to connect 14 words to explanatory sentences. For example, they had to connect “white blood cell” to “helps fight disease”. They were given the same task before and after the hypermedia assignment, to measure their prior knowledge as well as their knowledge at post-test. The number of items correct constituted their score on the measure. Internal consistency of the measure at pre-test was relatively poor (α = .51), due to the fact that most participants had very low prior knowledge. Thus, many correct items might be attributed to chance. At post-test, internal consistency was good (α = .83).
2.2.6 Knowledge transfer
To measure knowledge transfer, participants were asked to design a birthday party that was fun and good for one’s heart. They were asked to name three activities and explain why they were healthy. They received one point for each correct activity and explanation. The maximum obtainable score was therefore six. Activities were scored based on their novelty. For example, eating three different fruits only received one point, whereas eating a fruit salad, building a tree hut, and playing hide and seek, counted as three activities. Explanations were scored on their accuracy. Inaccurate (e.g., candy is healthy because it contains sugar) or non-explanations (e.g., “it is healthy because it is healthy”) received no points; incomplete explanations (e.g., “it is healthy because you have to run a lot”) received half points; and complete explanations (e.g., “fruit is healthy because it contains vitamins”) received a full point. To assess the quality of the scoring, 24 transfer assignments (22%) were scored by a second rater. Inter-rater reliability was good (intraclass r = .79).
This study was part of a larger data collection on the effect of cognitive predictors, such as executive functions, prior knowledge, and non-verbal reasoning on children’s hypermedia learning. Only the parts relevant to the present study are described below.
In each classroom, the study started with a classroom session. This session started with a short explanation of the study, where we explained that not all children would receive the same layout of the website. Next, the children received the declarative knowledge pre-test. After making an example item, they were given 15 minutes to complete the test. They were encouraged to fill in what they knew and to guess the rest. Finally, they received a short presentation of the hypermedia assignment.
After the classroom session, the children participated in an individual session outside the classroom. This session included the hypermedia assignment, the declarative knowledge post-test, and the transfer task. During the hypermedia assignment, participants first received another short explanation of the hypermedia environment. Next, they were given 45 minutes to complete the task, and were reminded thrice of the remaining time. Any questions about the content were not answered, but about the use of the computer were. Whenever children indicated they were finished before all time had elapsed, they were asked to reread the assignment page and to re-evaluate whether or not they had finished. After the hypermedia assignment, participants were given the declarative knowledge post-test, for which they again received 15 minutes. Finally, they were given the transfer task.
Testing in each classroom took place within the course of two weeks. Sessions never followed each other directly, to prevent memory and fatigue effects. After participating, all children received a small gift. No classroom credits were rewarded.
2.4.1 Qualitative exploration of macro-level navigation patterns
In order to investigate the first research question, on which macro-level navigation patterns could be distinguished, a qualitative analysis of the children’s navigation patterns was performed. For this purpose, we took a grounded approach. In the first stage, the type and number of macro-level patterns was established. For this purpose, a line graph was made with time on the x-axis and web page on the y-axis for each individual’s navigation activities. These graphs were then grouped together based on similarities and differences in their appearance (see also Figure 3) by the first author. After an initial grouping, the groups were re-evaluated to see whether initial groups were overlapping and could be combined, or whether any additional patterns could be distilled. This led to a final set of macro-level navigation patterns, based on which coding criteria were formulated (see Appendix A). Finally, to check for grouping errors, all graphs were coded based on the final coding criteria.
In the second stage, reliability of the grouping was established by a second rater. This rater was trained on the navigation patterns of a different data collection and a random half of the data presented here. Reliability was assessed on the other random half of the current data. Inter-rater reliability was sufficient (κ = .73).
2.4.2 Assessing cluster differences between conditions
In order to investigate the second research questions, on whether the prevalence of the navigation patterns depends on task structure, a χ² analyses was performed. As a post-hoc comparison, column proportions with Bonferroni corrections were assessed to establish which clusters differed in their expected frequency between the two conditions.
2.4.3 Assessing the effect of macro-level navigation patterns and task structure on learning outcomes
In order to investigate the third research question, on how navigation patterns and task structure related to learning outcomes, three steps were taken. First, it was tested whether task structure predicted learning outcomes. For declarative knowledge, an ANCOVA was performed, with pre-test as the covariate. For assignment quality and knowledge transfer, two independent samples t-tests were performed.
Second, it was investigated to which extent cluster membership predicted learning outcomes. Because of the low sample size for some clusters, non-parametric tests were used. A Kruskall-Wallis test was performed, with cluster membership as the independent variable, and the learning outcomes as the dependent variables. Because the Kruskall-Wallis test does not allow for the use of a covariate, declarative knowledge gain was used as the outcome variable, instead of knowledge at post-test. Post-hoc comparisons were done using separate Mann-Whitney tests. For these post-hoc comparisons, a more conservative significance level of α = .01 was used.
Finally, it was investigated whether the effect of cluster on the learning outcomes was dependent on task structure. Again, because of the low sample size for some clusters, a regular 2x2 ANOVA could not be performed to test the interaction between task structure and cluster membership. Therefore, the two conditions were analysed separately with the same method as described above (Kruskall-Wallis tests for assessing cluster differences, and Mann-Whitney tests for post-hoc comparisons).
The first research question asked which macro-level navigation patterns can be distinguished in children. From a qualitative analysis of their navigation activities, six macro-level patterns emerged. These are described below. Figure 3 shows an example graph for each pattern. In each of the panels of Figure 3 (a through f), one of the navigation patterns is shown. The x-axis shows the time and the y-axis shows which page the participant is viewing at that particular time. Each page has its own horizontal row, with the bottom nine rows corresponding to the nine content pages in their order of presentation. That is, the bottom row corresponds to the resource page that was presented at the top of the page, the second row from the bottom corresponds to the second, and so forth. The next four rows correspond to the four video pages, and the top row corresponds to the worksheet. Solid lines indicate that a participant continuously viewed this page, whereas interrupted lines indicate that he or she switched between pages.
1. Linear reading: The first navigation pattern was characterized by a relatively linear reading pattern, where the children read most of the resource pages, and primarily in the order in which they were presented in the content overview. Figure 3a illustrates this by the stair-like pattern in the resource pages. Some participants, but not all, viewed videos as they encountered them (not shown in the example figure). Their writing was interleaved with their reading, which could be seen by frequent switches between the resources and the worksheet. In the figure, this is visible through the interrupted lines in both the resource pages and the worksheet. Mostly, their writing started relatively early on.
2. Selective reading: In this pattern, children selected up to three resource pages that they used for their writing activities. This can be seen in Figure 3b, where the participant spent most of his or her time on three content pages. Some participants, but not all, first quickly browsed through the content pages in their order, before starting to write. Once they started engaging with their selected resources, their writing was interleaved, with many switches between the resource page and their worksheet, which can be seen by the interrupted line for the worksheet page that runs along the entire x-axis.
3. Video viewing: In this pattern, children showed a preference for watching video material over written resources, as could be seen (Figure 3c) by the relative sparsity in time spent on written resources, and longer stretches on video pages. For most of these children, their writing was interleaved, with many switches between the videos and the worksheet.
4. Massed writing: Contrary to the previous three patterns, these children massed their writing activity. Their navigation patterns could be characterized by long stretches in the worksheet in which the children did not switch back to the resources. Figure 3d illustrates this pattern by showing one shorter and one longer solid line for the worksheet. This solid line indicates that the participant did not switch back to a resource page while writing.
5. Late onset writing: This navigation pattern was characterized by a relatively late first engagement with the worksheet. For example, for the participant shown in Figure 3e, a first line appears for the worksheet roughly 12 minutes into the assignment. The navigation activities started with browsing or reading through the resources. Some children, but not all, did this in a linear fashion. Figure 3e shows an example of this linear fashion through the stair-like pattern in the resource pages during the first 10 minutes of the assignment. After some initial time engaging with the resource pages, they started to engage with the worksheet. At this stage, some participants chose two or three resource pages with which they interleaved their writing in the worksheet. Others, however, started to navigate in a pattern that was more similar to that of massed writing, with long stretches in the worksheet, and little reference to the resource pages.
6. Unpredictable reading: In the last pattern, the children showed a relatively random pattern of navigating. Their patterns showed many switches between resources pages, and in a non-linear fashion. No resource page dominated their pattern, nor did it show a clear preference for videos or written resources. Writing was predominantly interleaved with viewing the resource pages. Figure 3f illustrates this, by showing a relatively diffuse pattern with many switches between the resource and video pages.
Figure 3. Examples for the six navigation patterns: linear reading (a), selective reading (b), video viewing (c), massed writing (d), late onset writing (e), and unpredictable reading (f). The x-axis shows the time throughout the assignment. The y-axis shows the page the learner is viewing at that particular time.
The second research question asked whether the prevalence of the navigation patterns depended on task structure. A comparison of the frequency of occurrence of the six clusters within the two conditions (Table 1), showed that cluster membership and condition were statistically non-independent, χ² (5) = 13.43, p = .020. A post-hoc comparison of the column proportions showed that only the frequency of the linear cluster differed for the two conditions at the α = .05 level, indicating that linear reading was more likely in the high structure condition than the low structure condition.
Table 1
Distribution of the cluster over the two conditions
The third research question asked how navigation patterns and task structure related to learning outcomes. It was first investigated how task structure predicted learning outcomes. The descriptive statistics of the various learning measures are shown in Table 21 and the correlations in Table 3. The two conditions did not differ in terms of their declarative knowledge at post-test, when controlling for pre-test scores, F(1, 104) = .06, p = .804. Additionally, the two conditions did not differ in terms of knowledge transfer, t(105) = .32, p = .751, but did differ in terms of assignment quality, t(106) = 2.51, p = .014, d = 0.48, CI: [.91; 7.82]. On average, assignment quality was 4.37 points higher in the high structure environment than the low structure environment.
Table 2
Descriptive statistics for learning measures per condition
Table 3
Correlations between the learning outcomes
In the second step to investigate the third research question, it was investigated whether macro-level navigation patterns predicted learning outcomes, and whether the effects were different in the two task conditions. Descriptive statistics of the learning outcomes per cluster are displayed in Table 4, for each condition separately.
First, the main effect of cluster membership on declarative knowledge gain, assignment quality, and knowledge transfer was assessed. Overall, clusters did not differ in terms of the resulting declarative knowledge gain, H(5) = 10.16, p = .071, or knowledge transfer, H(5) = 3.63, p = .604. They did differ in assignment quality, however, H(5) = 35.04, p < .001. Post-hoc comparisons showed that linear reading resulted in a higher assignment quality than video viewing (Z = 4.009, p < .001, r = .60), massed writing (Z = 4.483, p < .001, r = .63), and late onset writing (Z = 3.864, p < .001, r = .55). Selective reading and unpredictable reading also had higher assignment quality than massed writing (selective: Z = 2.765, p = .006, r = .50; unpredictable: Z = 3.105, p = .002, r = .54).
To test the condition*cluster interaction, each condition was analysed separately. In the high structure condition, clusters did not differ in terms of declarative knowledge gain, H(5) = 9.68, p = .085, or knowledge transfer, H(5) = 2.75, p = .739. They did differ in terms of assignment quality, H(5) = 13.27, p = .021. Post-hoc comparisons showed that linear reading had higher assignment quality than video viewing (Z = 2.832, p = .005, r = .50) and massed writing (Z = 3.108, p = .002, r = .56). No other comparisons were significant at the α = .01 level. For the low structure condition, clusters did not differ in terms of knowledge transfer, H(5) = 2.55, p = .768. They did differ, however, in assignment quality, H(5) = 21.78, p = .001, and declarative knowledge gain, H(5) = 11.31, p = .045. Linear reading predicted higher assignment quality than video viewing (Z = 2.789, p = .005, r = .77), massed writing (Z = 2.817, p = .005, r = .63), and late onset writing (Z = 3.316, p = .001, r = .74). In terms of declarative knowledge gain, no post-hoc comparisons were significant at the α = .01 level2.
Table 4
Descriptive statistics for learning outcomes by cluster and condition
This study investigated the effect of task structure on macro-level navigation patterns and hypermedia learning outcomes in children. Six macro-level navigation patterns could be distinguished. Results showed that the prevalence of these patterns depended on task structure. Both task structure and navigation patterns predicted the quality of the written assignment, but not learning gain. The effect of navigation patterns on assignment quality was similar in both task conditions.
The first result was that six macro-level navigation patterns could be distinguished. This is a larger number than reported in earlier studies (e.g., Barab et al., 1997; Lawless & Kulikowich, 1996), probably because we also incorporated the writing process. Consequently, we could distinguish between interleaved and massed writing, as well as the onset of writing. There are also similarities with previously found navigation patterns. Notably, most similarities are found with the study by MacGregor (1999), who’s study was most similar in terms of the used environment and the age of the participants. First, the linear reading pattern showed similarities with the sequential studiers, who went through the resources systematically and spent most time on text pages. Second, the video viewing pattern was similar to the video viewers MacGregor found. Finally, while there is no complete overlap, there appear to be similarities between our selective reading pattern and the previously found concept connectors, as both accessed information selectively, and both scanned some information, while reading other information more thoroughly.
The second result was that the provided task structure affected the occurrence of the macro-level navigation patterns. Linear reading was far more common in the high structure environment. As such, it appears that the provided structure affected the reading and writing patterns of the participants. This finding is coherent with research on SRL, which showed that interventions that aided regulation by means of prompts, changed the learning process of the learners (Sonnenberg & Bannert, 2015). Furthermore, prior research has shown that advance organizers, which provide structure to the learning content, can aid learners navigation and comprehension on the Internet (Urakami & Krems, 2012). Similarly, hypertext environments with simpler lay-outs (such as hierarchical hypertexts or linear texts) are easier to navigate by children than environments with a more complex lay-out (such as networked hypertexts; e.g., Blom et al., 2018). As such, the provided structure may have decreased any potential disorientation that the children were experiencing.
The last result was that both task structure and navigation patterns predicted the quality of the written assignment. The linear reading pattern resulted in the highest quality assignments, as did the high structure environment. Since the linear reading pattern was also far more prevalent in the high structure environment than in the low structure environment, it might be possible that the effect of task structure on assignment quality is driven by this difference in macro-level navigation patterns. At the level of individual navigation events, it has already been shown that task structure affected navigation (Ignacio Madrid et al., 2009; Mobrand & Spyridakis, 2007), and that navigation affected assignment quality (Paans, Segers, Molenaar, & Verhoeven, 2018). This study adds that the relation between task structure, navigation patterns and assignment quality, can also be found at the macro level. The provided task structure may have reduced the cognitive load the learners experienced (see e.g., Zumbach & Mohraz, 2008), by providing a scaffold for the regulation of their learning process. External regulation, for example, by means of a tutor, has been shown to improve hypertext learning outcomes (Azevedo, Greene, & Moos, 2007). At the same time, the linear reading pattern may have ensured that learners saw more of the content, and in a logical order, which in turn may have resulted in fewer gaps in their comprehension of the learning material (Salmeron, Kintsch, & Cañas, 2006a, 2006b). Besides the effects of task structure and navigation patterns on assignment quality, the low structure environment also showed an effect of navigation patterns on declarative knowledge gain. However, post-hoc comparisons showed only marginal effects, even though effect sizes were large. As such, we tentatively suggest that massed writing is likely to be related to the lowest knowledge gains. Indeed, in the massed writing pattern, learners spent long stretches of time in the worksheet, without referring back to the resources. Consequently, learners may not have spent enough time acquiring new knowledge or challenging misconceptions. These results, however, will need to be replicated in another study.
A limitation of the study is the sample size. Because no a priori expectation was formulated with regard to the number of clusters, or their frequency of occurrence, no reliable estimate could be made for the required sample size. In the current study, the frequency of some clusters was relatively low, especially when task structure was taken into account. Consequently, not all analyses could be performed due to lack of power. A future study may seek to replicate and extend the current findings with a larger sample.
A second limitation relates to the difference between the two task conditions. The two conditions differed in three ways: the presence of a roadmap page, the layout of the worksheet, and the presence of guiding headers in the content overview. Consequently, it cannot be inferred from the data which of these three aspects drove the differences between the two conditions. Future research could investigate whether it was one single aspect, or a combination of these that led to the obtained results.
A final limitation relates to the transfer task that was used in this study. No effects on knowledge transfer were found for either cluster membership or task structure. A possible explanation for this lack of findings might be that the transfer task also made a demand on the participants’ writing skills. As such, the measure may have contained construct-irrelevant variance that obscured any potential results.
This study used a qualitative method to find clusters of macro-level navigation patterns. Future research may seek to replicate and quantify these clusters. First, replication is needed to ascertain whether similar patterns are found in different contexts, and whether the patterns are unidimensional. It is possible that different clusters can be found, or that clusters are overlapping in different task contexts. Second, the clusters may be quantified. Given the large amount of data that log-files provide, it can be challenging to decide which measures to extract from the log-files to describe the learning process. Moreover, a clear consensus regarding the methods of analyses for log-files is still lacking in the field (Van Laer & Elen, 2018). Therefore, the qualitative findings of this study could be used to inform the selection of potentially relevant navigation measures.
Another direction for future research is to investigate the stability of the navigation patterns across time. While this study distinguished six macro-level patterns, we cannot ascertain the extent to which children use the same pattern consistently across multiple assignments. Perhaps, children who adapt their macro-level navigation pattern to the task requirements and to their own domain knowledge will have better learning outcomes than those who consistently use the same macro-level pattern. Indeed, previous research has shown that learners with high or low prior knowledge have different needs in an assignment (Goldman, 2009; Kalyuga et al., 2003), and approach tasks differently (Lawless, Brown, Mills, & Mayall, 2003). Consequently, the provided structure should fit the needed support by the learner in order to be effective (Azevedo & Hadwin, 2005).
This study showed that a difference in task structure affected the macro-level navigation patterns of children and the quality of their written hypermedia assignment. A practical implication of these findings is that care should be taken when designing digital resources for primary school children. A relatively small difference in headers or answering formats may result in different learning strategies, as evidenced by navigation patterns, and even in different learning outcomes. As such, it is advisable to test new applications and websites rigorously, so that optimal learning is safeguarded. While such testing is not always performed, teachers could keep an extra eye on children’s learning behaviours online, to help prevent them getting disoriented or distracted (see also Scheiter & Gerjets, 2007).
Our study provides many new directions for research. Six macro-level navigation patterns were found that children used to learn in a hypermedia environment. On average, the quality of their written assignment was higher, and a linear reading pattern more prevalent, in a high structure hypermedia environment. Indeed, findings indicate that the linear reading pattern was related to a higher quality assignment compared to some other navigation patterns. Together these findings show that, while there are multiple ways to navigate through a hypermedia environment, not all ways are equally successful. Moreover, the provided structure in the environment may affect the occurrence of successful navigation patterns, and could therefore affect overall learning success in hypermedia.
1 Some participants had a score of zero on their post-test. Leaving these participants out of the analyses yielded similar results.
2 At the α = .05 level, results showed that both linear reading (Z = 2.18, p = .029, r = .49) and selective reading (Z = 2.42, p = .016, r = .54) had higher knowledge gains than massed writing.
Ananiadou, K., & Claro, M. (2009). 21st century skills
and competences for new millennium learners in OECD countries
. OECD Education Working Papers.
https://doi.org/10.1787/218525261154
Azevedo, R. (2009). Theoretical, conceptual, methodological, and
instructional issues in research on metacognition and
self-regulated learning: a discussion. Metacognition and
Learning, 4(1), 87–95.
https://doi.org/10.1007/s11409-009-9035-7
Azevedo, R., & Cromley, J. G. (2004). Does training on
self-regulated learning facilitate students’ learning with
hypermedia? Journal of Educational Psychology, 96(3),
523–535. https://doi.org/10.1037/0022-0663.96.3.523
Azevedo, R., Greene, J. A., & Moos, D. C. (2007). The effect
of a human agent’s external regulation upon college students’
hypermedia learning. Metacognition and Learning, 2(2–3),
67–87. https://doi.org/10.1007/s11409-007-9014-9
Azevedo, R., & Hadwin, A. F. (2005). Scaffolding
self-regulated learning and metacognition - implications for the
design of computer-based scaffolds. Instructional Science,
33(5–6), 367–379.
https://doi.org/10.1007/s11251-005-1272-9
Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process
mining techniques for analysing patterns and strategies in
students’ self-regulated learning. Metacognition and Learning,
9 (2), 161–185. https://doi.org/10.1007/s11409-013-9107-6
Barab, S. A., Bowdish, B. E., & Lawless, K. A. (1997).
Hypermedia navigation: Profiles of hypermedia users. Educational
Technology Research and Development, 45(3),
23–41. https://doi.org/10.1007/BF02299727
Bezdan, E., Kester, L., & Kirschner, P. A. (2013). The
influence of node sequence and extraneous load induced by
graphical overviews on hypertext learning. Computers in Human
Behavior, 29(3), 870–880.
https://doi.org/10.1016/j.chb.2012.12.016
Blom, H., Segers, E., Knoors, H., Hermans, D., & Verhoeven, L.
(2018). Comprehension and navigation of networked hypertexts. Journal
of Computer Assisted Learning, 34(3), 306–314.
https://doi.org/10.1111/jcal.12243
Boekaerts, M. (1999). Self-regulated learing: where we are today.
International Journal of Educational Research, 31,
445–457. https://doi.org/10.1016/S0883-0355(99)00014-2
de Jong, T., & van Joolingen, W. R. (1998). Scientific
discovery learning with computer simulations of conceptual
domains. Review of Educational Research, 68(2),
179–201. https://doi.org/10.3102/00346543068002179
Delgado, P., Vargas, C., Ackerman, R., & Salmeron, L. (2018).
Don’t throw away your printed books: a meta-analysis on the
effects of reading media on reading comprehension. Educational
Research Review, 25(January), 23–38.
https://doi.org/10.1016/j.edurev.2018.09.003
DeStefano, D., & LeFevre, J.-A. A. (2007). Cognitive load in
hypertext reading: a review. Computers in Human Behavior,
23(3), 1616–1641.
https://doi.org/10.1016/j.chb.2005.08.012
Eilam, B., & Aharon, I. (2003). Students’ planning in the
process of self-regulated learning. Contemporary Educational
Psychology, 28(3), 304–334.
https://doi.org/10.1016/S0361-476X(02)00042-5
Goldman, S. R. (2009). Explorations of relationships among
learners, tasks, and learning. Learning and Instruction,
19(5), 451–454.
https://doi.org/10.1016/j.learninstruc.2009.02.006
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C.,
& Brodowinska, K. (2012). Comprehending and learning from
internet sources: processing patterns of better and poorer
learners. Reading Research Quarterly, 47(4),
356–381. https://doi.org/10.1002/RRQ.027
Gorissen, C. J. J., Kester, L., Brand-Gruwel, S., & Martens,
R. (2015). Autonomy supported, learner-controlled or
system-controlled learning in hypermedia environments and the
influence of academic self-regulation style. Interactive
Learning Environments, 23(6), 655–669.
https://doi.org/10.1080/10494820.2013.788038
Greene, J. A., & Azevedo, R. (2010). The measurement of
learners’ self-regulated cognitive and metacognitive processes
while using computer-based learning environments. Educational
Psychologist, 45(4), 203–209.
https://doi.org/10.1080/00461520.2010.515935
Ignacio Madrid, R., Van Oostendorp, H., & Puerta Melguizo, M.
C. (2009). The effects of the number of links and navigation
support on cognitive load and learning with hypertext: the
mediating role of reading order. Computers in Human Behavior,
25(1), 66–75. https://doi.org/10.1016/j.chb.2008.06.005
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003).
The expertise reversal effect. Educational Psychologist,
38 (1), 23–31. https://doi.org/10.1207/S15326985EP3801_4
Klois, S. S., Segers, E., & Verhoeven, L. (2013). How
hypertext fosters children’s knowledge acquisition: the roles of
text structure and graphical overview. Computers in Human
Behavior, 29(5), 2047–2057.
https://doi.org/10.1016/j.chb.2013.03.013
Kong, Y., Seo, Y. S., & Zhai, L. (2018). Comparison of reading
performance on screen and on paper: a meta-analysis. Computers
and Education, 123(May), 138–149.
https://doi.org/10.1016/j.compedu.2018.05.005
Künsting, J., Wirth, J., & Paas, F. (2011). The goal
specificity effect on strategy use and instructional efficiency
during computer-based scientific discovery learning. Computers
and Education, 56(3), 668–679.
https://doi.org/10.1016/j.compedu.2010.10.009
Lawless, K. A., Brown, S. W., Mills, R., & Mayall, H. J.
(2003). Knowledge, interest, recall and navigation: a Look at
hypertext processing. Journal of Literacy Research, 35(3),
911–934. https://doi.org/10.1207/s15548430jlr3503_5
Lawless, K. A., & Kulikowich, J. M. (1996). Understanding
hypertext navigation through cluster analysis. Journal of
Educational Computing Research, 14(4), 385–399.
https://doi.org/10.2190/DVAP-DE23-3XMV-9MXH
MacGregor, S. K. (1999). Hypermedia navigation profiles: cognitive
characteristics and information processing strategies. Journal
of Educational Computing Research, 20(2),
189–206. https://doi.org/10.2190/1MEC-C0W6-111H-YQ6A
Malmberg, J., Järvenoja, H., & Järvelä, S. (2013). Patterns in
elementary school students′ strategic actions in varying learning
situations. Instructional Science, 41(5),
933–954. https://doi.org/10.1007/s11251-012-9262-1
McDonald, S., & Stevenson, R. J. (1996). Disorientation in
hypertext: the effects of three text structures on navigation
performance. Applied Ergonomics, 27(1), 61–68.
https://doi.org/10.1016/0003-6870(95)00073-9
Mobrand, K. A., & Spyridakis, J. H. (2007). Explicitness of
local navigational links: comprehension, perceptions of use, and
browsing behavior. Journal of Information Science, 33(1),
41–61. https://doi.org/10.1177/0165551506068144
Molenaar, I., & Järvelä, S. (2014). Sequential and temporal
characteristics of self and socially regulated learning. Metacognition
and Learning, 9(2), 75–85.
https://doi.org/10.1007/s11409-014-9114-2
Paans, C., Molenaar, I., Segers, E., & Verhoeven, L. (2019).
Temporal variation in children’s self-regulated hypermedia
learning. Computers in Human Behavior, 96,
246–258. https://doi.org/10.1016/j.chb.2018.04.002
Paans, C., Segers, E., Molenaar, I., & Verhoeven, L. (2018).
The quality of the assignment matters in hypermedia learning. Journal
of Computer Assisted Learning, 34(6).
https://doi.org/10.1111/jcal.12294
Pieschl, S., Stahl, E., Murray, T., & Bromme, R. (2012). Is
adaptation to task complexity really beneficial for performance? Learning
and Instruction, 22(4), 281–289.
https://doi.org/10.1016/j.learninstruc.2011.08.005
Reimann, P. (2009). Time is precious: variable- and event-centred
approaches to process analysis in CSCL research. International
Journal of Computer-Supported Collaborative Learning , 4(3),
239–257. https://doi.org/10.1007/s11412-009-9070-z
Rotherham, A. J., & Willingham, D. T. (2010). 21st century
skills - Not new, but a worthy challenge. American Educator,
34(1), 17–20. https://doi.org/10.1145/1719292.1730970
Rouet, J.-F. (2003). What was I looking for? The influence of task
specifity and prior knowledge on students’ search strategies in
hypertext. Interacting with Computers, 15,
409–428. https://doi.org/10.1016/S0953-5438(02)00064-4
Salmeron, L., Kintsch, W., & Cañas, J. J. (2006a). Coherence
or interest as basis for improving hypertext comprehension. Information
Design Journal, 14(1), 45–55.
https://doi.org/10.1075/idj.14.1.06sal
Salmeron, L., Kintsch, W., & Cañas, J. J. (2006b). Reading
strategies and prior knowledge in learning from hypertext. Memory
and Cognition, 34(5), 1157–1171.
https://doi.org/10.3758/BF03193262
Salmeron, L., Naumann, J., García, V., & Fajardo, I. (2017).
Scanning and deep processing of information in hypertext: an eye
tracking and cued retrospective think-aloud study. Journal of
Computer Assisted Learning, 33(3), 222–233.
https://doi.org/10.1111/jcal.12152
Scheiter, K., & Gerjets, P. (2007). Learner control in
hypermedia environments. Educational Psychology Review,
19(3), 285–307. https://doi.org/10.1007/s10648-007-9046-3
Schoor, C., & Bannert, M. (2012). Exploring regulatory
processes during a computer-supported collaborative learning task
using process mining. Computers in Human Behavior, 28(4),
1321–1331. https://doi.org/10.1016/j.chb.2012.02.016
Segers, E., & Verhoeven, L. (2009). Learning in a sheltered
Internet environment: the use of WebQuests. Learning and
Instruction, 19(5), 423–432.
https://doi.org/10.1016/j.learninstruc.2009.02.017
Sobocinski, M., Malmberg, J., & Järvelä, S. (2017). Exploring
temporal sequences of regulatory phases and associated
interactions in low- and high-challenge collaborative learning
sessions. Metacognition and Learning, 12(2),
275–294. https://doi.org/10.1007/s11409-016-9167-5
Sonnenberg, C., & Bannert, M. (2015). Discovering the effects
of metacognitive prompts on the sequential structure of
SRL-processes using process mining techniques. Journal of
Learning Analytics, 2(1), 72–100.
https://doi.org/10.18608/jla.2015.21.5
Urakami, J., & Krems, J. F. (2012). How hypertext reading
sequences affect understanding of causal and temporal relations in
story comprehension. Instructional Science, 40(2),
277–295. https://doi.org/10.1007/s11251-011-9178-1
van der Stel, M., & Veenman, M. V. J. (2008). Relation between
intellectual ability and metacognitive skillfulness as predictors
of learning performance of young students performing tasks in
different domains. Learning and Individual Differences,
18(1), 128–134.
https://doi.org/10.1016/j.lindif.2007.08.003
Van Laer, S., & Elen, J. (2018). Towards a methodological
framework for sequence analysis in the field of self-regulated
learning. Frontline Learning Research, 6(3),
228–249. https://doi.org/10.14786/flr.v6i3.367
Veenman, M. V. J. (2011). Learning to self-monitor and
self-regulate. In R. Mayer & P. Alexander (Eds.), Handbook
of research on learning and instruction. New York, NY:
Routledge.
Veenman, M. V. J. (2015). Metacognition. Handbook of
Individual Differences in Reading, Reader, Text, and Context
. https://doi.org/10.4324/9780203075562.ch3
Winne, P. H., & Hadwin, A. F. (1998). Studying as
self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C.
Graesser (Eds.), Metacognition in educational theory and
practice (pp. 277–304). Mahwah, NJ: Lawrence Erlbaum.
Winne, P. H., Hadwin, A. F., & Gress, C. L. Z. (2010). The
learning kit project: software tools for supporting and
researching regulation of collaborative learning. Computers
in Human Behavior, 26 (5), 787–793.
https://doi.org/10.1016/j.chb.2007.09.009
Winne, P. H., & Nesbit, J. C. (2010). The psychology of
academic achievement. Annual Review of Psychology, 61,
653–678. https://doi.org/10.1146/annurev.psych.093008.100348
Zimmerman, B. J. (1989). A social cognitive view of self-regulated
academic learning. Journal of Educational Psychology, 81(3),
329–339. https://doi.org/10.1037/0022-0663.81.3.329
Zumbach, J., & Mohraz, M. (2008). Cognitive load in hypermedia
reading comprehension: influence of text type and linearity. Computers
in Human Behavior, 24(3), 875–887.
https://doi.org/10.1016/j.chb.2007.02.015
Flowchart for coding the macro-level navigation patterns.
